Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions docs/architecture/modules/runtime-outcome-reconciliation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# Runtime outcome reconciliation

This module is the evidence-driven, privileged exit for a managed DIRECT execution parked after
the provider dispatch boundary. It converges known provider evidence without executing the Agent
again.

## Admission and evidence

The API requires the managed-runtime and outcome-reconciliation feature gates, the
`outcome:reconcile` permission, an authenticated same-tenant Principal, and an idempotency key. It
does not require the direct-cutover gate because disabling new admission must not strand existing
work.

The request contains a complete public `RuntimeObservation`, its canonical digest, a bounded
evidence reference, and a bounded operator reason. Only `SUCCEEDED`, `FAILED`, `CANCELED`, and
`TIMED_OUT` observations are accepted. Execution, assignment, digest, phase, and provider identity
evidence must match the persisted Runtime execution. All phases require empty usage because parking
has already conservatively settled budget and this slice does not yet support actual-usage evidence.
Terminal evidence cannot retain governed-action or wait requests, and success cannot carry an error.
Success remains limited to mapping output; `output_artifact_refs` may accompany the canonical
observation but this slice terminates the Task only from the mapping output. Artifact materialization
and post-reconciliation actual-usage accounting are explicit follow-up work. Other terminal phases
cannot carry successful output.

## Atomic convergence

After locating the execution without a lock, the service locks Task, Run, latest Attempt, then
RuntimeExecution and revalidates the complete parked quartet. In one UoW it records or reuses exact
immutable observation evidence, reconciles Runtime and business state, adds a TaskResolution and
`agentmesh.runtime.outcome-reconciled` Outbox event, and stores the idempotency result. It never
holds a provider call inside a transaction because it never calls a provider at all.

Exact replay returns the existing resolution. A different request using the same key, conflicting
evidence, stale ownership, or a concurrently settled execution fails closed. Competing operators
therefore have one committed winner.

Confirmed success at or after the UTC budget deadline leaves Runtime, Run, and Attempt succeeded
but places the Task in `WAITING_APPROVAL` with candidate output. Parking already settled budget and
released quota, so reconciliation does not repeat those operations. A confirmed cancellation only
maps Task/Run/Attempt to canceled when a persisted cancel intent exists; otherwise the Runtime is
canceled and the business objects fail with `runtime.unrequested_cancellation`.

## Rollback boundary

The writer uses reader/schema compatibility delivered by A4.1b.2a. Once new observation or
resolution values have been written, migration 0048 is the schema floor. Operators may disable the
writer gate and roll the application back to the compatibility release, but must not downgrade to
0047. This slice does not provide reviewed/coordinated cutover, generic subprocess authority, or
production durable reattach.
27 changes: 26 additions & 1 deletion docs/implementation-status.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Implementation status

Status: Alpha baseline
Last updated: 2026-08-21
Last updated: 2026-08-23

This page records what the repository actually implements. The formal L2 documents describe the
target architecture; an implemented vertical slice does not imply that every capability in its
Expand Down Expand Up @@ -150,6 +150,31 @@ A4.1b.2a reconciliation reader compatibility:
new value, 0048 becomes the schema floor; application rollback targets this compatibility
release rather than 0047.

A4.1b.2b evidence-driven outcome reconciliation (writer slice):

- A privileged `POST /api/v1/runtime-executions/{execution_id}/reconcile-outcome` command accepts
a canonical, identity-bound terminal `RuntimeObservation`; it requires the managed-runtime and
outcome-reconciliation gates, `outcome:reconcile`, an authenticated same-tenant Principal, and
an `Idempotency-Key`. It deliberately does not depend on the new-Run cutover gate.
- The command is the only exit from the managed DIRECT reconciliation states. It locks and
revalidates the Task/Run/latest Attempt/Runtime quartet, never calls or redispatches a provider,
and commits evidence, Runtime and business convergence, TaskResolution, Outbox, and idempotency
in one UoW. Exact replay is side-effect free; conflicting evidence fails closed.
- All confirmed outcomes require empty usage because parking has already conservatively settled
budget; terminal evidence also rejects remaining governed-action/wait requests, and success
rejects an error. Confirmed success retains the A4.1 mapping-output limit. An observation at or after
the pinned deadline preserves the provider success but moves the Task to `WAITING_APPROVAL` with
candidate output. Confirmed cancellation maps business objects to `CANCELED` only when a
persisted cancel intent exists; otherwise they fail as `runtime.unrequested_cancellation`.
- Parking already conservatively settles budget and releases quota, so reconciliation never does
either twice. Completed-task memory capture remains transactional and research materialization
remains post-commit best effort. PostgreSQL coverage exercises atomic convergence and rollback,
competing conclusions, replay, pre-existing evidence, stale fencing, and zero redispatch.
- This remains CI/test-only and disabled by default. Once a writer stores 0048-only values, schema
0048 is the rollback floor; roll back the application to the deployed b.2a compatibility release,
not to a pre-0048 reader. Reviewed/coordinated authority and production durable reattach remain
outside this slice.

## Current runnable baseline

AgentMesh currently provides durable direct, independently reviewed, and coordinated Subtask DAG
Expand Down
16 changes: 9 additions & 7 deletions docs/operations/runtime-direct-cutover-rollback.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Deterministic direct-runtime admission rollback

Last updated: 2026-08-21
Last updated: 2026-08-23

This runbook describes the A4.1a/A4.1b.1 CI/test-only admission and managed DIRECT Worker path.
This runbook describes the A4.1a/A4.1b CI/test-only admission, managed DIRECT Worker, and
evidence-driven reconciliation path.
It is not a production runtime cutover procedure.

## Scope
Expand All @@ -29,13 +30,14 @@ immutable persisted snapshot.
4. Inspect RuntimeExecution phase, current owner/fence, latest Attempt, Inbox, and reconciliation
Outbox evidence. `DISPATCHING` or later with an expired owner and no reattach proof must park as
`RECONCILIATION_REQUIRED`; it must not be redispatched or replaced by an ordinary Run.
5. A4.1b.1 deliberately has no exit from `RECONCILIATION_REQUIRED`. Escalate and preserve the
evidence until the privileged A4.1b.2 reconcile command is available. Manual status edits,
direct database repair, and blind provider retry are prohibited.
5. Use only the gated, permission-protected runtime outcome reconciliation command to converge a
`RECONCILIATION_REQUIRED` execution from canonical provider evidence. The command remains
available when direct-cutover admission is off and never redispatches the provider. Manual
status edits, direct database repair, and blind provider retry are prohibited.

Migration 0047 keeps legacy rows valid. Migration 0048 is the expand phase for future Runtime
outcome reconciliation readers and storage; this compatibility release does not write its new
values, so a clean 0048-to-0047 downgrade remains supported before writer activation. Once a later
release writes `RECONCILED` observation evidence or `RECONCILE_RUNTIME_*` TaskResolution actions,
values, so a clean 0048-to-0047 downgrade remains supported before writer activation. Once the
writer release stores `RECONCILED` observation evidence or `RECONCILE_RUNTIME_*` TaskResolution actions,
0048 becomes the schema floor. Roll application binaries back to the 0048 compatibility release,
not to an older reader, and never rewrite reconciliation audit evidence to force a downgrade.
8 changes: 5 additions & 3 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Design and delivery roadmap

Status: Alpha
Last updated: 2026-08-20
Last updated: 2026-08-23

路线图使用可验证的垂直切片推进。阶段编号描述交付成熟度,不等同于架构文档的 L0–L3。
各正式 L2 模块的当前代码成熟度与下一交付队列见
Expand Down Expand Up @@ -172,14 +172,16 @@ Exit signal:用户可从模板创建公司、绑定真实 Agent,在不伪造
- [x] A4.1b.1 CI/test-only managed DIRECT Worker authority(fenced dispatch、原子终结、
reconciliation-required 停车;不含 reconcile command)
- [x] A4.1b.2a reconciliation reader/schema compatibility(0048 expand-only;不含 writer/API)
- [x] A4.1b.2b evidence-driven privileged outcome reconciliation(canonical observation、原子收敛、
幂等/并发保护、无 provider redispatch;默认关闭)
- [ ] MCP write 和 fake external action 通过统一 Intent/Permit/Receipt/Reconciliation
- [ ] Chaos smoke 证明核心 crash windows 收敛且无重复不可逆副作用

Exit signal:同一部署管理 LangGraph 与非 LangGraph Agent;两者使用同一 Task/Run/Attempt、
身份、治理、Artifact 和恢复语义,并由机器可读故障报告证明关键不变量。

当前 A4.0 conformance harness 已在 PR #150 完成,A4.1a admission 与 A4.1b.1 managed DIRECT
Worker authority/atomic parking 已交付,A4.1b.2a reader/schema compatibility 已完成。
A4.1b.2b 仍需受权限控制、证据驱动的 reconcile command;
Worker authority/atomic parking 已交付,A4.1b.2a reader/schema compatibility 与 A4.1b.2b
受权限控制、证据驱动的 reconcile command 已完成
完整 A4 还需 chaos、parity、reviewed/coordinated cutover 和生产 durable runtime。#135/#136
继续保持开放。
86 changes: 82 additions & 4 deletions src/agentmesh/api/runtime_routes.py
Original file line number Diff line number Diff line change
@@ -1,23 +1,29 @@
from __future__ import annotations

from datetime import datetime
from typing import Annotated
from typing import Annotated, Any
from uuid import UUID

from fastapi import APIRouter, Depends, Query, Request
from pydantic import BaseModel
from fastapi import APIRouter, Depends, Header, Query, Request
from pydantic import BaseModel, ConfigDict

from agentmesh.api.feature_routes import require_feature
from agentmesh.api.schemas import TaskResolutionResponse
from agentmesh.api.security import PrincipalDependency, require_permission
from agentmesh.application.runtime_reconciliation import (
RuntimeOutcomeReconciliationResult,
RuntimeOutcomeReconciliationService,
)
from agentmesh.application.runtime_services import RuntimeRegistryService
from agentmesh.domain.errors import AuthorizationDenied
from agentmesh.domain.errors import AuthorizationDenied, InvalidTaskInput
from agentmesh.domain.identity import Permission
from agentmesh.domain.runtime_execution import (
RuntimeExecution,
RuntimeRegistration,
RuntimeVersion,
)
from agentmesh.features import Feature
from agentmesh.runtime_sdk import RuntimeContractError, RuntimeObservation

router = APIRouter(prefix="/api/v1", tags=["runtime-control-plane"])
_dependencies = [
Expand All @@ -26,6 +32,7 @@
]
Limit = Annotated[int, Query(ge=1, le=100)]
Offset = Annotated[int, Query(ge=0)]
IdempotencyKey = Annotated[str, Header(alias="Idempotency-Key", min_length=1, max_length=200)]


class RuntimeRegistrationResponse(BaseModel):
Expand Down Expand Up @@ -87,6 +94,23 @@ class RuntimeObservationResponse(BaseModel):
provider_event_present: bool


class ReconcileRuntimeOutcomeRequest(BaseModel):
model_config = ConfigDict(extra="forbid")

# Keep the versioned Runtime contract opaque to Pydantic so malformed
# values are normalized by the SDK boundary below instead of being echoed
# in FastAPI's default validation response.
observation: Any
evidence_digest: str
evidence_reference: str
reason: str


class ReconcileRuntimeOutcomeResponse(BaseModel):
execution: RuntimeExecutionResponse
resolution: TaskResolutionResponse


def _service(request: Request) -> RuntimeRegistryService:
service = request.app.state.container.runtime_service
if service is None:
Expand All @@ -97,6 +121,18 @@ def _service(request: Request) -> RuntimeRegistryService:
RuntimeServiceDependency = Annotated[RuntimeRegistryService, Depends(_service)]


def _reconciliation_service(request: Request) -> RuntimeOutcomeReconciliationService:
service = request.app.state.container.runtime_reconciliation_service
if service is None:
raise RuntimeError("Runtime reconciliation service is not configured")
return service


RuntimeReconciliationServiceDependency = Annotated[
RuntimeOutcomeReconciliationService, Depends(_reconciliation_service)
]


def _principal_uuid(principal: PrincipalDependency) -> UUID | None:
try:
return UUID(principal.principal_id)
Expand Down Expand Up @@ -207,3 +243,45 @@ def list_observations(
RuntimeObservationResponse(**value)
for value in service.list_observations(execution_id, limit=limit, offset=offset)
]


@router.post(
"/runtime-executions/{execution_id}/reconcile-outcome",
response_model=ReconcileRuntimeOutcomeResponse,
dependencies=[
*_dependencies,
Depends(require_feature(Feature.OUTCOME_RECONCILIATION)),
Depends(require_permission(Permission.OUTCOME_RECONCILE)),
],
)
def reconcile_runtime_outcome(
execution_id: UUID,
payload: ReconcileRuntimeOutcomeRequest,
principal: PrincipalDependency,
service: RuntimeReconciliationServiceDependency,
idempotency_key: IdempotencyKey,
) -> ReconcileRuntimeOutcomeResponse:
if principal.tenant_id != service.tenant_id or not principal.authenticated:
raise AuthorizationDenied("Runtime tenant scope denied")
try:
observation = RuntimeObservation.from_dict(payload.observation)
except RuntimeContractError as exc:
# Runtime contract errors can contain field-level details derived from an
# untrusted request. Keep the public error stable and bounded while the
# domain exception handler maps it to HTTP 422.
raise InvalidTaskInput(
"Runtime reconciliation observation is invalid"
) from exc
result: RuntimeOutcomeReconciliationResult = service.reconcile_outcome(
execution_id,
principal=principal,
observation=observation,
evidence_digest=payload.evidence_digest,
evidence_reference=payload.evidence_reference,
reason=payload.reason,
idempotency_key=idempotency_key,
)
return ReconcileRuntimeOutcomeResponse(
execution=_execution(result.execution),
resolution=TaskResolutionResponse.from_domain(result.resolution),
)
3 changes: 3 additions & 0 deletions src/agentmesh/application/ports.py
Original file line number Diff line number Diff line change
Expand Up @@ -212,6 +212,9 @@ def update_observation_outcome(
*,
outcome: RuntimeObservationOutcome,
) -> None: ...
def find_cancel_intent(
self, execution_id: UUID, *, tenant_id: str
) -> RuntimeLifecycleIntent | None: ...
def add_lifecycle_operation(self, value: RuntimeLifecycleIntent) -> None: ...
def find_lifecycle_operation(
self, execution_id: UUID, *, tenant_id: str, operation_id: str
Expand Down
Loading