Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,28 @@

## Unreleased

### Telemetry, Token-Conservation Analytics & Operational Reporting — vNext.10.2 Phase 9

- Added a versioned Job/Mission execution telemetry report derived from
durable WorkUnit, routing, attempt, research, cooldown, verification,
closure, and autonomy facts. Reports keep unknown usage as `null`, expose
coverage, deduplicate replayed attempt IDs, redact diagnostics, and remain
observational rather than granting authority or completion.
- Added formal Secondary success/repair/fallback funnels,
`StrongBuilderAvoidanceRatio` with Strong evaluator separation, Strong
direct/fallback attribution, research avoidance/reuse and lifecycle
breakdowns, cooldown productivity/avoidable-idle diagnostics, restart and
candidate reuse accounting, completed-work redo detection, and a compact
Phase 10-compatible qualification summary.
- Added `specbridge report job <jobId>` with JSON and verbose formats plus the
read-only MCP `job_report` tool. CLI reports persist under
`.specbridge/reports/` by default; MCP derives the same contract without
writing runtime state.
- Added deterministic mixed-compute, Strong-only replay, research, cooldown,
restart/deduplication, token coverage, intervention, reliability, old-Job,
corruption, CLI, and MCP qualification coverage, and regenerated the public
contracts, tool reference, and Claude/Codex plugin bundles.

### Subscription Cooldown Continuation & Compute Survival — vNext.10.2 Phase 8

- Added durable, resource-scoped Strong subscription cooldown state at the
Expand Down
15 changes: 15 additions & 0 deletions contracts/cli-commands.json
Original file line number Diff line number Diff line change
Expand Up @@ -761,6 +761,21 @@
}
}
},
"report": {
"options": [
"--help"
],
"subcommands": {
"job": {
"options": [
"--help",
"--json",
"--no-persist",
"--verbose"
]
}
}
},
"research": {
"options": [
"--help"
Expand Down
1 change: 1 addition & 0 deletions contracts/mcp-contract.json
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
"job_cancel",
"job_list",
"job_read",
"job_report",
"mission_answer",
"mission_assess",
"mission_begin",
Expand Down
5 changes: 4 additions & 1 deletion docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,9 @@ the intent in the evening, run one command, read the report in the morning.
self-repair that cannot weaken its own constraints.
- [Autonomy telemetry](autonomy/telemetry.md) — the honest report, where an
unknown is `null` and never zero.
- [Execution telemetry and token conservation](autonomy/execution-telemetry.md)
— versioned Phase 9 Job/WorkUnit routing, research, cooldown, reliability,
verification, and closure analytics without new authority.
- [Zero-touch certification](autonomy/certification.md) — the sixteen-fault
matrix, and the three defects it found.
- [Operator setup](autonomy/operator-setup.md) — the evening checklist.
Expand Down Expand Up @@ -243,7 +246,7 @@ the intent in the evening, run one command, read the report in the morning.
## MCP & frontend plugins

- [MCP server](mcp-server.md) — the local stdio server.
- [MCP tool reference](mcp/tool-reference.md) — all 77 tools (generated).
- [MCP tool reference](mcp/tool-reference.md) — all 78 tools (generated).
- [MCP tools](mcp-tools.md) · [resources](mcp-resources.md) ·
[prompts](mcp-prompts.md) · [CLI/MCP parity](cli-mcp-parity.md).
- [Claude Code integration](claude-code-integration.md) — both directions.
Expand Down
93 changes: 93 additions & 0 deletions docs/autonomy/execution-telemetry.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# Execution telemetry and token-conservation reports

Phase 9 turns durable Job, WorkUnit, routing, research, cooldown,
verification, closure, and autonomy records into one versioned operational
report. The report is observational: it cannot approve work, change routing,
grant authority, or make a Job complete.

## Read a report

```bash
specbridge report job <jobId>
specbridge report job <jobId> --verbose
specbridge report job <jobId> --json
```

The human view is concise by default. `--verbose` adds bounded WorkUnit
accounting and the cooldown timeline. The JSON view carries
`schemaVersion: "1.0.0"` and, unless `--no-persist` is used, is saved at
`.specbridge/reports/job-<jobId>-telemetry.json`.

MCP clients can call the read-only `job_report` tool. It derives the same
report without persisting a file.

## Formal metric definitions

A ratio is represented as `{ numerator, denominator, value }`. `value` is
`null` when the denominator is zero or the underlying durable facts do not
exist. Unknown measurements are never reported as zero.

| Metric | Numerator | Denominator | Exclusions and unknown semantics |
| --- | --- | --- | --- |
| `StrongBuilderAvoidanceRatio` | Eligible, completed implementation WorkUnits with no `STRONG` or `STRONG_FALLBACK` builder attempt | Eligible, completed implementation WorkUnits | Excludes `STRONG_REQUIRED`, cancelled, authority-pending, research-pending, context-pending, and incomplete work. A Strong evaluator is not a Strong Builder. `null` when no eligible implementation WorkUnit completed. |
| `SecondaryInitialSuccessRate` | WorkUnits whose initial `SECONDARY` attempt passed without repair | WorkUnits with an initial `SECONDARY` attempt | `null` when Secondary was never initially attempted. Replayed attempt IDs are counted once. |
| `SecondaryRepairRecoveryRate` | WorkUnits whose `SECONDARY_REPAIR` path passed | WorkUnits that received a `SECONDARY_REPAIR` attempt | `null` when repair was never attempted. |
| `SecondaryToStrongFallbackRate` | Secondary-attempted WorkUnits that later used `STRONG_FALLBACK` | WorkUnits with any Secondary implementation attempt | `null` when Secondary was never attempted. |
| `ResearchAvoidanceRatio` | Research-gate considerations resolved without a new QUICK or DEEP provider call | Research-gate considerations | `null` where gate aggregates are unavailable. The report labels existing aggregate gate telemetry as `WORKSPACE`; it does not pretend it is Job-scoped. Job-scoped provider records remain Job-scoped. |
| `ResearchReuseRate` | Reused prior research reports | Reused reports plus new provider calls | `null` when neither reuse nor a new call occurred. |
| `UsefulWorkDuringSubscriptionCooldown` | Distinct WorkUnits durably recorded as completed during a Strong subscription cooldown | Not a ratio | Includes productive Secondary and research progress while Strong-required work waits. Duplicate WorkUnit IDs within the same objective are counted once. |
| `ZeroTouchAfterSeal` | `true` only when `humanInterventionsAfterSeal` is zero | Not a ratio | Correct authority escalation is reported separately. A missing seal boundary is diagnosed by the underlying autonomy telemetry; it is never inferred from a successful outcome. |
| `CompletedWorkRedoCount` | Implementation attempts that began after the same WorkUnit had already been durably integrated | Not a ratio | Resource waits and replayed copies of the same durable attempt ID are excluded. Candidate reuse and restart recovery are reported separately. |

## Work accounting

Every current WorkUnit belongs to exactly one reporting category:
`completed`, `failed`, `cancelled`, `waiting`, `not-ready`,
`human-authority-pending`, `research-pending`, or `context-pending`. This is
reporting classification only and does not alter the WorkUnit state machine.

The same accounting is rolled up per Objective and for the whole Job;
`missionId` links the Job report to its sealed Mission when that durable
binding exists. Secondary eligibility reports both the eligible total and an
explicit ineligible total, with Strong-required, research, authority,
context, and not-ready reasons kept separate.

Implementation builder calls and evaluator calls are separate populations.
`STRONG` and `STRONG_FALLBACK` count as Strong Builder calls;
non-local `EVALUATOR` ledger entries count only as Strong evaluator calls.

## Token and research coverage

Provider-reported input and output tokens are summed only when present.
`knownTokens` is the sum of observed components; `completeTokens` is `null`
unless every attempt reported both components. Coverage states how many
attempts reported any, input, and output usage. Research cost and duration
follow the same rule: absence is `null`, not free or instantaneous.

Research reports expose `JOB`, `WORKSPACE`, or `NONE` scope. Lifecycle
breakdowns cover conversation, spec draft, intake decision, and runtime
investigation. Workspace-scoped legacy aggregates are explicitly labeled so
cross-Job evidence cannot be mistaken for a Job-only total.

## Provenance, safety, and compatibility

The report records Job and objective graph revisions, event and execution
ledger watermarks, seal references, the current non-secret strategy summary,
and whether the current runtime policy differs from the sealed binding.
Diagnostic text is bounded and redacted. Prompts, transcripts, raw provider
payloads, environment secrets, and credentials are never report fields.

Older Jobs remain reportable. Missing Phase 7/8, research, seal, or closure
records produce empty or `null` sections plus diagnostics rather than
invented facts. Inputs and output arrays are bounded, and duplicate durable
attempt IDs are deterministically deduplicated.

## Comparing routing strategies

Efficiency claims require comparable verification and closure outcomes.
The JSON report can carry an explicitly identified observed or qualification
baseline; it reports whether outcomes match before presenting Strong Builder
call reduction. `correctnessEqual` remains false when trusted verification or
closure evidence is unavailable, even when both missing values match. This is
evidence for qualification and operations, not a learned-routing or
production-readiness claim.
5 changes: 5 additions & 0 deletions docs/autonomy/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,11 @@ first process restart — and process restarts are the normal case here.
specbridge autonomy report <jobId>
```

For the Phase 9 Job/WorkUnit routing, research, cooldown, token-conservation,
verification, and closure report, see
[Execution telemetry and token conservation](execution-telemetry.md) and run
`specbridge report job <jobId>`.

```
Autonomy
✓ humanInterventionsAfterSeal: 0
Expand Down
3 changes: 2 additions & 1 deletion docs/mcp/tool-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ Generated from the authoritative registries of the `specbridge` MCP server
(version 1.1.0). Tool names, resource URI templates, and prompt
names are stable contracts — see docs/stability/public-contracts.md.

## Tools (77)
## Tools (78)

| Tool | Access | Summary |
| --- | --- | --- |
Expand All @@ -22,6 +22,7 @@ names are stable contracts — see docs/stability/public-contracts.md.
| `job_cancel` | write | Cancel a job (final, idempotent, evidence preserved) |
| `job_list` | read-only | List long-running orchestration jobs |
| `job_read` | read-only | One job in depth: graph, attempts, questions, checkpoint |
| `job_report` | read-only | Durable Job operational and token-conservation report |
| `mission_answer` | write | Record the user’s answer to one discovery question |
| `mission_assess` | write | Record a governed structured discovery assessment |
| `mission_begin` | write | Begin Mission Discovery from a product direction |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,12 @@
"bytes": 155994
},
"cli.cjs": {
"sha256": "9b6d58aef80578e88c81a1c48606cb13ac0d02d41d9acccc7e3cc67348c1c698",
"bytes": 5739368
"sha256": "91d51d28006410e893ba0f19b1989e0e170a4c2820bf3fd2770a84686777fe2c",
"bytes": 5814917
},
"mcp-server.cjs": {
"sha256": "f096d9dff325c106c15334e6c67baef15771789e5aa2ddd06858894987ae9b4c",
"bytes": 3669226
"sha256": "0cf0895d3af59c3de92fbf899965f2c093145e1f5aa3bf23846ec59e9a5568d6",
"bytes": 3758229
}
}
}
Loading
Loading