[Feature]: Harness Watch — automated paired comparison of OpenHands vs other ACP harnesses, with harness-neutral telemetry

[simonrosenberg]

Umbrella issue spanning OpenHands/software-agent-sdk, OpenHands/benchmarks, All-Hands-AI/evaluation, OpenHands/openhands-index-results, and LiteLLM. Keep it whole until P0 proves the end-to-end design.

Scope and delivery map

Harness Watch should automatically answer on identical work:

1. Did OpenHands solve more or less than another harness?
2. How much provider-visible inference, time, and money did each harness consume?
3. Which paired instances drove a material difference, and what is the smallest experiment worth running next?

The six sections are:

1. Measurement plane — capture comparable provider requests and costs for every harness.
2. Harness integrations — make OpenCode first-class, then add Pi and Hermes through the proven contract.
3. Paired runner and panel — run every harness on the same model and instance manifest.
4. Comparison report — compute deterministic paired outcomes and efficiency statistics.
5. Detection and scheduling — run weekly, confirm large deltas, and file a tracking report.
6. Trajectory diagnosis and experiment handoff — explain the instances driving a confirmed delta and propose a testable ablation.

Each section separates the smallest complete P0 from ordered P1 follow-ups that must not delay it.

P0 operating envelope

P0 is one vertical slice:

• harnesses: OpenHands and OpenCode;
• one explicitly chosen, shared model, initially a non-Anthropic model such as GLM-5.2;
• one fixed, committed panel of roughly 30–50 instances from one benchmark family;
• one run per harness/instance each week, with one automatic confirmation rerun only when an alarm fires;
• one paired comparison of OpenHands against OpenCode;
• internal/tracking-issue reporting first.

OpenCode is the comparator implicated by the motivating data, and a second harness proves every framework boundary: integration, telemetry, manifest execution, reporting, scheduling, confirmation, and diagnosis. Pi and Hermes are the first two P1 integrations, added without redesign. This is a repeatable declared comparison, not a universal leaderboard.

---

Problem and current evidence

rajshah4/harness-benchmark shows that the harness alone can materially change correctness, time, provider requests, context, cache use, and cost. On its GLM-5.2 full-stack incident task, OpenHands used about 2.5× OpenCode's input tokens and over 3× its cost (95 versus 76 provider calls). A repeat used 129 calls at about 79K average input tokens/call for OpenHands, versus 69/46K for Pi and 76/36K for OpenCode. The repository also has eight short tasks, two medium projects, repeats, a Sonnet lane, and a current-main follow-up.

The evidence is useful but not yet a population estimate:

• most cells have one trial and the tasks are a convenience sample;
• some follow-ups changed skills, tools, model, cache behavior, or harness revision together;
• the Sonnet and GLM ordering differs, showing harness × model interaction;
• verifier success does not establish equal maintainability or architecture.

Manual attribution changed with better evidence: the estimated 33–52% browser-schema spend became a directly measured ~2,336 tokens/call (~6.5%); a 72.3% cache-read result proved to be a bad provider run and repeated at ~87%; reducing default skills from ~59 to 11 materially changed current-main. Browser failures, loop length, skill context, cache behavior, and model interaction also mattered. These useful corrections took a Slack thread and a day of trace inspection.

We already evaluate Claude Code, Codex, and Gemini CLI through ACP and publish alternative-agent results at index.openhands.dev. What is missing is a standing paired comparison with trustworthy inference accounting and an automatic path from a confirmed difference to inspectable evidence.

---

1. Harness-neutral measurement plane

Why the current metrics cannot be compared directly

Native telemetry measures SDK completions; ACPAgent._record_usage() and acp.completion measure one outer ACP prompt, inside which a subprocess may make many requests. Schema v6 of summarize_efficiency.py calls len(metrics.costs) iterations, but zero/unknown-cost completions are absent and auxiliary calls are not iterations. ACP tracing is therefore behavioral evidence, not an internal-request counter.

PR #4623's prompt decomposition exists only at the native SDK LLM boundary, so it is OpenHands self-diagnosis rather than a cross-harness authority. Cache accounting is also unsafe: MetricsSnapshot.cache_hit_rate, summarize_efficiency.py, and recalculate_costs.py infer provider semantics from token magnitudes. P0 must use explicit semantics for its chosen route and return unknown otherwise.

Existing neutral source

Every attempt already has a per-instance LiteLLM virtual key, used by native OpenHands through build_eval_llm() and injected into ACP subprocesses by build_acp_agent(). LiteLLM_SpendLogs records request identity/model/status, hashed key, tokens, spend, timestamps, cache fields, normalized usage_object, provider-specific additional_usage_values, and—when enabled—request/response bodies.

Production retains rows for 30 days, cleans nightly, stores prompts, disables client no-log, exposes /spend, and uses chart 0.48.0. This makes the ledger available, not infallible: asynchronous/database logging can fail, so coverage and totals must be reconciled.

Verified against OpenHands/litellm@8a3a0f4:

• /spend/logs/v2 requires both dates, returns at most 100 rows/page, and sorts newest-first.
• its api_key filter is verbatim against the stored hash, so callers must pass sha256(virtual_key.encode()).hexdigest().
• Key deletion does not cascade into spend logs because api_key is a plain string, not a foreign key.
• run attribution must use metadata.spend_logs_metadata; arbitrary virtual-key metadata is not copied to spend logs.
• rows may contain messages, response, proxy_server_request, and sensitive metadata; export only an allowlist.
• a row is a proxy request, not an agent iteration or provider attempt until retries/fallbacks are calibrated.

P0 — ship now

• Add get_key_request_logs() in benchmarks/utils/litellm_proxy.py: hashed key, explicit UTC bounds, full pagination, and existing retry/backoff.
• Export only request ID, call type, model/group/provider, token/spend/cache/status/timestamps, usage_object, additional_usage_values, and Harness Watch attribution. Never export bodies, headers, or unprojected metadata.
• Put experiment_id, block_id, run_id, instance_id, agent_type, and attempt in spend_logs_metadata. Capture every attempt—including failed attempts before key deletion—in EvalOutput or versioned per-attempt telemetry JSONL.
• Wait for quiescence and reconcile row spend with /key/info within a declared tolerance. Missing, partial, unreconciled, or unexpectedly empty telemetry is incomplete, not zero; only pre-inference setup failures may legitimately have no rows.
• Normalize fresh/cache input for the P0 route from explicit semantics, preserve raw projected fields, and emit unknown instead of using the magnitude heuristic.
• Rename the comparable count to proxy_requests. Update summarize_efficiency.py schema v7 to use ledger rows when present and stop presenting len(metrics.costs) as comparable iterations.
• Add a pinned-LiteLLM container test with a fake upstream covering known calls, zero cost, failure/retry, pagination, hashed filtering, attribution, and body exclusion. Mock-only tests miss the silent hash-filter failure.
• On at least five smoke instances, verify that native OpenHands and OpenCode produce nonempty rows, the requested model is observed, and proxy spend reconciles.

P1 — ordered follow-ups

• P1.1 — Neutral request-shape aggregates: Derive body-free system/instruction, tool-schema, history, latest-message, role/message/tool counts, cache-eligible prefix, and request-parameter features at the proxy boundary.
• P1.2 — Request-role classification: Identify primary decisions, condensation, titles, summaries, subagents, retries, fallbacks, and auxiliary inference where evidence permits; total spend remains authoritative.
• P1.3 — Central cache normalization: Replace three magnitude heuristics with one versioned provider/API map for reads, writes, unknown coverage, and billing semantics.
• P1.4 — Least-privilege telemetry endpoint: Serve only safe projections/aggregates under a scoped evaluation credential, never a master key or prompt bodies.
• P1.5 — Prompt-composition self-diagnosis: Reuse feat(sdk): add per-call prompt token composition metrics #4623 offline and optionally at runtime for OpenHands; do not make cross-harness claims until the neutral plane exposes equivalent features.
• P1.6 — Historical backfill: Optionally recover recent rows for exploratory variance estimates, clearly labelled observational and never required for Harness Watch.

---

2. OpenCode first, then Pi and Hermes

“First-class” means a recognized SDK provider/agent, benchmark command, pinned image dependency, isolated headless model/proxy configuration, workflow/metadata/index labels, and an acceptance test. Custom ACP remains useful for prototypes but skips provider behavior and isolation; a run that silently uses stored defaults is invalid.

Current external status

Harness	ACP status	Integration notes
OpenCode	First-party opencode acp, documented and listed by ACP clients.	Provider config supports baseURL, {env:VAR} keys, and model registration. Config merges across sources, so isolate roots and set primary/auxiliary models explicitly. The SDK already auto-approves permissions; OpenCode modes are agent profiles, not permission bypass.
Pi	Third-party pi-acp, which starts a separately installed pi --mode rpc; ACP-listed.	Pin both packages, isolate ~/.pi, and configure the provider/model separately. earendil-works/pi#175—not the earlier incorrect #4444—tracks relevant work. The MVP adapter has documented limitations, so conformance is mandatory.
Hermes	First-party hermes acp, hermes-acp, and python -m acp_adapter, with tests/docs.	Pin Hermes plus its ACP dependency, isolate HERMES_HOME, configure model/proxy headlessly, and verify its curated toolset. #569 is stale; Hermes is not blocked on ACP.

OpenCode is already an allowed AgentName in openhands-index-results; Pi and Hermes labels need to be added before publishing them there.

P0 — ship now

• Add OpenCode ACPProviderInfo with detection, pinned/default command, model selection, and required isolation; install the pinned binary and record its resolved version.
• Add acp-opencode consistently to benchmark agent types, _ACP_COMMANDS, credential/base-URL configuration, argument parsing, SDK/evaluation workflows, output metadata, and index label mappings.
• Register the proxy provider/model, select primary and auxiliary models, use env key substitution, isolate config/data/cache, and disable update/config drift. Fail on unexpected models. Keep file_secrets=(); interactive /connect credentials are irrelevant.
• Acceptance-test noninteractive startup, identity/version, requested model and exclusive proxy routing, read/edit/test, permissions, telemetry reconciliation, and clean second-run isolation.
• Run a five-instance OpenHands-versus-OpenCode smoke comparison before enabling the weekly workflow.

P1 — ordered follow-ups

• P1.1 — Pi integration: Add ACPProviderInfo/acp-pi; pin and record Pi plus pi-acp; isolate settings; verify pi --mode rpc; route only the declared model/key; add labels; pass P0 acceptance checks; add the third manifest cell.
• P1.2 — Hermes integration: Add ACPProviderInfo/acp-hermes; pin and record Hermes plus ACP dependency; isolate HERMES_HOME; route only the declared model/key; add labels; verify its curated toolset with P0 checks; add the fourth cell.
• P1.3 — Parameterized conformance suite: Generalize the P0 checks across startup, identity/version, routing, read/edit/test, permissions, telemetry, and clean-run isolation for current and future harnesses.
• P1.4 — Broader model compatibility: Add explicit harness/model compatibility blocks for Anthropic, OpenAI, Gemini, and additional open-model routes. Never create a blind Cartesian product containing unsupported or semantically different cells.
• P1.5 — Harness capability manifests: Version and report effective tools, skills, MCP servers, condensation, retries, permissions, maximum iterations, browser availability, and request routes so changes are visible rather than implicit.
• P1.6 — Adapter maturity gates: Track pi-acp limitations and upstream changes, add cancellation/resume/MCP/failure-path tests where relevant, and prevent an adapter regression from being misreported as a harness-quality regression.
• P1.7 — Additional ACP harnesses: Add new harnesses through the same conformance contract rather than bespoke workflow branches.
• P1.8 — Non-ACP adapters: Use Harbor or another terminal adapter only for harnesses that genuinely lack ACP, and label transport as part of the treatment rather than merging it silently with ACP results.

---

3. Paired runner and instance panel

Pairing supplies P0's power: both harnesses receive the same task snapshot and model, and the unit is a complete block. Prompts, tools, loop control, condensation, and verification are part of the harness treatment; model/backend or image drift, order, missing cells, and leaked configuration are confounds.

P0 — ship now

• Commit one versioned manifest with experiment ID, model/route, benchmark revision, image/verifier digests, instance IDs, harness versions/config hashes, budgets, timeouts, and two cells.
• Select one fixed 30–50-instance panel from one family with a recorded seed/cost estimate; pass the byte-identical list through the existing instance-selection path.
• Replace the current one-global-agent_type dispatch with manifest-driven matrix entries. Generate only declared (harness, model, config) cells rather than a blind agent_type × model cross-product.
• Randomize/interleave order so short provider incidents do not systematically affect one harness.
• Compare only blocks whose two cells match model, task/image/verifier revision, configuration, budget, and complete telemetry. Report incomplete blocks; never impute zero.
• Record every evaluation attempt and use intent-to-treat totals—including retry cost and time—as the P0 product-level result. Also report the final-attempt outcome so infrastructure failures remain inspectable.
• Keep the P0 panel fixed while the framework stabilizes. The report must clearly state that results apply to this model and panel, not every task or model.

P1 — ordered follow-ups

• P1.1 — Replicated variance pilot: Repeat a representative subset at least three times/harness to estimate stochastic variance and simulate power versus budget before claiming small differences.
• P1.2 — Multiple-model blocks: Run at least two shared models and estimate/report harness × model interaction. The GLM/Sonnet reversal shows that a single-model harness ranking does not generalize.
• P1.3 — Unbiased efficiency panel: Sample slowly from a declared target distribution with explicit family weights and committed seeds (ISO week where useful). Label equal-family versus natural-frequency weighting.
• P1.4 — Discrimination panel: Separately sample instances with historical resolve probability ~0.2–0.8 using {cost, resolved} priors in openhands-index-results. Do not use its biased accuracy as the population headline; retain inclusion probabilities for weighting.
• P1.5 — Fixed and sealed canaries: Combine public fixed canaries with sealed rotating anchors; repeated canaries are correlated observations, not fresh sample size.
• P1.6 — Multi-benchmark coverage: Extend manifests to SWE-bench, SWT-bench, Commit0, GAIA, SWE-bench Multimodal, and other families, preserving family-specific outcomes and avoiding unjustified pooling.
• P1.7 — Cost-aware sampling: Size panels from a dollar budget using historical per-instance cost while preserving declared sampling probabilities and target-population weights.

---

4. Deterministic paired comparison report

P0 reporting should be simple enough to audit from the per-instance table. Statistics are computed deterministically; an LLM does not calculate metrics or decide significance.

P0 metrics

Layer	Metric	P0 source and interpretation
Outcome	Resolved / benchmark score	Verifier within complete blocks; preserve available partial/subtest scores.
Spend	Total proxy spend	Intent-to-treat sum across every attempt. This is the primary resource measure.
Requests	Proxy requests	Ledger row count, split by success/failure/status. Do not call it iterations.
Tokens	Prompt, normalized fresh/cache input, output	Ledger plus selected provider semantics; unknown when incomparable.
Time	Wall clock and request latency	Separate harness runtime from provider latency where timestamps permit.
Reliability	Attempts, timeouts, infra failures, telemetry coverage	Report explicitly; never silently drop failures from efficiency summaries.

P0 — ship now

• Join both cells by complete block and emit a versioned comparison.json plus a human-readable Markdown report.
• Compare OpenHands with OpenCode on complete instances using a pairwise schema that accepts Pi/Hermes later without inventing a global winner.
• For outcomes, report wins/losses/ties, percentage-point difference, and exact McNemar on discordant pairs; do not treat repeated weeks as independent.
• For cost, time, requests, and tokens, report paired differences/ratios, mean, median, P90, paired-bootstrap intervals, and complete-pair n.
• Report both expected cost per attempted instance and portfolio cost per resolution (sum(cost) / sum(resolved)) with a paired block bootstrap. Keep cost conditional on success separate; these answer different questions.
• Include a sortable instance table for outcome and resource deltas with both trajectories and telemetry links.
• Header the report with harness/adapter/model versions, route, known tools/skills, browser, condenser, limits, budget, image, verifier, retries, and telemetry coverage.
• State the scope limitation in every report: equal verifier scores do not establish equal maintainability or architecture, and one model/panel does not establish a universal harness ranking.
• Add tests using synthetic paired data for ties, missing cells, zero successes, zero/unknown costs, heavy-tailed runaway instances, and bootstrap reproducibility.

P1 — ordered follow-ups

• P1.1 — Hierarchical repeated analysis: Once models/weeks/families/replicates exist, model harness, model, interaction, family, instance, replicate, and week effects instead of pooling tests.
• P1.2 — Cost-effectiveness reporting: Add paired net-benefit or Pareto-frontier views so a more expensive harness can be judged against any quality improvement rather than declaring cheapest automatically best.
• P1.3 — Target-population weighting: Publish weights for rotating/family/discrimination panels and both macro-family and deployment-weighted summaries where useful.
• P1.4 — Multiple-comparison control: Predeclare primary comparator/metrics or apply Holm/FDR/hierarchical testing as the number of harnesses, models, and outcomes grows.
• P1.5 — Partial quality and maintainability: Preserve subtests/deterministic signals; sample human review if architecture matters. Do not replace correctness with an unvalidated LLM judge.
• P1.6 — Behavioral summaries: Report tool-category proportions, repeated reads/commands, failures, edit churn, and verification per instance. Calls are nested and shares compositional; no raw-call χ².
• P1.7 — Historical observational baseline: Intersect existing OpenHands and alternative-agent instance results across SWE-bench, SWT-bench, Commit0, GAIA, and SWE-bench Multimodal to exercise the report and estimate rough variance. Label it observational because runs differ in date, harness/SDK version, and configuration; matching instance IDs alone does not make it a controlled pair.
• P1.8 — Index publication: Publish stable, sufficiently powered efficiency metrics to index.openhands.dev only after schema/version/config semantics are established. Keep noisy weekly diagnostics internal.
• P1.9 — Eval-monitor view: Add block-level harness comparison, telemetry coverage, trajectory links, and historical trends to eval-monitor.

---

5. Detection, confirmation, and weekly scheduling

P0 uses one cheap confirmation rather than sophisticated sequential inference; one noisy run cannot become a causal GitHub issue.

Keep two concepts separate:

• Competitive difference: OpenHands differs materially from OpenCode on the paired panel.
• Self-regression: OpenHands moved relative to its own previous version on fixed tasks.

P0 — ship now

• Add harness-watch.yml: weekly/manual, explicit manifest, budget/concurrency guards, and last-success marker. Reuse secret-scan-agent.yml operations, but keep execution/statistics deterministic.
• Configure practical alarms, initially ≥10-point resolved deficit or >1.5× cost/fresh-input/request ratio without a resolved advantage. Publish effect and uncertainty; these are operational triggers, not scientific truth.
• On breach, rerun only affected complete blocks once with fresh workspaces/keys and interleaved order.
• File/update a tracking issue only if complete telemetry confirms the direction; include both runs, n, effects, intervals, versions, and leading instances.
• Never call a non-significant accuracy result parity. With roughly 40 instances, P0 is designed to detect large resource gaps and only large outcome gaps.
• Track workflow/evaluation failures separately from harness losses so missing jobs cannot create a competitive alarm.

P1 — ordered follow-ups

• P1.1 — Sequential monitoring: Replace weekly p-value gates with posterior monitoring, e-values, or alpha-spending around predeclared practical margins.
• P1.2 — Robust self-regression baseline: Add paired change-point/EWMA-style monitoring over fixed and sealed canaries, accounting for repeated-instance correlation and model/provider drift.
• P1.3 — Empirical power budgeting: Simulate detectable effects from pilot paired differences/discordance. A three-point resolve difference likely needs hundreds to thousands of unique pairs.
• P1.4 — Adaptive confirmation: Rerun only influential/discordant blocks until a predefined evidence or budget boundary, rather than always repeating the same fixed count.
• P1.5 — Ownership and escalation policy: Define who owns the tracking issue, response time, suppression/acknowledgement behavior, and when a confirmed regression blocks a release.

---

6. Trajectory diagnosis and experiment handoff

The analysis agent inspects a deterministic dossier; it does not calculate statistics. Raw event indices do not align, ACP tool_kind is coarse, and raw_input may be absent. Preserve events and never treat raw tool-name frequency as quality.

High-value trajectory signals include:

• time/cost to relevant read, material edit, test/pass, final edit, and verification;
• verifier progress per request, token, dollar, and minute;
• relevant versus unnecessary file coverage; repeated reads/commands with unchanged output/state;
• edit churn/reversions, final-diff efficiency, failing-tool loops, and time to change strategy;
• testing after the final edit, stopping with known failures, context/condensation changes, crashes, retries, and recovery.

Any oracle patch or hidden-verifier information is used only after the run for analysis and is never exposed to the evaluated harness.

P0 — ship now

• Select outcome-discordant instances, then largest paired resource residuals; send only this dossier.
• Give a read-only agent tools to list pairs, view timelines/event ranges, final patches/test progress, and request/context curves.
• Require a structured output containing: hypothesis, alternative_hypotheses, category, supporting_evidence with instance/event IDs, counterevidence, estimated_impact, affected_instances, predicted_observable_change, and one single_variable_experiment.
• Ground every quantitative claim in report/tool output; the agent interprets behavior but never invents or recomputes statistics.
• Post only after confirmation, linking every cited trajectory location.
• Stop at an experiment proposal in P0. Do not automatically modify repositories or push branches.

P1 — ordered follow-ups

• P1.1 — Deterministic trajectory normalizer: Map native/ACP events to observe/read, edit, test, browse, delegate, plan, and finish while preserving names, mapping version, and confidence.
• P1.2 — Semantic milestone alignment: Align scan, discovery, edit, test, failure, recovery, pass, and verification rather than event number.
• P1.3 — Corpus-wide counterexample search: Give the diagnosis agent tools to find matched runs where a proposed behavior occurs without the outcome, or the outcome occurs without the behavior, before accepting a mechanism claim.
• P1.4 — Blinded analyst and skeptic: Anonymize labels, collect two independent hypotheses, and have a skeptic search counterevidence.
• P1.5 — Known-regression validation: Inject known changes such as redundant tool schemas, disabled condensation, repeated reads, a failing browser tool, or altered retries, then measure whether the analyzer identifies the mechanism with grounded evidence.
• P1.6 — Request/trajectory linkage: Join request shapes to milestones to attribute cost to loop length, context, tools, cache, or recovery.
• P1.7 — Refactor propose-harness: Make the broad results-URL agent accept an approved hypothesis and create one scoped branch/configuration; today it accepts no such object and may push several speculative branches.
• P1.8 — Closed-loop experiment evaluation: Run approved single-variable candidates immediately on the same paired panel/replicates, compare them with the parent configuration, and promote only changes whose predicted observable effect is reproduced.

---

What already exists

Capability	Location	Current state
ACP provider registry/config/isolation	software-agent-sdk/openhands-sdk/openhands/sdk/settings/acp_providers.py	Claude Code, Codex, Gemini; extend with OpenCode, then Pi/Hermes
Custom ACP escape hatch	ACPServerKind = Literal[..., "custom"] and acp_command	Good for prototyping, not accepted P0 benchmarking
ACP benchmark construction	benchmarks/utils/acp.py	Commands/env/setup for Claude, Codex, Gemini
Per-attempt key/scalar spend	benchmarks/utils/litellm_proxy.py, Evaluation._execute_single_attempt()	Extend to request rows and every attempt
Per-request ledger	LiteLLM production	Needs safe projection, pagination, reconciliation, coverage
ACP tracing	sdk/agent/acp_tracing.py	Outer turn plus tool events; behavior source, not internal request count
Conversation archives	Evaluation._capture_conversation_archive()	Available per instance
Historical instance outcomes	openhands-index-results	Useful observational prior; not a controlled pair
Efficiency summarizer	evaluation/eval-job/scripts/summarize_efficiency.py schema v6	Run-level; ACP request/iteration interpretation must be fixed
Scheduled workflow pattern	evaluation/.github/workflows/secret-scan-agent.yml	Reuse operations, not agentic statistics
Harness proposer	evaluation/propose-harness-action/	Broad prototype; requires P1 refactor before closed-loop use
Index agent labels	openhands-index-results/scripts/validate_schema.py	OpenCode exists; add Pi/Hermes when publishing

---

P0 definition of done

P0 is complete when one scheduled workflow can:

1. run OpenHands and OpenCode on the same committed 30–50-instance manifest and shared model;
2. prove the expected harness/model/config for every accepted cell;
3. capture safe, reconciled request-level LiteLLM telemetry for every attempt;
4. publish an auditable paired outcome/cost/request/token/time report with confidence intervals, n, configuration, and missingness;
5. confirm a threshold breach with a targeted rerun before filing it as a competitive difference;
6. attach a small, evidence-linked trajectory diagnosis and a human-reviewable single-variable experiment proposal.

No P1 feature is required.

---

Alternatives considered

• Harness metrics: Not authoritative because native and ACP granularity differs; retain for corroboration/diagnosis.
• Laminar token authority: No; ACP has one LLM span per outer turn. Keep traces for behavior.
• Raw spend logs: No; rows can contain prompts/responses/bodies. Export only a versioned allowlist.
• Recording sidecar per instance: Deferred. The existing ledger is enough for P0 totals. A proxy-boundary derived feature plane becomes valuable in P1 because the ledger projection alone cannot explain tool-schema/system/history composition.
• Harbor for the target harnesses: Rejected because OpenCode, Pi, and Hermes have ACP routes. P0 needs only the OpenCode route; keep Harbor/non-ACP adapters for genuinely incompatible future harnesses.
• Larger unpaired A/B: No; identical blocks remove task-difficulty variance at the same budget.
• A generic framework before real runs: Rejected. P0 supports exactly OpenHands and OpenCode on one model and one panel. Pi and Hermes then exercise the proven extension contract instead of delaying the first end-to-end result.
• Automatic branch creation in P0: Rejected. Observational trajectory diagnosis proposes an experiment; it does not establish causality or authorize code changes.

---

Known limits and decisions needed

• ~40 pairs can detect 2–3× resource gaps when variance is reasonable, not small accuracy differences or parity.
• Prompts, tools, skills, condensation, retries, and verification are the harness treatment; P0 records rather than equalizes them.
• One non-Anthropic model simplifies cache interpretation, but conclusions remain model-specific; multi-model lanes are P1.
• The ledger sees proxy requests/spend, not every logical decision.
• Provider/proxy failures count toward reliability but must be classified separately from reasoning differences.

Team decisions required before implementation:

1. the P0 standing model;
2. the P0 benchmark family and fixed panel budget;
3. practical confirmation thresholds;
4. the owner of the tracking issue and weekly triage;
5. whether P0 reports remain internal or are public before replicated/multi-model P1 results exist.

Grounded initially in software-agent-sdk@910115b3e, benchmarks@f60e4ed1, evaluation@d869853, openhands-index-results@3015ac6, litellm@8a3a0f4, and the production LiteLLM configuration. External harness status rechecked on 2026-08-25 against current OpenCode, pi-acp / earendil-works/pi#175, and the current Hermes ACP implementation.

Related SDK work: #4623 (OpenHands prompt composition), #4083 (tool search / deferred schema loading), and #4267 (existing OpenCode custom-ACP report).

[github-actions]

⚠️ Not yet ready-for-dev

This issue does not yet meet the readiness criteria, so the
ready-for-dev label has not been applied (or has been removed).
The criteria are type-specific:

Bug reports (bug label, see .github/ISSUE_TEMPLATE/bug_template.yml):

• The ### Actual Behavior section must describe reproducible SDK steps
  and include a command such as python, pytest, uv, or pip.
• An ### Acceptance Criteria section with at least one checklist item (- [ ] …).

Enhancements (enhancement label, see .github/ISSUE_TEMPLATE/feature_request.yml):

• A ### Desired Behavior section.
• An ### Acceptance Criteria section with at least one checklist item.

What is missing:

• Add a ### Desired Behavior section describing the behavior you want.
• Add an ### Acceptance Criteria section with testable checklist items.

Edit the issue to address the gaps and the ready-for-dev label will be
applied automatically.

Note: Section headings require exact ### matches from the templates.
If a missing type label is reported, no action is needed from external
contributors — maintainers will add the label during triage.

This is an automated check — no AI was used to generate this comment.