fix: accept Claude native trajectories

[vincentkoc]

What does this PR do?

Marks Claude Code native trajectories as real harness events so completed Claude runs remain eligible for native aggregation.

Why?

The native runner already converts Claude Code stream-json output into trajectory.json, but the harness allowlist omitted claude-code. Aggregation therefore classified otherwise valid Claude runs as trajectory_unavailable.

Changes

• add claude-code to the real-trajectory harness allowlist
• assert every pinned native harness reports real_harness_events

Live proof

A native Claude Code control run completed four tasks with four result files, four valid completed results, and no infrastructure failures. Aggregation classified every trajectory as real; the run remained ineligible only because it was intentionally marked as an exploratory four-task subset.

• coverage: 4/4
• score: 0.75
• exact passes: 3
• trajectory complete: true
• trajectory status: real for all four tasks
• infrastructure failures: 0
• exclusion reason: exploratory_subset

Tests

• python -m pytest -q passes locally (444 passed, 5 skipped)
• python -m ruff check clawbench app.py scripts tests passes locally
• live Claude Code native run aggregates as real harness events

[clawsweeper]

Codex review: blocked before merge. Reviewed September 4, 2026, 2:59 PM ET / 18:59 UTC.

ClawSweeper review

What this changes

The PR classifies Claude Code native trajectories as real harness events and adds coverage for the native-harness classification.

Merge readiness

⛔ Blocked before merge - 4 items remain

Keep open: current main still excludes Claude Code from real native trajectories despite its existing validated stream reconstruction. This member-authored PR is a narrow fix, but its exact head conflicts with main and its benchmark-eligibility policy needs explicit approval.

Priority: P2
Reviewed head: ac1c974c12749dddb5d49af734561cec59910c65
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	The patch is small and consistent with current trace reconstruction, with rebase and benchmark-policy approval remaining before merge.
Proof confidence	🌊 off-meta tidepool	Not applicable: The PR body reports a completed four-task Claude control run with real aggregation status, and the external-contributor proof gate does not apply because the author association is MEMBER.
Patch quality	🐚 platinum hermit (4/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Not applicable	Not applicable: The PR body reports a completed four-task Claude control run with real aggregation status, and the external-contributor proof gate does not apply because the author association is MEMBER.
Evidence reviewed	5 items	Current main excludes Claude Code: Current main includes Claude Code among pinned harnesses but omits it from the real-trajectory allowlist, so its manifest mode is unsupported. Current main reconstructs validated Claude traces: The Claude Code parser writes a real trajectory only after a successful terminal event and records observed-model validation; current tests exercise that reconstructed trajectory and canonical model identity. Eligibility depends on the changed mode: Native aggregation requires real_harness_events in the run manifest as well as real per-task trajectories; otherwise it excludes the run for trajectory unavailability.
Findings	None	None.
Security	None	None.

How this fits together

ShellBench’s native evaluator reconstructs each harness’s trace and aggregates completed results into eligibility and scores. This classification determines whether Claude Code’s reconstructed trace can count in that aggregation.

flowchart LR
  A[Claude Code stream output] --> B[Trajectory reconstruction]
  B --> C[Trajectory classification]
  C --> D[Native result aggregation]
  D --> E[Eligibility and scores]

Loading

Decision needed

Question	Recommendation
Should validated Claude Code stream-json reconstruction qualify as real trajectory evidence for native benchmark aggregation?	Approve and rebase: Approve Claude’s validated reconstruction as comparable evidence, resolve the two conflicts, and rerun focused native-evaluation coverage.

Why: The code already reconstructs and validates Claude traces, but changing the allowlist changes which runs may enter benchmark results; that comparability policy cannot be inferred from implementation alone.

Before merge

• Resolve merge risk (P1) - Accepting the new allowlist makes reconstructed Claude Code runs eligible benchmark evidence; maintainers should explicitly confirm that this trace fidelity meets the project’s comparison standard.
• Resolve merge risk (P1) - The exact head has merge conflicts in both changed files, so landing it without a rebase would discard current-main changes.
• Complete next step (P2) - Resolve the two current-main merge conflicts and obtain explicit approval that validated Claude Code stream-json reconstruction may qualify as real benchmark trajectory evidence.
• Resolve maintainer decision - Resolve the maintainer decision shown above before merge.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Focused delta	production +1, tests +12, 2 files affected	The implementation is narrowly scoped to one eligibility allowlist and one regression assertion.

Merge-risk options

Maintainer options:

1. Approve the evidence contract after rebase (recommended)
   Resolve the current-main conflicts and explicitly approve validated Claude stream reconstruction before it can change benchmark eligibility.
2. Preserve the exclusion
   Keep Claude Code output diagnostic-only if maintainers do not yet consider its reconstruction comparable benchmark evidence.

Technical review

Best possible solution:

Rebase the focused classification and regression test onto current main, retain the existing Claude stream validation, and merge only after maintainers approve its use as comparable benchmark evidence.

Do we have a high-confidence way to reproduce the issue?

Yes, source inspection gives a high-confidence path: a completed Claude Code run can reconstruct as real, yet current main writes an unsupported manifest mode that aggregation excludes.

Is this the best way to solve the issue?

Yes, conditionally: adding Claude Code to the existing allowlist is the narrowest repair once maintainers approve its reconstructed stream as comparable benchmark evidence.

AGENTS.md: not found in the target repository.

Codex review notes: model internal, reasoning high; reviewed against c1a79f731541.

Labels

Label justifications:

• P2: This is a bounded native-evaluation eligibility correction with limited operational blast radius.
• merge-risk: 🚨 other: Merging changes which reconstructed runs may enter benchmark aggregation, a research-validity risk not covered by the specific compatibility categories.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The PR body reports a completed four-task Claude control run with real aggregation status, and the external-contributor proof gate does not apply because the author association is MEMBER.

Evidence

What I checked:

• Current main excludes Claude Code: Current main includes Claude Code among pinned harnesses but omits it from the real-trajectory allowlist, so its manifest mode is unsupported. (scripts/native_eval/models.py:103, c1a79f731541)
• Current main reconstructs validated Claude traces: The Claude Code parser writes a real trajectory only after a successful terminal event and records observed-model validation; current tests exercise that reconstructed trajectory and canonical model identity. (scripts/native_eval/harness_trajectories.py:316, c1a79f731541)
• Eligibility depends on the changed mode: Native aggregation requires real_harness_events in the run manifest as well as real per-task trajectories; otherwise it excludes the run for trajectory unavailability. (scripts/native_eval/aggregate.py:993, c1a79f731541)
• Exact branch cannot merge cleanly: A read-only three-way merge reports conflicts in both changed files against current main; resolving them must retain main’s later native-evaluation changes. (scripts/native_eval/models.py:103, ac1c974c1274)
• Native-evaluation provenance: History records Vincent Koc’s merged native-evaluation work that established the runner and its later trace reconstruction, while later main work also changed the conflicting surfaces. (scripts/native_eval/models.py:94, 884dd1bb5511)

Likely related people:

• vincentkoc: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Resolve the two current-main merge conflicts and rerun focused native-evaluation coverage.
• Obtain explicit approval that validated Claude Code stream-json reconstruction is comparable real benchmark evidence.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (66 earlier review cycles; latest 8 shown)

• reviewed 2026-08-09T19:45:59.678Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-09T22:01:21.887Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-11T23:10:51.821Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-12T01:22:50.827Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-14T10:11:27.098Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-14T11:15:30.965Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-21T23:55:18.903Z sha ac1c974 :: needs maintainer review before merge. :: none
• reviewed 2026-08-29T02:02:08.794Z sha ac1c974 :: needs maintainer review before merge. :: none