docs: benchmark v3 across change kind, tooling, and model

[janicduplessis]

Description

The first v3 draft still could not support its own headline: expo run:ios can stay alive with Metro, Codex JSON events have no timestamps, the native plist edit did not force compilation, cache and runner state were asymmetric, and its post-hoc n=3 repeats selected the fastest noisy screening results. An adversarial pass found 38 concrete protocol defects spanning measurement, fixture pins, cache state, runner isolation, cost formulas, and statistics.

This revision makes v3 an executable, staged experiment. The native comparison is pre-registered and confirmatory; JavaScript model breadth is explicitly descriptive. v2 remains historical context only because its runner and timing setup differ.

Solution

docs/agent-benchmark-v3.md now specifies:

• A coordinator watcher that measures dispatch to a newly installed, live app process independently of the agent's build command, plus a separate bundle/change-proof gate captured at the first app-alive observation.
• A compiled Swift native edit carrying the run id, exact prompts for all four arm/variant combinations, and a derived Trailhead fixture that removes the Stim cue and normalizes the Pod lock to the pinned CocoaPods version.
• Exact tool, OS, simulator, model, reasoning, and service-tier pins; same-volume clone requirements; scoped Xcode state; fixed Metro state; fresh simulators; disk, load, and thermal gates; and explicit Stim-only portable artifact state as the feature under test.
• Disposable runner homes with zero skills in control and exactly the shipped Stim skill in the Stim arm, plugins disabled, empty Claude MCP configuration, Task/subagent prohibition, and stim/agent-device quarantine.
• Correct runner-specific token formulas and raw vectors. Dollar figures are secondary notional list costs; Codex tier verification is marked unavailable when CLI evidence omits the returned tier.
• Three excluded calibration pairs, a sample size fixed before confirmatory timing, balanced order, a 40-second precision target, immutable invalid-run records, and no aggregate claims from n=1 screening.

The machine-local Codex pilot helper at stim-bench-coordinator/v3-20260902/ implements preparation, preflight, runner smokes, one-run dispatch, idempotent collection, explicit one-run cleanup, and reporting. It does not yet implement Claude dispatch, balanced scheduling, sample-size calculation, or the final manifest allowlist required before calibration and confirmatory stages. Its golden contains pre- and post-normalization iOS artifacts; preflight resolves the exact post-normalization 7d04e2...-debug-sim key before each run and verifies that the Stim shim targets the pinned CLI checkout. Live profile smokes proved the control runner sees no skills and cannot resolve Stim, while the Stim runner sees only Stim and resolves that pinned shim.

The OpenAI recorder pilot completed valid JavaScript pairs on gpt-5.6-sol and gpt-5.6-luna. Luna measured 138.872s with Stim versus 406.534s in control (2.93x), with changed-bundle proof at 150.924s versus 478.739s (3.17x); Sol measured 122.393s versus 510.613s (4.17x). These n=1 pairs are descriptive, not statistically conclusive. The pool code was pinned but not exercised because every timed run created and deleted a fresh simulator; the result measures Stim orchestration and portable-cache behavior, not parked-simulator adoption. Requested priority service is recorded, but Codex did not return tier evidence, so reports label all cost estimates tier-unverified. Two invalid Luna control attempts and the earlier legacy pair remain preserved for harness audit rather than silently reclassified. The collector now recovers exact worktree evidence from stamped events after cleanup and requires it for validity.

Test plan

• pnpm run format:check
• pnpm run lint
• pnpm run build
• pnpm run typecheck
• pnpm run knip (existing pod configuration hint only)
• pnpm test (83 files, 3,329 tests)
• pnpm run test:e2e (20 tests)
• pnpm run test:runtime
• Real golden build with Xcode 26.6: 1,394 compile actions, pre- and post-normalization artifacts retained, the current 7d04e2...-debug-sim key resolved by preflight, app installed and launched on a fresh iPhone 17 / iOS 26.5 simulator, then worktree/device cleanup.
• Strict Codex profile smokes with gpt-5.6-luna: both returned turn.completed; control exposed zero skills and no Stim binary, Stim exposed only the Stim skill and pinned shim.
• Valid Sol and Luna OpenAI recorder pairs with app-alive proof, captured-bundle proof, exact cleanup, command/token/cost collection, and idempotent recollection. Invalid Luna control attempts remained recorded and were excluded from the matched result.
• Official direct model pages checked on 2026-09-03 for Sol and Luna token prices.

Fixes #281