diff --git a/README.md b/README.md index 690cec47..be8b0c42 100644 --- a/README.md +++ b/README.md @@ -79,7 +79,7 @@ ClawCodex keeps your request prefix **byte-stable**, so DeepSeek's prompt cache # The pi-style minimal harness, built in -### Terminal-Bench 2.1 A/B (`fix-git` + `pypi-server`), same model, same wheel, one flag apart: **2/2 solved in both modes** — nano at **$0.0042 vs $0.0186 total (4.4× cheaper)**, up to 5.2× less input, faster on both. +### Full Terminal-Bench 2.1, same model, head-to-head with the [pi harness](https://pi.dev) (its own TB setup, vision+websearch matched): **nano 64/89 (71.9%) at $1.31 vs pi 63/89 (70.8%) at $2.01** — equal-or-better score, **35% cheaper**, k=1. Six tools, a **~2K-token** fixed payload (vs ~16K default), zero per-turn injections, /eco on. `clawcodex --nano -p ""` ports the pi harness's edit ladder (multi-edit + fuzzy match), diff --git a/docs/nano.md b/docs/nano.md index bc05a7ad..7a7f2b19 100644 --- a/docs/nano.md +++ b/docs/nano.md @@ -36,7 +36,10 @@ Measured on terminal-bench 2.1 (`fix-git` + `pypi-server`, deepseek-v4-flash, identical wheel, k=1): **both tasks solved in both modes (reward 1.0)** — nano at **$0.0042 total vs $0.0186 (4.4× cheaper)**, e.g. fix-git: 54.8K vs 282.6K input tokens, $0.00165 vs -$0.01015, 75 s vs 128 s. Full tables: `eval/harbor/RUN_NANO_TB21.md`. +$0.01015, 75 s vs 128 s. On the full 89-task suite head-to-head with +the pi harness (both with vision+websearch, deepseek-v4-flash at max +thinking, k=1): **nano 64/89 ($1.31) vs pi 63/89 ($2.01)**. Full +tables: `eval/harbor/RUN_NANO_TB21.md`. ## What nano does differently diff --git a/eval/harbor/RUN_NANO_TB21.md b/eval/harbor/RUN_NANO_TB21.md index 27f0b5a0..9c2d96e6 100644 --- a/eval/harbor/RUN_NANO_TB21.md +++ b/eval/harbor/RUN_NANO_TB21.md @@ -63,6 +63,25 @@ Aggregate: **2/2 pass in both modes; nano $0.0042 vs default $0.0186 — 4.4× cheaper** at equal quality. Nano's fix-git trajectory was 10 focused Bash calls, zero tool errors. +## Full-suite nano vs pi (89 tasks, deepseek-v4-flash, max thinking, k=1) + +Same wheel lineage, pi ran its TB extension (vision+websearch); the nano +arm matched it (`NANO_VISION=openai:gpt-5.6-luna NANO_WEBSEARCH=1`): + +| | nano `tb21-nano-flash-max-2` | pi `tb21-pi-flash-max-2` | +|---|---|---| +| solved | **64/89 (71.9%)** | 63/89 (70.8%) | +| cost | **$1.31** | $2.01 | +| runtime | 5h38m | 5h54m | + +The pre-fix nano baseline (`tb21-nano-flash-max-1`, no vision/websearch, +before the care/verification guidelines and staleness soft-refresh) was +60/89 at $1.02. Six of the eleven tasks v2 gained are directly +attributable to shipped fixes (vision ×1, websearch ×1, care ×1, +timeout-race mechanics ×3); seven v1 passes flipped red on k=1 +variance (build/training-time jitter). At this k, treat 64-vs-63 as +parity-to-slight-edge on score with a durable ~35% cost advantage. + Nano sends six tools and a ~2K-token fixed payload (vs ~16K default), no per-turn injections, /eco on. A trivial live A/B outside Harbor (deepseek-v4-pro, write-and-verify, both solved in 4 turns) showed the