Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ ClawCodex keeps your request prefix **byte-stable**, so DeepSeek's prompt cache

# The pi-style minimal harness, built in

### Terminal-Bench 2.1 A/B (`fix-git` + `pypi-server`), same model, same wheel, one flag apart: **2/2 solved in both modes** — nano at **$0.0042 vs $0.0186 total (4.4× cheaper)**, up to 5.2× less input, faster on both.
### Full Terminal-Bench 2.1, same model, head-to-head with the [pi harness](https://pi.dev) (its own TB setup, vision+websearch matched): **nano 64/89 (71.9%) at $1.31 vs pi 63/89 (70.8%) at $2.01** — equal-or-better score, **35% cheaper**, k=1.

Six tools, a **~2K-token** fixed payload (vs ~16K default), zero per-turn injections, /eco on.
`clawcodex --nano -p "<task>"` ports the pi harness's edit ladder (multi-edit + fuzzy match),
Expand Down
5 changes: 4 additions & 1 deletion docs/nano.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,10 @@ Measured on terminal-bench 2.1 (`fix-git` + `pypi-server`,
deepseek-v4-flash, identical wheel, k=1): **both tasks solved in both
modes (reward 1.0)** — nano at **$0.0042 total vs $0.0186 (4.4×
cheaper)**, e.g. fix-git: 54.8K vs 282.6K input tokens, $0.00165 vs
$0.01015, 75 s vs 128 s. Full tables: `eval/harbor/RUN_NANO_TB21.md`.
$0.01015, 75 s vs 128 s. On the full 89-task suite head-to-head with
the pi harness (both with vision+websearch, deepseek-v4-flash at max
thinking, k=1): **nano 64/89 ($1.31) vs pi 63/89 ($2.01)**. Full
tables: `eval/harbor/RUN_NANO_TB21.md`.

## What nano does differently

Expand Down
19 changes: 19 additions & 0 deletions eval/harbor/RUN_NANO_TB21.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,25 @@ Aggregate: **2/2 pass in both modes; nano $0.0042 vs default $0.0186 —
4.4× cheaper** at equal quality. Nano's fix-git trajectory was 10 focused
Bash calls, zero tool errors.

## Full-suite nano vs pi (89 tasks, deepseek-v4-flash, max thinking, k=1)

Same wheel lineage, pi ran its TB extension (vision+websearch); the nano
arm matched it (`NANO_VISION=openai:gpt-5.6-luna NANO_WEBSEARCH=1`):

| | nano `tb21-nano-flash-max-2` | pi `tb21-pi-flash-max-2` |
|---|---|---|
| solved | **64/89 (71.9%)** | 63/89 (70.8%) |
| cost | **$1.31** | $2.01 |
| runtime | 5h38m | 5h54m |

The pre-fix nano baseline (`tb21-nano-flash-max-1`, no vision/websearch,
before the care/verification guidelines and staleness soft-refresh) was
60/89 at $1.02. Six of the eleven tasks v2 gained are directly
attributable to shipped fixes (vision ×1, websearch ×1, care ×1,
timeout-race mechanics ×3); seven v1 passes flipped red on k=1
variance (build/training-time jitter). At this k, treat 64-vs-63 as
parity-to-slight-edge on score with a durable ~35% cost advantage.

Nano sends six tools and a ~2K-token fixed payload (vs ~16K default), no
per-turn injections, /eco on. A trivial live A/B outside Harbor
(deepseek-v4-pro, write-and-verify, both solved in 4 turns) showed the
Expand Down
Loading