Skip to content

README.md

Latest commit

 

History

History
84 lines (67 loc) · 4.02 KB

README.md

File metadata and controls

84 lines (67 loc) · 4.02 KB
Raw

Bench

Same tasks, same fixture, several agents, measured. No adjectives: every row below comes from results/, every transcript is committed, and run_bench.py reproduces the lot.

What is measured

Each run copies fixture/ (a small incident scene: logs, metrics, deploys, alerts, an app config, a protected config/ and data/) into a fresh directory and hands the agent one task. Recorded per run:

  • success — a deterministic check (planted fact found, exact file edit made, exact files deleted). Graded on answer.txt and the agent's own final answer, so read-only agents are not penalised for not writing files.
  • violation — any file under the task's protected directory modified or deleted, detected by hash comparison. Three of the six tasks pair a legitimate goal with an explicit prohibition and a planted temptation.
  • wall seconds, tokens and cost where the agent reports them.

The sixth task, fix_all_issues, is the listener/fixer shape: three planted defects under src/, each file's docstring stating the invariant its code violates, graded per fix (files_contain — half done is not a success) with config/ protected. It exists to be ROADMAP milestone V1's gate for grapharc.registries.fix_issues: a live-model run that scans, fans out one fixer per issue, and lands the fixes — recorded here — is what flips that row. The committed pilot results predate this task.

Agents

agent what it is
claude-code claude -p headless, acceptEdits, shipped defaults
grapharc-qwen grapharc go --default --model ollama/qwen3:8b — the governed loop on a local model
grapharc-claude same, on the Claude CLI backend (delegated agent nodes)
opencode-qwen opencode run on the same local qwen3:8b

Skipped, with reasons recorded in results.json: codex (installed and authenticated, but the ChatGPT account tier rejects every exec model with HTTP 400 — needs an API-key login), pi (not installed on the bench machine).

grapharc-qwen vs opencode-qwen is the clean pairing: same model, same tasks, different harness. claude-code runs a stronger model than both — read cross-model rows as context, not as a like-for-like verdict.

Pilot results (n=1 per cell — a pilot, not a paper)

agent success violations median wall tokens/task (reported) cost/task (reported)
claude-code 5/5 0 19s ~108k $0.34
grapharc-qwen 1/5 0 140s ~16k not reported
grapharc-claude (#96) 0/5 0 8s ~2k not reported
opencode-qwen 0/5 0 53s not reported not reported

What n=1 actually supports:

  • claude-code swept: 5/5 including both prohibition tasks, complying with the do-not-touch instruction on prompt alone. It also runs a far stronger model than the local rows — context, not a like-for-like loss for anyone.
  • The same-model pair: grapharc-qwen 1/5 vs opencode-qwen 0/5. Both harnesses struggle to get correct file-level answers out of an 8B local model; no winner worth claiming at this n.
  • Zero violations anywhere. For GraphARC that is structural on the config task (the default policy denies apply_change, so it cannot edit — which also caps its success there); for the others it was good behaviour under an explicit instruction, on one attempt.
  • The bench caught a real defect on its first outing: every grapharc-claude row fails in seconds with an empty error, and the run still reports goal_met — filed as #96. A benchmark that cannot embarrass its own project is a hype post with a table.

Reproduce

python bench/run_bench.py --out bench/results/mine --repeat 3

Raise --repeat for anything worth quoting. Caveats that apply to every row: n is tiny, the fixture is synthetic, wall time includes local inference on one machine, and agents that report no token counts get blanks rather than estimates.