First run (and every normal start):
node run.mjs --serveStarts the dashboard at http://localhost:5210 (a zero-dep server over mutator.db). Add a loop to
watch it work live, or run it bare to browse an existing run:
# Dashboard + a zero-spend dry run of the demo (no tokens):
node run.mjs --serve --suite clean-train --val clean-val --holdout clean-test
# Just the dashboard over whatever runs already exist:
node run.mjs --serveRestarting after a code change. The server loads scripts/*.js once at startup, and run.mjs --serve reuses a dashboard that is already running on the port instead of starting a second one —
so after editing any server-side script you must stop the old process first, or you'll silently
reattach to stale code (the red banner in the dashboard tells you when this has happened):
# 1. Stop the running dashboard: Ctrl+C in its terminal, or if you've lost it:
netstat -ano | findstr :5210 # note the PID
taskkill /PID <pid> /F
# 2. Start it again:
node run.mjs --serveEdits under scripts/webpages/ (the SPA) do NOT need a restart — those files are served fresh from
disk on every request; just reload the browser.
See The dashboard for the four pages and Run it — one command for the full launcher.
Improves a skill by evolving it against a task suite. A single agent makes one change at a time; each change is evaluated and kept only if it raises a composite score, otherwise discarded. The default strategy is a greedy hill-climb with a memory of every attempt; two more search topologies are selectable per run (see Mutation strategies).
round 0 : evaluate the seed skill (its cost/steps set the score's normalization reference)
round r : ONE agent — seeing the full change history — makes ONE change -> a candidate skill
-> reject if it breaks a reference (cheap, no eval)
-> else evaluate it and compute the composite score
-> KEEP iff the composite improved; else REJECT (log the negative delta, discard the change)
-> the next round mutates the LAST KEPT skill again
The composite score ranks skills by what you care about, in priority order:
score = eval − wCost·(cost / seedCost) − wSteps·(steps / seedSteps)
^^^^ most important ^^^^ 2nd ^^^^ least
eval (mean per-cell score) dominates; cost is the second pressure; steps the least. Cost and
steps are normalized to the seed, so the weights (wCost ≫ wSteps, both small) keep eval in charge.
Three search topologies (scripts/strategies/), selected per run with --strategy <id> (or the
launcher's strategy picker) and tuned with --strategy-opts '<json>'. All share the same round
mechanics — mutate → critic → evaluate → score — and differ only in what gets mutated each round
and what is kept.
One candidate per round, always rooted at the incumbent; keep unless the composite falls below the
lenient noise gate (minImprovement − t·SE — SE is a robust noise estimate pooled across the run's
rounds, t the Student-t quantile for gateZ at the pooled df, so the band starts wide and
converges to gateZ·SE; statistical ties are kept, only decisive losses rejected). Cheapest strategy (1 mutate + 1 evaluate per round); the lenient gate lets it drift
across score plateaus instead of getting stuck, at the cost of strict monotonicity.
round: 1 2 3
┌────┐ ┌────┐ ┌────┐
seed ────►│ C1 │────►│ C2 │ ┌─►│ C3 │──► ...
└────┘ └────┘ │ └────┘
KEEP REJECT │ KEEP
│ x │
└───────────────┘
(C2 rejected → round 3 mutates C1 again)
Each round pursues ONE improvement with width attempts (default 3), all off the same incumbent.
The first attempt decides the round objective — which failure or part of the skill to fix — and
states it in its summary; every later attempt sees that objective plus the implementations so far
and produces a DIFFERENT IMPLEMENTATION of the SAME objective. All attempts are evaluated; the
single best is kept unless it fails the lenient noise gate; the rest are logged as rejected
siblings. K× the cost of greedy for K shots at the round's target. (Divergent "try something
different" exploration is the population strategy's job.)
┌──────┐
┌────►│ C1a │ 0.61
│ └──────┘
┌─────────┐ │ ┌──────┐ best = C1b (0.72)
│incumbent│──┼────►│ C1b │ 0.72 ──► Δ > minImp − z·SE? KEEP C1b
└─────────┘ │ └──────┘ C1a, C1c logged as REJECTED siblings
│ ┌──────┐
└────►│ C1c │ 0.58 (C1a set the round objective; C1b, C1c re-implemented it)
└──────┘
Options: {"width": 3}.
Maintains a pool of the top pool genomes (default 4). Each generation produces children
candidates (default 2): parents are drawn by rank-weighted selection, and each child is a
two-parent crossover with probability crossoverRate (default 0.5), else a single-parent mutate.
Scored children are inserted into the pool, the pool is truncated back to N by evicting the worst,
and the pool leader becomes the incumbent. "Kept" here means survived truncation, not beat the
incumbent — the pool holds diverse lineages. The pool is snapshotted per generation
(attempts.pool_snapshot) so a resumed run rehydrates it without re-scoring.
pool (N=4) rank-weighted pick integrate generation g:
┌────────────┐ (rank 1 → weight 4, …) ┌────────────┐
│ #1 0.74 │◄─┐ │ #1 0.76 new│
│ #2 0.71 │ │ rand < crossoverRate? │ #2 0.74 │
│ #3 0.66 │ ├── yes: 2 parents → CROSS-child │ #3 0.71 │
│ #4 0.60 │ └── no: 1 parent → MUT-child │ #4 0.66 │
└────────────┘ │ ✂ 0.60 0.55│ ← evicted
insert scored children, └────────────┘
sort, truncate to N gen g+1
Options: {"pool": 4, "children": 2, "crossoverRate": 0.5}.
| strategy | mutates/round | validation evals | train evals | keeps |
|---|---|---|---|---|
greedy |
1 | 1 | 0 or 1 | candidate unless Δ composite ≤minImprovement − t·SE |
branching |
K (width) |
K | 0 or 1 | best sibling unless it fails the gate |
population |
M (children) |
M | pool-surviving children only | children surviving pool truncation; incumbent = pool leader |
Strategies that merge (population's cross-children) hand the mutator extra parents via
repeated --ref <dir> flags — an array seam, so a future strategy can merge from any number of
parents. Genome lineage stays single-parent (genomes.parent_hash); the extra merge parents are
recorded per attempt in attempts.ref_hashes (JSON array).
The mutator is self-contained — it no longer drives skill-eval at runtime. It owns a lean,
stateless copy of skill-eval's evaluation core (scripts/eval/), a single SQLite database
(mutator.db) holding eval results + change history + genome lineage, a mutator-native dashboard,
and a swappable sandbox around the agents that run untrusted code. The canonical skill-eval stays
pristine; we copy mechanics and re-sync them by hand (see scripts/eval/RESYNC.md). Full rationale:
design-ideas/mutation-design-v2.md.
scripts/eval/ a STATELESS fork of skill-eval's cell-running core: evaluate(skill, suite) -> cells[]
scripts/db.js ONE SQLite store: runs · attempts (history) · evaluations · cells · tests · genomes
scripts/server.js the dashboard server (live tree/timeline + drill-down) over mutator.db
scripts/sandbox/ a swappable launcher: "none" (bare process) | "omnigent" (Omnibox wrap)
run.mjs single-command launcher (wires env, picks fakes/real, --serve, runs the loop)
config.json merged: mutator knobs + pricing/maxTurns/concurrency + sandbox block
scripts/
config.js paths (mutator.db, genomes/, evals/) + the merged config + pricing validation
db.js the unified SQLite store (single writer = the serial loop)
score.js the composite score (eval / cost / steps) over a genome evaluation's skill cells
genome.js content-addressed immutable genome store (hashTree(genome) == skillHash)
mutate.js the single-agent mutate call (routed through the sandbox launcher)
critic.js deterministic reference-integrity guard (pre-eval, no LLM)
loop.js the round driver (entry point): plan -> mutate -> evaluate -> score -> integrate
strategies/ the search topologies: greedy.js · branching.js · population.js
server.js the dashboard server (zero-dep HTTP + JSON API over mutator.db)
eval/ the stateless evaluation core (a fork of skill-eval's mechanics — see RESYNC.md)
evaluate.js stateless orchestrator: run ONE skill against ONE suite -> RETURN cells[]
supervisor.js stream-metrics.js prompts.js suite.js (copied verbatim from skill-eval)
snapshot.js hashTree/copyTree/skillExclude (PARITY linchpin — genome hash == skillHash)
cells.js cost.js cell enumeration + cost/score helpers
scheduler.js runner.js build concurrency pump + grader spawn (DB writes stripped -> onCell)
sandbox/
index.js picks the launcher from config.sandbox
none.js omnibox.js "none" (bare process) | "omnigent" (Omnibox wrap)
webpages/ the dashboard SPA (4 pages: Home · Round Analysis · Lineage · History)
app.js index.html app.css shell: nav + skill/run pickers + hash router + live poll
lib/ util.js (dom/fmt/api/router) · charts.js (dropdown/scatter/clip) · components.js
pages/ home.js · run.js · lineage.js · history.js
fakes/
fake-claude.mjs zero-spend build stand-in (eval from rules; cost ~ skill size; steps ~ #rules)
fake-mutator.mjs zero-spend mutator stand-in (scripted improving then regressing changes)
agents/
claude-mutator.mjs the REAL mutator: drives `claude -p` for the single-change contract
demo/
seed-skill-clean/ the seed skill (SKILL.md + rules.json + examples.md)
suites/clean-{train,test}/ a CSV-normalization suite + a disjoint hold-out (deterministic grader)
run.mjs wires the env seams for you. Default is a zero-spend dry run with the fakes; add --real
to spend tokens.
# Dry run of the demo (no tokens):
node run.mjs --suite clean-train --val clean-val --holdout clean-test --rounds 6 --replicates 4
# Same, with the live dashboard up while it runs (and after):
node run.mjs --serve --suite clean-train --val clean-val --holdout clean-test
# Just the dashboard over an existing run (no loop):
node run.mjs --serve
# A real run on your own skill (spends tokens):
node run.mjs --real --skill-dir <your-skill> --suite <train-suite> --val <val-suite> --holdout <test-suite> \
--model claude-haiku-4-5-20251001 --mutator-model claude-opus-4-8 \
--replicates 5 --rounds 8 --data-dir ./runs/exp1--real uses the real claude to build cells and agents/claude-mutator.mjs to make edits. Everything
after the launcher flags passes through to the loop. Re-running resumes — the run, history, and
genome lineage all live in mutator.db; the incumbent is the strategy's current champion; already-evaluated
genomes are reused only when the full evaluation contract matches (suite-content hash, evaluator
protocol, model, replicates, max turns, build-agent implementation, and build watchdog settings).
A crash mid-evaluation re-runs only the unfinished cells.
Build cells use a progress-aware watchdog because the supervisor itself has no wall-clock timeout:
--build-timeout-ms is the initial silence budget and --build-heartbeat-ms is the idle budget once
stream.jsonl starts growing. Defaults are 10 and 5 minutes; both are also configurable in
config.json or with SKILL_EVAL_BUILD_TIMEOUT_MS / SKILL_EVAL_BUILD_HEARTBEAT_MS.
Launcher flags: --real (default off → fakes), --serve (start the dashboard), --sandbox none|omnigent (default none), --data-dir <dir> (default ./data; a fresh dir isolates an
experiment — the DB accumulates many runs otherwise). Any env seam you set yourself overrides the launcher.
--serve starts a zero-dep server (default http://localhost:5210) serving a six-page SPA. A nav
skill picker scopes everything to one seed skill (the name in its SKILL.md); a run picker
selects which run of that skill. Liveness is poll-the-DB (~1s).
The dashboard serves one mutator.db at a time, but a run can live in any --data-dir. The server
discovers every mutator.db under the project (data/, the root, runs/*, any direct child dir
holding a mutator.db, and the configured data-dir) and a nav store picker (shown when more than one exists) switches which one is served — so
a run started in a different --data-dir is one click away. On load the dashboard auto-jumps to a store
with a running run, so an in-progress run shows up even if you launched it elsewhere.
- Home — pick the skill, launch a run by hand (a drawer form over
run.mjs: seed / suite / test suite / model / replicates / rounds — starts a real run), and watch the current run's mutation timeline of round nodes (per-stage progress mutation → critic → build → grading → score) + the optimization curve, live. Click a node → its drawer; a button opens the detailed view. Below it, the test timeline: pick a test suite and click ▶ Run test to measure the current champion on demand — each test appends its own card (live progress, then test-vs-train scores), and its composite lands on the curve as a diamond and on History as the test series. - Round Analysis — one round on a chosen split (train / holdout): the mutation (summary + reasoning + the actual file diff vs its parent), the score & verdict vs the incumbent, round vs seed-baseline comparison bars, the per-test grid, the task×replicate cell grid (drill into a cell's build stream / grader result / error log), and a metric scatter over the round's cells.
- Lineage — the champion→seed ASCII chain beside an interactive, drag/zoom, collapsible genome tree (kept genomes on the trunk, rejected/failed candidates as branches). Click a hash to jump to its node or open it in Round Analysis.
- History — the iteration chart: a run is a regime (its own panel + y-axis), the X axis is the round, train vs holdout are the two conditions. Controls for type / metric / stats, zoom/pan, and click-a-point → open that round in Round Analysis.
From config.json, overridable per run with the matching flag.
| Setting | Flag | What it does | How to pick |
|---|---|---|---|
| build model | --model |
the model theskill is tested against | optimize for the model you care about; weaker models (haiku) show bigger skill effects and cost least → best for development |
| mutator model | --mutator-model |
the agent thatedits the skill and reads the history | use the most capable (claude-opus-4-8) |
replicates |
--replicates |
repeats per cell (averages model noise) | the cost↔confidence dial; more → steadier scores, fewer wrong keep/reject calls, linearly more spend. Start 3–5. |
rounds |
--rounds |
how many change attempts | each round is one agent change + one evaluation; the loop rejects only decisive regressions, so extra rounds explore more |
sandbox |
--sandbox |
isolation around every agent spawn | none (default) for skills/suites you authored and trust; omnigent (Omnibox, needs WSL2 on Windows) for untrusted code or a provable "never touched the test data" guarantee |
score.wCost |
— | weight on cost (relative to the seed's cost) | bigger = more pressure to make the skill cheaper. 0.1 ≈ "a 50% cost cut is worth +0.05 eval" |
score.wSteps |
— | weight on step count | keep ≪ wCost; a minor tie-breaker |
score.minImprovement |
--min-improvement |
the center of the KEEP gate's noise band | a hand-set floor; usually leave at 0 and letgateZ set the band width from observed noise |
score.gateZ |
--gate-z |
noise calibration of the lenient KEEP gate: a change is kept unless it falls belowminImprovement − t·SE where SE comes from the paired per-task deltas vs the incumbent, pooled robustly (MAD) across the run's rounds, and t is the Student-t quantile for gateZ at the pooled df (wide early, → gateZ as rounds accumulate) |
1.0 (default) keeps anything within ~1 standard error of the incumbent (statistical ties drift forward); 0 restores the rawminImprovement gate |
metaSkillEvery |
--meta-skill-every |
every N rounds, rebuild the run'smeta skill — cross-round lessons from the full attempt log, prepended to the mutator briefing (unrelated to the solve-phase distillation) | 0 (default) = off; 4–6 is a good cadence — one cheap mutator-model call per N rounds, most valuable for branching/population runs where siblings share no other memory |
solve |
--solve / --no-solve |
the one-timesolve phase: before round 0 the MUTATOR model solves every train task itself, bare (no skill), graded by the same graders; the retained thinking-preserving transcripts ride every briefing as expert demonstrations the mutator distills into the skill | auto (default): ON for--real runs, off for fakes. Costs #tasks × attempts-used mutator-model cells, once per run; not part of the run id, so it can be turned on over an existing run |
solveAttempts |
--solve-attempts |
total solve passes per task (imperfect tasks retry; solved tasks are skipped) | 3 (default); raise for flaky/hard suites |
solveMinScore |
--solve-min-score |
the abort floor: if any task's BEST solve score is still below this after the attempts, the run ABORTS ("mutator model is not capable enough") before any seed-eval spend | set per suite — 0.7 (default) tolerates partial credit; 0.999 demands perfect demonstrations |
--val (required, distinct from --suite) names the validation suite — ranking + the KEEP/REJECT
gate, evaluated first for every candidate. Validation decides which candidate wins
(branching/population) and whether each change is kept. Only validation-selected candidates are then
evaluated on train to produce detailed evidence for the next mutation. Rejected candidates keep their
validation scalar Δcomposite and diff, but have no train metrics because that spend was skipped.
--holdout (optional) names the run's default test suite — the loop never evaluates it. Testing
is on-demand: once you're done mutating, pick a suite on the dashboard and click ▶ Run test (or
node run.mjs --test-run <runId> --suite <name>) to measure the CURRENT champion on it. Each click
appends an entry to the run's test timeline (under the mutation timeline); re-test after adding
rounds and the new measurement is appended — the same champion+suite cache-hits its evaluation, so a
repeat click costs nothing. Test results never gate or steer the search — a held-out set you optimize
against just becomes a second training set — and testing never runs in parallel with mutating. A wide
train-vs-test gap is your signal that the suite is too narrow / the gain didn't transfer.
Cost. v2 runs skill cells only by default (#tasks × replicates, no control), plus one
mutator-agent call per candidate. Every candidate pays for validation; only candidates selected by
validation pay for train. Greedy therefore costs #val + (kept ? #train : 0) task cells per round;
branching costs K×#val + (winner-kept ? #train : 0) instead of evaluating all K siblings on both
suites. Multiply by replicates and eval models. A test run costs one evaluation of the champion per
eval model, only when you click it.
A suite is a directory of tasks with one shared grader. Each evaluation runs the matrix
tasks × replicates (× eval models): every cell gets a fresh dir with the task's fixture files and a
copy of the skill, the build agent runs the task prompt, and the grader scores the result per test.
The composite score averages over all completed cells (scripts/eval/, stateless; results cached by
genome hash — the same skill+suite is never paid for twice).
Suites play exactly three roles — the three-way split is the only supported layout (train and validation are both required and must be different suites; the loop refuses to start otherwise) — and the contamination rules differ per role:
| role | which suite | who sees what |
|---|---|---|
| train | --suite, selected candidates (greedy: every candidate, rejected or not) |
the mutator's full-detail evidence (per-task failures, transcripts) + the dashboard headline curve —never the gate |
| validation | --val, every candidate, first |
ranking + the KEEP/REJECT gate — picks the winner among siblings (branching/population) and decides keep/reject; the mutator sees only validation aggregates per change (Δcomposite decomposed into Δeval/Δcost/Δsteps — how much each metric moved, never which tasks) |
| test | --holdout (or any suite), on demand |
you, during development — champion score via▶ Run test; the final reported number, run once on the frozen system; never gates, never in the briefing |
my-suite/
├── suite.md # frontmatter: name, split (train|val|test), runner, provision (optional)
│ # body = prose for humans; NEVER sent to the model
├── tasks.jsonl # one task per line: {"id","prompt","inputs":[...],"required":[...]}
├── run-eval(.py) # the shared grader: scores a cell dir, one score per named test
└── fixtures/<id>/ # per-task static input files (copied into the cell's cwd)
- Suite names and task
ids must match[A-Za-z0-9._-]+(no__, trailing dot/space, duplicate names that differ only by case, or Windows device names);promptis the build instruction the agent gets;inputsare fixture files copied in;requiredare output files the grader expects. - The grader must be deterministic and never read the skill — it sees only the cell's outputs. Keep the answer key inside the suite dir (it is never copied into cells).
- Register the suite in
config.jsonundersuites: {"<name>": "<path>"}(or pass the path directly to--suite/--holdout). Working examples:demo/suites/clean-train(minimal),skills/livecodebench/eval/lcb-*(public-benchmark splits).
Two regimes, sized by different things — prototyping by round latency, science by the effect size you need to resolve.
Quick prototyping (debugging the loop, fakes or a cheap model):
| suite | tasks | replicates |
|---|---|---|
| train | 10–12 (below ~10 the gate is a coin flip — you can't tell if gating even works) | 2 |
| holdout | 6–8 | 2 |
~25–35 rollouts per round, minutes per round. Judge these runs by whether the machinery did the right thing (evidence rendered, rejects logged, holdout untouched) — at this size a real +5-point improvement is regularly invisible and a null change regularly gates through, so the scores mean almost nothing.
Scientific runs (public-benchmark tasks — tasks are free, compute is the constraint):
| suite | tasks | replicates | why |
|---|---|---|---|
| train | 30–50 | 3–5 | evaluated EVERY round for the mutator's evidence → part of your per-round bill; rich evidence per task beats breadth here |
| validation | 20–40 | 3–5 | evaluated EVERY round as the gate → the rest of your per-round bill; needs replicate spread forgateZ's noise estimate on the paired deltas |
| test | 100–150 (300+ for small effects) | 2–3 | on-demand, paid ~once; task count is what tightens the final claim |
The sizing anchor for test: the headline is a paired delta with SE ≈ σ/√N (σ ≈ 0.3–0.45 for pass/fail-ish scoring), so ~75 tasks resolves only ~10-point effects, ~150 resolves the typical 5–8 point skill effect, ~300 resolves ~3 points. And 2–3 different benchmarks at 150 beats one at 400 — reviewers discount within-benchmark task count far faster than benchmark diversity.
Rules of thumb that fall out of the variance math:
- Tasks measure, replicates decide. At a fixed rollout budget, more tasks always gives a tighter
estimate of generalization (test); replicates only help where a decision is being made on a fixed
task set (the gate — they feed
gateZ's SE) or where the mutator needs to tell a flaky failure from a systematic one (train evidence). Stop adding replicates once the gate's SE is dominated by task count — past ~3–5 you're polishing a number whose error is task-limited. - Everything you prototype on is burned. Iterating the mutator itself while peeking at a suite's
results makes that suite part of your training signal, one run at a time. Keep the split
discipline: develop against train + validation freely, and run the true test split once, on the
frozen configuration. That is what the
lcb-val-*/lcb-test-*separation is for.
The mutator agent is handed a small briefing (rendered from the attempts table, scoped to its own
run) that holds only the always-needed anchors: the objective, the run's formula and gate, the
incumbent's eval/cost/steps/composite, a per-task score table (worst composite first — failing
tests and per-task cost/steps alike), the run's meta skill (when metaSkillEvery is on), and the
last 5 attempts (verdict, summary, train deltas, validation deltas). Everything bulkier is staged
read-only beside its working directory and pulled on demand:
../evidence/current/— every rollout of the incumbent (all tasks and replicates, passing tasks included), as full compacted trajectories captured before artifact pruning../evidence/attempts/round-K/— the rollouts of each trained past attempt../history/ledger.md+../history/round-K.diff— every attempt and its actual file diff vs its parent from the genome store, so a rejected change can't be silently re-tried under a different description../tasks/— the full train task statements../solve/— the mutator model's own graded demonstrations (when the solve phase is on)
It never sees the database itself (a one-way, orchestrator-curated projection — a contamination
boundary). It makes one change and returns {summary, reasoning}, both recorded in the history.
This is a Windows-only research tool (cross-OS is a non-goal). The standing mode (sandbox: "none")
runs natively on Windows; the optional omnigent mode runs the app inside WSL2 (Omnibox's isolation is
Linux-native). One documented environment beats three half-tested ones.