From cb1efec9f52b532a1440ca12518de9776675c2c2 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Mon, 20 Jul 2026 00:02:31 +0200 Subject: [PATCH 1/8] =?UTF-8?q?fix:=20BA-15=20broken-arbiter=20blockers=20?= =?UTF-8?q?=E2=80=94=20a=20faulty=20sensor/verifier=20is=20named,=20never?= =?UTF-8?q?=20coerced=20into=20a=20model=20failure?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adopter feedback (forbidden-zone audit of the close chain) validated against shipped code with a deterministic pre-fix POC (poc/ba15-broken-sensor.mjs): Sensor seam (refineLeaf.sensor): - a THROWING sensor fell through to a bare {incomplete, best:null} — byte-identical to a provider death, the model's unjudged work destroyed - a MALFORMED verdict ({}, 'ok', {ok:true}) read as pass:false with critique:null, so refine burned every attempt re-sending the PLAIN task (zero feedback), then surfaced converged-shaped as {result, verdict:{}} Verifier seam (caller opts.evaluate; found by a follow-up probe that falsified the first-pass "no live laundering path" assessment): - a THROWING caller verifier CRASHED the whole recurse() run plain-worker (and laundered to a bare {incomplete} under refineLeaf) - a GARBAGE return rode out converged-shaped as {result, verdict:{}} Fix (recurse.js only, refine.js untouched): both caller arbiters are wrapped — a non-Halt throw or a malformed return (neither boolean `pass` nor valid tri-state `status`) stops at the FIRST broken close and returns a labeled {incomplete, blocker:'broken-sensor'|'broken-verifier'} + receipts.blockerDetail, with `best` preserving the unjudged work (BA-5). One verifyOrBlock helper covers all five dispatch paths; the default Evaluator rubric path is never labeled (provider-class faults — narrowest guard); HaltError passes through clean on both seams. `return await` is load-bearing at the call sites (the POC's Halt-control arm caught an un-awaited returned promise escaping the try's catch mid-refactor). +13 mutation-checked tests; suite 756 pass / 0 fail; typecheck clean. New public surfaces (MINOR): blocker 'broken-sensor'|'broken-verifier', receipts.blockerDetail. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf --- CHANGELOG.md | 17 ++++ CLAUDE.md | 2 +- README.md | 2 +- bareagent.context.md | 2 + docs/01-product/RLM_PRD.md | 42 +++++++++ poc/ba15-broken-sensor.mjs | 183 +++++++++++++++++++++++++++++++++++++ src/recurse.js | 158 ++++++++++++++++++++++++++++---- test/recurse.test.js | 160 ++++++++++++++++++++++++++++++++ 8 files changed, 545 insertions(+), 21 deletions(-) create mode 100644 poc/ba15-broken-sensor.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index 720455a..91a2213 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,23 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog](https://keepachangelog.com/). Versioning: [SemVer](https://semver.org/). +## [Unreleased] + +### Fixed + +- **A broken `refineLeaf.sensor` is now a named blocker, never coerced into a model failure (BA-15).** + + Adopter feedback (a forbidden-zone audit of the close chain) claimed — and a pre-fix POC confirmed in shipped code (`poc/ba15-broken-sensor.mjs`, deterministic/offline) — two silent coercions at the sensor seam: + + - **A sensor that THROWS** (the caller's test runner crashed: ENOENT, harness syntax error) fell through to the bare `{ incomplete, best: null }` — byte-identical to a provider death, with the model's unjudged work destroyed. "Didn't judge" collapsed into "model failed." + - **A sensor that returns a MALFORMED verdict** (`{}`, a string, `{ok:true}` — anything without a boolean `pass` or valid tri-state `status`) read as `pass:false` with `critique:null`, so every retry re-sent the PLAIN task with **zero feedback**, burned all `maxIterations` against the broken arbiter, then surfaced as a **converged-shaped** `{ result, verdict: {} }` — an honest-looking model non-recovery pinned on a broken judge. + + Now the sensor call is wrapped (recurse.js, the seam only — `refine.js` untouched): a non-Halt throw or malformed return **stops the loop at the FIRST broken close** and returns a labeled `{ incomplete, blocker: 'broken-sensor' }` plus `receipts.blockerDetail` (what the sensor did — threw with which message, or which malformed shape), with `best` preserving the model's last attempt (BA-5: the work was never judged, not judged-and-failed). Extends the BA-11 `blocker` taxonomy; a `HaltError` thrown by the sensor stays a clean governance halt; both documented verdict shapes (`{pass: boolean}` / `{status}`) behave byte-identically to before. A hung sensor remains the caller's responsibility (run untrusted checks in an isolated child process with a timeout — documented, no timeout knob). + + **The same fault class was live at the verify slot** (found by a follow-up probe after the sensor fix — the initial "the Evaluator leg has no live laundering path" assessment was wrong for recurse's *caller* seam): a **throwing** caller `opts.evaluate` **crashed the whole `recurse()` run** as an uncaught exception on the plain-worker path (and laundered to a bare `{ incomplete }` under `refineLeaf`), while a **garbage** return rode out **converged-shaped** as `{ result, verdict: {} }`. The caller verifier is now wrapped identically (`blocker: 'broken-verifier'`, first broken close stops, `best` preserves the unjudged result) via one `verifyOrBlock` helper across all five dispatch paths (worker / refineLeaf / scan / partition / fanout). The **default Evaluator rubric path is deliberately not labeled** (it constructs well-formed Verdicts; its failures are provider-class faults — narrowest guard). En route the POC's Halt-control arm caught a real regression in the refactor itself: `return verifyOrBlock(...)` inside a `try` let a verifier `HaltError` escape the catch (a promise returned un-awaited exits the try before settling) — fixed with `return await` + a dedicated regression test. + + New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` on `RecurseResult`/receipts; `receipts.blockerDetail`. +13 mutation-checked tests (throw / malformed shapes / provider-fault distinguishability / Halt passthrough on both seams / valid-shape controls / fanout-path coverage / the `return await` guard); full suite 756 pass / 0 fail, typecheck clean. Pre-fix evidence: `poc/ba15-broken-sensor.mjs`. + ## [0.30.0] - 2026-07-16 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index f98e639..1e0ed66 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,7 +14,7 @@ Node.js >= 18, pure JS + JSDoc, `node:test` for testing. Flat `src/` layout with | Evaluator | src/evaluator.js | Output-side judge (mirror of Planner): `evaluate(goal, result, criteria) → Verdict` by `predicate` (deterministic, no tokens), `rubric` (LLM), or `agentic` (tool-running critic). Tri-state `Verdict {status: satisfied/needs_revision/failed, pass (derived), score, critique, suggestions}` — mirrors Anthropic Managed Agents "Outcomes" (eval-assist F1). The rubric path runs an **isolated adversarial grader** (separate context window + harsh independent system prompt, never the generator's transcript) — that isolation, not a feedback knob, defeats the self-evaluation trap (A1). Optional `contract` (definition of done, A3) is graded against, not the loose goal. Judge tokens forward to the gate via `onLlmResult` (budget visibility). Composes AROUND a Loop (never inside `loop.js`). Built flagged-and-deletable, no POC (D11 — self-grading blind spot already settled by GAN + litectx R-S8). Third criteria type `agentic` (D9/A2): an **isolated tool-running critic** — a fresh Loop with scoped functional tools (`barebrowse`/`baremobile`, set on the Evaluator or per-call `opts.tools`) and a harsh independent prompt that **exercises** the live artifact (clicks, reads console/network) rather than reading text; each critic round forwards to `onLlmResult` (re-tagged `evaluate`), a governance HaltError re-throws clean, and the critic loop is gate-bounded via `opts.policy` | | refine | src/refine.js | Thin Loop-agnostic generate→evaluate→regenerate loop (port of Outcomes' iterate→grade→revise). Drives caller-supplied `attempt`/`evaluate` until satisfied, terminal `failed`, or `maxIterations` (real bound = bareguard maxTurns/budget). Threads `critique` forward (fresh-feedback default D6/A1 — anchoring on a failed answer defeats the verifier) + shared `contract` to both sides. HaltError propagates clean | | remember | src/remember.js | eval-assist F5 — the consolidation pass (formerly the PRD's parked "future glue", now built). `remember(spans, {provider, store, contract?, metadata?, ctx?, onLlmResult?}) → {facts, spans}`: one cheap LLM pass per harvested stash span distills DURABLE facts (drops ephemeral chatter, superseded values, unanswered questions; deduped within the call), each written `kind:'fact'` (authoritative — not overridable) through the **generic four-verb `Store` socket** — backend-agnostic (JsonFile/SQLite/litectx/custom), **no litectx coupling** (the rejected alt — reading litectx's promotion count — reaches past the socket into one backend). Composes AROUND a Loop (never inside `loop.js`); optional, flagged-and-deletable. Budget visibility: each pass forwards `usage` to `onLlmResult` (mirror of Evaluator); provider HaltError propagates clean. Gives `metrics.memory.facts` its honest writer via `ctx.recordMemoryOp('facts')` — **disjoint from `stored`** (writes WITHOUT ctx → a fact counts once). Distiller prompt live-validated on the real Anthropic wire (`poc/f5-remember-distill.mjs`): recall / faithfulness-to-correction (final 60 not stale 100) / discrimination (chatter→0). Loop never imports this file | -| recurse | src/recurse.js (+ src/recurse-prompts.js, src/recurse-synthesize.js) | RLM `recurse()` primitive — the decompose→fan-out→verify→synthesize entry point (RLM_PRD §10 steps 3, 4, 5 & 7: NB-1 glue + NB-4 + NB-5 + NB-3 reduce + NB-2 forced fan-out + RC-5 retrieval modes). `recurse(task, ctx, opts) → {result, verdict, receipts}` (convergence) / `{incomplete, best, receipts}` (guard exhaustion or dead worker — RC-9, never a faked pass). "B-shell with an A-tool" (§4.2): a deterministic shell offers the model an in-process `spawn_child` A-tool (§4.5 POC default — fresh `Loop`/fresh window per self-call, ~0ms/node vs ≥90ms fork) it MAY use to decompose. `assessComplexity` is a HINT not a gate: routes `simple`→single-shot, flags `critical`→forced adversarial verify (the `isCritical` safety floor). Termination is bareguard's via `ctx.depth`→`policy` (NO 2nd guard layer, §6); `opts.maxDepth` is only the topology knob that stops OFFERING spawn (`maxDepth=1`⇒flat fan-out). **⚠️ COST IS OPEN BY DESIGN — no intrinsic total-work cap:** the Family-A default can spawn up to a Loop's `HARD_ROUND_LIMIT` (100) children/level × `maxDepth`, so TOKEN/$ spend compounds and is bounded ONLY by the gate (POC `rlm-defer2-history-overflow.mjs` saw a weak model do 40–117 calls in one run). **Run with bareguard (or ANY token/USD cap); ungoverned it CAN burn tokens.** Forced `fanout`/`partition` ARE bounded (deterministic count + concurrency cap); the open path is the model-driven default; `maxDepth:1` is the local brake without a gate. NB-4 capability-scrub (RC-11/12, verify-closed step 6): deeper workers get fewer tools (spawn dropped EXACTLY at cap) + a depth-aware prompt (none→"prefer direct action"→"deepest level/cannot delegate/honest-incomplete"); tool set monotone (child ⊆ parent); `maxDepth=1`⇒flat (no nesting). Boundary is cap-INCLUSIVE (`depth>=maxDepth`⇒deepest suffix), mutation-proved across a 0→1→2 nesting. Copy-on-return (RC-2) by construction: child sees only its subtask (fresh msg array), only its result string crosses back. **Audit-safe ctx (F16/BA-1):** `recurse` threads its wiring blob (incl. the live `provider`) down the tree for self-calls, but a wired gate records the per-run ctx VERBATIM as `_ctx` — so `auditSafeCtx()` STRIPS the provider (and thus `apiKey`) from the ctx at every governance boundary (worker `Loop.run`, both pre-wave `ctx.policy` checkpoints, `scanCount`); the worker still gets the provider as a Loop constructor option (`Loop.run` never reads `ctx.provider`). Pairs with bareguard's secret-redaction (BG-1) as defense-in-depth. **`opts.context` (BA-9/F19):** an optional read-only working-context string (paths/cwd) prepended to EVERY worker's task message as a `Working context:` block + forwarded to the Planner as `info` (path-aware slices) + shown to the verifier (neutral FACTS, not a stance) — fixes a sliced worker that can't LOCATE its artifact (relayfact probe-04: workers guessed `.`/`~`/`/tmp`); CARRIES DOWN via `forChild` like `persona` but distinct (persona=privileged SYSTEM stance, context=USER-message run-state, so callers stop laundering cwd through persona); live 0/3→3/3 (`poc/ba9-context-thread.mjs`). **`opts.refineLeaf` (BA-8/F17, opt-in):** turns a DEFINITE leaf (`!canSpawn`) into a bounded generate→sense→regenerate loop reusing `refine.js` — `{sensor, maxIterations?, temperatures?, rejectedBuffer?}`; `sensor`=caller's DETERMINISTIC close (test/compile/lint, not a model judge), its GAP fed FRESH into the next attempt with **ESCALATING temperature** (default `[0.2,0.7,1.0]` — load-bearing WITHOUT memory: flat temp recovered 0/5 as a weak model regenerated identical wrong code ignoring crisp feedback; escalation → 0/5→2-3/5, `poc/ba8-leaf-refine.mjs`). **`rejectedBuffer` (BA-14):** a SkillOpt-shaped rejected-attempt buffer — surface the model's OWN prior failed attempts VERBATIM ("write something structurally DIFFERENT") via `refine.js`'s newly-threaded `attempt` `history`. DIRECTED diversity that ANTAGONIZES temperature's RANDOM diversity: temperature MONOTONICALLY degrades the buffer (`poc/ba14b`: flat 0.2/0.7/1.0 → 100/70/50%), so when it engages the temp is HELD flat at `temps[0]`, never escalated — REFINING BA-10's "temperature is secondary" to "with a buffer, temperature is harmful". Trigger: `true` force-on / `false` force-off / UNSET = ADAPTIVE (engage only once a temperature was DROPPED — a temp-fixed model where escalation is inert and the buffer is the sole lever, `poc/ba14`: 50%→100%). Live verify-shipped on real temp-fixed `claude-sonnet-5` (`poc/ba14-verify-shipped.mjs`: engaged 6/6, ledger reached the wire; efficacy an honest NULL — the lift is a WEAK-model/fixation phenomenon, cost-neutral otherwise, so ADAPTIVE-not-always-on). `receipts.refineLeaf.rejectedBuffer` reports engagement. Gate-bounded per attempt; honest non-recovery (`receipts.refineLeaf.passed=false`), never faked; carries down (engages at the leaves); error-keyed `recall` stays the CALLER's `opts.tools` (litectx-agnostic). Shipped-vs-POC smoke `poc/ba89-shipped-smoke.mjs`. **Governance deny-spin (BA-11/F35):** a worker whose Loop short-circuits a consecutive-policy-deny spin (the Loop `maxConsecutiveDenials` guard, default 3) surfaces as a LABELED `{incomplete, blocker:'governance-deny'}` (on BOTH the plain-worker and `refineLeaf` paths + `receipts.blocker`) — so a caller distinguishes a governance block (widen scope / re-gate / escalate) from a model failure, instead of the probe-16 burn ($1 to the cap, sensor never reached). Live verify-shipped `poc/ba11-verify-shipped.mjs` (haiku, stopped at 3 denials). `Evaluator` fills the verify slot (`opts.evaluate` overrides); **NB-3 reduce** (src/recurse-synthesize.js, `synthesize(task, results, opts)`): `opts.synthesize` = a **function** (deterministic code-reduce over child `results` — the §9.1 aggregation path, LLM arithmetic is the weak link) OR a strategy string `'concat'`/`'merge'` (lossless join / isolated Loop-driven subjective merge); reduce fires only on a node that spawned (a leaf keeps its own answer; each level reduces ITS children); default (Family-A) = the parent model's own closing-turn synthesis. **RC-5 retrieval modes BUILT (step 7, src/recurse-retrieval.js)** — context is a HANDLE routed by TASK SHAPE (§9.2.1), not one default: count/"all"→`opts.retrieval:'scan'` (DEFAULT when `opts.corpus` present) = scan every slice + LLM-judge + **CODE-count** over a **generic array slice-source** (`opts.corpus={id,text}[]`, NOT litectx), `opts.window`≈8 (the one calibrated number, recall knee) + `opts.passes`=2 (deterministic rotation union, NO RNG → RC-3 holds); needle→`'search'` (opt-in) = `ctx.litectx.recall` handle tool, embeddings on, `fact`/`episode`, capped `KNN_K=8` — CANNOT count; exact→`'exact'` (opt-in) = embeddings-free code-side AND-term filter tool. **Per-query face `'tools'` (opt-in, step-7 follow-on DONE):** offers `scan_count` (the new exported `buildScanTool`, the complete code-count path) ALONGSIDE `search_memory`/`exact_match` so a Family-A worker routes PER SUB-QUERY by the tool DESCRIPTIONS (scan="use for how many/all/count"; search="never count") — a MIXED needle+count task keeps both; completeness guard does NOT fire for `'tools'` (scan always offered, complete path always reachable). RC-9 floor (dead window→"count is a floor") + unwrapped `HaltError` hold at the tool boundary; async source materialized once (cached); `forChild` strips it (children don't inherit). Live-validated `poc/rlm-scan-as-tool.mjs` (worker picked search-for-needle + scan-for-count, code-known truth, neutral prompt). Completeness-contract guard UPGRADES a capped `search` on a "how many/all" ask to scan (upgrade-ONLY, never silent downgrade). RC-2 (judge matches only ids it was shown — a hallucinated id is intersected away) + RC-9 (dead window/empty corpus → `{incomplete, missingSlices}`, never a fabricated `count:0`) hold; gate-Halt mid-scan → clean incomplete. **Backend split:** litectx has no exhaustive rank-free enumerate verb today (every read FTS-gated), so scan reads the array slice-source; the "corpus already in litectx" case + the data-driven *width* count wait on the litectx `enumerate` verb (spec handed off: docs/01-product/prd.md) and drop in behind the same socket with ZERO recurse changes. `opts.tools` remain caller-supplied handle tools. Measured on AG News; naive "search→count" DROPPED. Backward-compatible: no corpus + no retrieval ⇒ unchanged (Family A). NB-5 decomposition policy + few-shot in recurse-prompts.js. **Family-B forced fan-out (`opts.count`/`mode:'fanout'`, NB-2) BUILT (step 5):** deterministic count→`Planner`(`{count}` seam = exactly N independent steps)→`runPlan`(waves)→NB-3 reduce(`'concat'` default)→verify; count = `opts.count` (overrides) else the **live-calibrated** tier map medium/complex/critical→**2/4/6** (an overridable default, not a discovered constant — knee location is topology). Each slice is a fresh-window child (MAY itself self-decompose under Family A); RC-9 dead-slice→`missingSlices` (no survivor-sum); planner/child/reduce/verify `HaltError`→clean incomplete. **Metered + pre-wave checkpoint (§6):** the decomposition call forwards usage (`onLlmResult`) so it's not invisible to the budget, and the cheap decompose RESOLVES the unknown→known width, after which `ctx.policy('recurse_fanout', {count, depth})` runs BEFORE the worker wave — a `HaltError` (budget / ≥80% HITL pause, bareguard's decision) → clean incomplete before any worker spends (bounds the concurrent burst to zero); a plain deny is advisory (allowlist-safe). **`opts.count` = fixed/semantic FLOOR;** the **data-driven *width* PARTITION** is BUILT (litectx 0.26 `enumerate`): `recurse(task, ctx, {mode:'partition', corpus, workerBudget?})` measures the corpus → `width=max(floor, ⌈size/workerBudget⌉)` (capped by guards + size) parallel scan-workers → union-count; a distinct path from `recurseFanout`'s semantic split, same FLOOR. `litectxCorpus(litectx,{kind})` is the resident slice-source (paginates `enumerate`); `opts.corpus` also accepts an in-hand array OR any async `()=>Promise` (recurse stays litectx-agnostic). Pre-wave `recurse_partition` checkpoint + RC-9 hold. `enumerate` was verified vs the spec DoD before building (`poc/litectx-enumerate-verify.mjs`). Composes AROUND a Loop (never imported by loop.js); validated by 71 mutation-checked offline tests + a live `--fanout` smoke + **step-8 e2e replay (verify-shipped-vs-POC, `poc/rlm-step8-shipped-replay.mjs`): the SHIPPED scan reproduces §9.2.1 on AG News (recall 0.88/precision 0.97/err 9%, code-count===matchedIds) — a real regression caught en route (a colon-ambiguous id prompt under-recalled 0.93→0.29; fixed with ` => ` display + verbatim-copy)** | +| recurse | src/recurse.js (+ src/recurse-prompts.js, src/recurse-synthesize.js) | RLM `recurse()` primitive — the decompose→fan-out→verify→synthesize entry point (RLM_PRD §10 steps 3, 4, 5 & 7: NB-1 glue + NB-4 + NB-5 + NB-3 reduce + NB-2 forced fan-out + RC-5 retrieval modes). `recurse(task, ctx, opts) → {result, verdict, receipts}` (convergence) / `{incomplete, best, receipts}` (guard exhaustion or dead worker — RC-9, never a faked pass). "B-shell with an A-tool" (§4.2): a deterministic shell offers the model an in-process `spawn_child` A-tool (§4.5 POC default — fresh `Loop`/fresh window per self-call, ~0ms/node vs ≥90ms fork) it MAY use to decompose. `assessComplexity` is a HINT not a gate: routes `simple`→single-shot, flags `critical`→forced adversarial verify (the `isCritical` safety floor). Termination is bareguard's via `ctx.depth`→`policy` (NO 2nd guard layer, §6); `opts.maxDepth` is only the topology knob that stops OFFERING spawn (`maxDepth=1`⇒flat fan-out). **⚠️ COST IS OPEN BY DESIGN — no intrinsic total-work cap:** the Family-A default can spawn up to a Loop's `HARD_ROUND_LIMIT` (100) children/level × `maxDepth`, so TOKEN/$ spend compounds and is bounded ONLY by the gate (POC `rlm-defer2-history-overflow.mjs` saw a weak model do 40–117 calls in one run). **Run with bareguard (or ANY token/USD cap); ungoverned it CAN burn tokens.** Forced `fanout`/`partition` ARE bounded (deterministic count + concurrency cap); the open path is the model-driven default; `maxDepth:1` is the local brake without a gate. NB-4 capability-scrub (RC-11/12, verify-closed step 6): deeper workers get fewer tools (spawn dropped EXACTLY at cap) + a depth-aware prompt (none→"prefer direct action"→"deepest level/cannot delegate/honest-incomplete"); tool set monotone (child ⊆ parent); `maxDepth=1`⇒flat (no nesting). Boundary is cap-INCLUSIVE (`depth>=maxDepth`⇒deepest suffix), mutation-proved across a 0→1→2 nesting. Copy-on-return (RC-2) by construction: child sees only its subtask (fresh msg array), only its result string crosses back. **Audit-safe ctx (F16/BA-1):** `recurse` threads its wiring blob (incl. the live `provider`) down the tree for self-calls, but a wired gate records the per-run ctx VERBATIM as `_ctx` — so `auditSafeCtx()` STRIPS the provider (and thus `apiKey`) from the ctx at every governance boundary (worker `Loop.run`, both pre-wave `ctx.policy` checkpoints, `scanCount`); the worker still gets the provider as a Loop constructor option (`Loop.run` never reads `ctx.provider`). Pairs with bareguard's secret-redaction (BG-1) as defense-in-depth. **`opts.context` (BA-9/F19):** an optional read-only working-context string (paths/cwd) prepended to EVERY worker's task message as a `Working context:` block + forwarded to the Planner as `info` (path-aware slices) + shown to the verifier (neutral FACTS, not a stance) — fixes a sliced worker that can't LOCATE its artifact (relayfact probe-04: workers guessed `.`/`~`/`/tmp`); CARRIES DOWN via `forChild` like `persona` but distinct (persona=privileged SYSTEM stance, context=USER-message run-state, so callers stop laundering cwd through persona); live 0/3→3/3 (`poc/ba9-context-thread.mjs`). **`opts.refineLeaf` (BA-8/F17, opt-in):** turns a DEFINITE leaf (`!canSpawn`) into a bounded generate→sense→regenerate loop reusing `refine.js` — `{sensor, maxIterations?, temperatures?, rejectedBuffer?}`; `sensor`=caller's DETERMINISTIC close (test/compile/lint, not a model judge), its GAP fed FRESH into the next attempt with **ESCALATING temperature** (default `[0.2,0.7,1.0]` — load-bearing WITHOUT memory: flat temp recovered 0/5 as a weak model regenerated identical wrong code ignoring crisp feedback; escalation → 0/5→2-3/5, `poc/ba8-leaf-refine.mjs`). **`rejectedBuffer` (BA-14):** a SkillOpt-shaped rejected-attempt buffer — surface the model's OWN prior failed attempts VERBATIM ("write something structurally DIFFERENT") via `refine.js`'s newly-threaded `attempt` `history`. DIRECTED diversity that ANTAGONIZES temperature's RANDOM diversity: temperature MONOTONICALLY degrades the buffer (`poc/ba14b`: flat 0.2/0.7/1.0 → 100/70/50%), so when it engages the temp is HELD flat at `temps[0]`, never escalated — REFINING BA-10's "temperature is secondary" to "with a buffer, temperature is harmful". Trigger: `true` force-on / `false` force-off / UNSET = ADAPTIVE (engage only once a temperature was DROPPED — a temp-fixed model where escalation is inert and the buffer is the sole lever, `poc/ba14`: 50%→100%). Live verify-shipped on real temp-fixed `claude-sonnet-5` (`poc/ba14-verify-shipped.mjs`: engaged 6/6, ledger reached the wire; efficacy an honest NULL — the lift is a WEAK-model/fixation phenomenon, cost-neutral otherwise, so ADAPTIVE-not-always-on). `receipts.refineLeaf.rejectedBuffer` reports engagement. Gate-bounded per attempt; honest non-recovery (`receipts.refineLeaf.passed=false`), never faked; carries down (engages at the leaves); error-keyed `recall` stays the CALLER's `opts.tools` (litectx-agnostic). Shipped-vs-POC smoke `poc/ba89-shipped-smoke.mjs`. **Governance deny-spin (BA-11/F35):** a worker whose Loop short-circuits a consecutive-policy-deny spin (the Loop `maxConsecutiveDenials` guard, default 3) surfaces as a LABELED `{incomplete, blocker:'governance-deny'}` (on BOTH the plain-worker and `refineLeaf` paths + `receipts.blocker`) — so a caller distinguishes a governance block (widen scope / re-gate / escalate) from a model failure, instead of the probe-16 burn ($1 to the cap, sensor never reached). Live verify-shipped `poc/ba11-verify-shipped.mjs` (haiku, stopped at 3 denials). **Broken-sensor blocker (BA-15):** a `refineLeaf.sensor` that THROWS (non-Halt) or returns a MALFORMED verdict (neither boolean `pass` nor valid tri-state `status`) is a faulty ARBITER, not a model failure — the loop stops at the FIRST broken close (a retry against a broken judge carries zero feedback; pre-fix it burned all attempts then surfaced converged-shaped) and returns `{incomplete, blocker:'broken-sensor'}` + `receipts.blockerDetail`, `best` preserving the last unjudged attempt (BA-5); sensor `HaltError` stays a clean halt; a HUNG sensor is the caller's to bound (isolated child process + timeout — no knob). The SAME class held at the verify slot: a caller `opts.evaluate` that throws/returns garbage is `blocker:'broken-verifier'` via one `verifyOrBlock` helper on ALL five dispatch paths (pre-fix: a throw CRASHED the run plain-worker / laundered bare under refineLeaf; garbage rode out converged-shaped) — the default Evaluator rubric path is never labeled (provider-class faults, narrowest guard); `return await` is load-bearing at the call sites (an un-awaited returned promise escapes the try's Halt catch). Validated pre-fix `poc/ba15-broken-sensor.mjs`. `Evaluator` fills the verify slot (`opts.evaluate` overrides); **NB-3 reduce** (src/recurse-synthesize.js, `synthesize(task, results, opts)`): `opts.synthesize` = a **function** (deterministic code-reduce over child `results` — the §9.1 aggregation path, LLM arithmetic is the weak link) OR a strategy string `'concat'`/`'merge'` (lossless join / isolated Loop-driven subjective merge); reduce fires only on a node that spawned (a leaf keeps its own answer; each level reduces ITS children); default (Family-A) = the parent model's own closing-turn synthesis. **RC-5 retrieval modes BUILT (step 7, src/recurse-retrieval.js)** — context is a HANDLE routed by TASK SHAPE (§9.2.1), not one default: count/"all"→`opts.retrieval:'scan'` (DEFAULT when `opts.corpus` present) = scan every slice + LLM-judge + **CODE-count** over a **generic array slice-source** (`opts.corpus={id,text}[]`, NOT litectx), `opts.window`≈8 (the one calibrated number, recall knee) + `opts.passes`=2 (deterministic rotation union, NO RNG → RC-3 holds); needle→`'search'` (opt-in) = `ctx.litectx.recall` handle tool, embeddings on, `fact`/`episode`, capped `KNN_K=8` — CANNOT count; exact→`'exact'` (opt-in) = embeddings-free code-side AND-term filter tool. **Per-query face `'tools'` (opt-in, step-7 follow-on DONE):** offers `scan_count` (the new exported `buildScanTool`, the complete code-count path) ALONGSIDE `search_memory`/`exact_match` so a Family-A worker routes PER SUB-QUERY by the tool DESCRIPTIONS (scan="use for how many/all/count"; search="never count") — a MIXED needle+count task keeps both; completeness guard does NOT fire for `'tools'` (scan always offered, complete path always reachable). RC-9 floor (dead window→"count is a floor") + unwrapped `HaltError` hold at the tool boundary; async source materialized once (cached); `forChild` strips it (children don't inherit). Live-validated `poc/rlm-scan-as-tool.mjs` (worker picked search-for-needle + scan-for-count, code-known truth, neutral prompt). Completeness-contract guard UPGRADES a capped `search` on a "how many/all" ask to scan (upgrade-ONLY, never silent downgrade). RC-2 (judge matches only ids it was shown — a hallucinated id is intersected away) + RC-9 (dead window/empty corpus → `{incomplete, missingSlices}`, never a fabricated `count:0`) hold; gate-Halt mid-scan → clean incomplete. **Backend split:** litectx has no exhaustive rank-free enumerate verb today (every read FTS-gated), so scan reads the array slice-source; the "corpus already in litectx" case + the data-driven *width* count wait on the litectx `enumerate` verb (spec handed off: docs/01-product/prd.md) and drop in behind the same socket with ZERO recurse changes. `opts.tools` remain caller-supplied handle tools. Measured on AG News; naive "search→count" DROPPED. Backward-compatible: no corpus + no retrieval ⇒ unchanged (Family A). NB-5 decomposition policy + few-shot in recurse-prompts.js. **Family-B forced fan-out (`opts.count`/`mode:'fanout'`, NB-2) BUILT (step 5):** deterministic count→`Planner`(`{count}` seam = exactly N independent steps)→`runPlan`(waves)→NB-3 reduce(`'concat'` default)→verify; count = `opts.count` (overrides) else the **live-calibrated** tier map medium/complex/critical→**2/4/6** (an overridable default, not a discovered constant — knee location is topology). Each slice is a fresh-window child (MAY itself self-decompose under Family A); RC-9 dead-slice→`missingSlices` (no survivor-sum); planner/child/reduce/verify `HaltError`→clean incomplete. **Metered + pre-wave checkpoint (§6):** the decomposition call forwards usage (`onLlmResult`) so it's not invisible to the budget, and the cheap decompose RESOLVES the unknown→known width, after which `ctx.policy('recurse_fanout', {count, depth})` runs BEFORE the worker wave — a `HaltError` (budget / ≥80% HITL pause, bareguard's decision) → clean incomplete before any worker spends (bounds the concurrent burst to zero); a plain deny is advisory (allowlist-safe). **`opts.count` = fixed/semantic FLOOR;** the **data-driven *width* PARTITION** is BUILT (litectx 0.26 `enumerate`): `recurse(task, ctx, {mode:'partition', corpus, workerBudget?})` measures the corpus → `width=max(floor, ⌈size/workerBudget⌉)` (capped by guards + size) parallel scan-workers → union-count; a distinct path from `recurseFanout`'s semantic split, same FLOOR. `litectxCorpus(litectx,{kind})` is the resident slice-source (paginates `enumerate`); `opts.corpus` also accepts an in-hand array OR any async `()=>Promise` (recurse stays litectx-agnostic). Pre-wave `recurse_partition` checkpoint + RC-9 hold. `enumerate` was verified vs the spec DoD before building (`poc/litectx-enumerate-verify.mjs`). Composes AROUND a Loop (never imported by loop.js); validated by 71 mutation-checked offline tests + a live `--fanout` smoke + **step-8 e2e replay (verify-shipped-vs-POC, `poc/rlm-step8-shipped-replay.mjs`): the SHIPPED scan reproduces §9.2.1 on AG News (recall 0.88/precision 0.97/err 9%, code-count===matchedIds) — a real regression caught en route (a colon-ambiguous id prompt under-recalled 0.93→0.29; fixed with ` => ` display + verbatim-copy)** | | SkillRegistry | src/skills.js | eval-assist F2 skill mechanism (PRD §2.3–2.7): operator-registered `{name, description, instructions, tools}` bundles surfaced by PROGRESSIVE DISCLOSURE — one meta-tool `skill_use` carries a one-liner catalog in its description; `skill_use({name})` returns the skill's `instructions` AS the tool result and unlocks its (auto-`_`-prefixed) tools, called natively next round. The only Loop coupling is the tools-as-thunk primitive (D4): pass `skills.activeTools` (bound) as the `tools` thunk. Governance unchanged (D5): the gate judges `(tool, args)` blind to origin — skills affect DISCOVERY, never AUTHORIZATION. Separator is `_` not `.` (POC-corrected — providers' `^[a-zA-Z0-9_-]+$` rejects a dot); `SAFE_NAME` enforced at `register()` (fail-fast vs mid-run provider error). Collision check across native/MCP/skill names commits nothing on failure. Loop never imports this file | | stash | src/stash.js | eval-assist F2 reference skill (PRD §2.8–2.14): compaction-first context hygiene. `createStashSkill(options) → {skill, trim}` — register `skill`, wire `trim` into `Loop({ trim })`. Tools (`stash_checkpoint`/`stash_compact`/`stash_restore`) QUEUE intent (args-only); the fold runs in `trim(msgs, ctx)` (loop.js:487) — the one seam with the live transcript — at a clean round boundary (a synchronous fold would orphan the triggering compact's own tool pair). **Two transcript invariants held by construction:** tool_call/tool_result pairing (fold whole rounds) + Anthropic user/assistant alternation (no consecutive-role merging) — so the fold evicts whole rounds and any inline note is a synthetic `assistant(tool_call)+tool(result)` PAIR on a user→assistant boundary (a bare note breaks alternation; a `system` note is hoisted+clobbers the system prompt). Anchor = identity-ref to the boundary message, NOT an injected marker. Two strategies (D10): `summarize` (default, lossy gist via `ctx.summarize`; degrades LOUDLY to a lossless park when unwired) + `stash` (lossless verbatim → litectx stash table or run-scoped in-process Map). Stance (D13): litectx `episode` upsert on compact. Auto-trigger (D11/§2.11, opt-in `compaction:{ceilingTokens,triggerAt,strategy,keepHeadTurns,keepRecentTurns}`): folds the MIDDLE on measured `ctx.usage.inputTokens/ceiling > triggerAt` (Loop publishes `ctx.usage`; all bareagent, never a bareguard halt bound). LRU label backstop (§2.13). Loop never imports this file | | StateMachine | src/state.js | Task lifecycle (pending/running/done/failed/waiting/cancelled) | diff --git a/README.md b/README.md index cd0d4f8..75637a5 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ Every piece works alone — take what you need, ignore the rest. Two axes: **Act ### Recurse — break a hard task into a tree *(the RLM primitive)* -`recurse(task, ctx, opts)` does **decompose → fan-out → verify → synthesize** in one call — Recursive Language Models as a single import, composed *around* the Loop (never a new engine). The default is **model-driven**: the worker is handed a `spawn_child` tool and decides whether to split, bounded by depth + bareguard (no second guard layer). Forced fan-out (`count` / `mode:'fanout'`) and data-driven width (`mode:'partition'`, measured from a corpus) are opt-in. Give workers a stance with `opts.persona` (prepended to every worker, carries down the tree, deliberately kept out of the isolated verifier), and tell them *where they are* with `opts.context` (a read-only paths/cwd blob threaded to every worker so a sliced child can locate its artifact — facts, not a stance). For a leaf that should self-correct, pass `opts.refineLeaf` (opt-in): a definite leaf becomes a bounded generate→sense→regenerate loop driven by *your* deterministic sensor (test/compile/lint — one that judges the *returned* result, never a worker side-effect it could game), feeding the gap back (with escalating temperature on models that accept it; on a temperature-fixed model like `claude-sonnet-5` the gap critique carries recovery, and the receipt records the effective temps). On a temperature-fixed model, `refineLeaf.rejectedBuffer` adds a second lever — it feeds the model's own prior *failed attempts* back verbatim ("write something structurally different"), the directed-diversity complement to temperature's random diversity (adaptive by default; the two are antagonistic, so it holds temperature flat when it engages). The headline guarantee: **aggregation is code, never a model-stated number**, and a dead worker or exhausted guard returns an honest `{ incomplete, missingSlices }` — never a faked pass. +`recurse(task, ctx, opts)` does **decompose → fan-out → verify → synthesize** in one call — Recursive Language Models as a single import, composed *around* the Loop (never a new engine). The default is **model-driven**: the worker is handed a `spawn_child` tool and decides whether to split, bounded by depth + bareguard (no second guard layer). Forced fan-out (`count` / `mode:'fanout'`) and data-driven width (`mode:'partition'`, measured from a corpus) are opt-in. Give workers a stance with `opts.persona` (prepended to every worker, carries down the tree, deliberately kept out of the isolated verifier), and tell them *where they are* with `opts.context` (a read-only paths/cwd blob threaded to every worker so a sliced child can locate its artifact — facts, not a stance). For a leaf that should self-correct, pass `opts.refineLeaf` (opt-in): a definite leaf becomes a bounded generate→sense→regenerate loop driven by *your* deterministic sensor (test/compile/lint — one that judges the *returned* result, never a worker side-effect it could game), feeding the gap back (with escalating temperature on models that accept it; on a temperature-fixed model like `claude-sonnet-5` the gap critique carries recovery, and the receipt records the effective temps). On a temperature-fixed model, `refineLeaf.rejectedBuffer` adds a second lever — it feeds the model's own prior *failed attempts* back verbatim ("write something structurally different"), the directed-diversity complement to temperature's random diversity (adaptive by default; the two are antagonistic, so it holds temperature flat when it engages). A *broken* arbiter — a sensor or a caller `evaluate` that throws or returns a malformed verdict — is named, never blamed on the model: the node stops at the first broken close and returns `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier' }` with the model's unjudged work preserved. The headline guarantee: **aggregation is code, never a model-stated number**, and a dead worker or exhausted guard returns an honest `{ incomplete, missingSlices }` — never a faked pass. Over a corpus, context reaches a worker as a **handle routed by question shape** (`opts.retrieval`): diff --git a/bareagent.context.md b/bareagent.context.md index 80c4135..0bd036d 100644 --- a/bareagent.context.md +++ b/bareagent.context.md @@ -712,6 +712,8 @@ const out = await recurse('Audit auth.js, billing.js, gateway.js for authz bugs' > **Sensor integrity (`refineLeaf.sensor`):** the sensor must judge the **returned result** (tamper-proof — build/run the returned string in isolation), never a worker **side-effect** a worker with edit tools could game (writing a passing file then returning junk, or editing the failing test itself). The loop optimizes against whatever the sensor reads — keep the close outside what the worker can write. (RSI field lesson: reward-hacking appeared in every optimization loop with a gameable close. **Demonstrated in-repo** — `poc/sensor-gaming-blocked.mjs`: the moment the honest path was blocked, `claude-sonnet-5` faked a pass on an unsatisfiable on-disk check **5/5** — both by editing the test and by making the returned function non-pure — while a tamper-proof close that runs the returned artifact in isolation was not gameable.) +> **Broken arbiter ≠ failing model (BA-15):** a `refineLeaf.sensor` — or a caller `opts.evaluate` verifier — that **throws** (your test runner crashed — ENOENT, a harness syntax error) or returns a **malformed verdict** (anything without a boolean `pass` or a valid tri-state `status`) is a faulty **arbiter**, not a model failure. The loop stops at the **first** broken close (it never retries against a broken judge — each retry would carry zero feedback) and returns a labeled `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier' }` plus `receipts.blockerDetail` (what the arbiter did), with `best` preserving the model's unjudged work — it was never judged, not judged-and-failed. The label names WHICH knob to fix. Check `out.blocker` before debugging the model: `'broken-sensor'` / `'broken-verifier'` mean fix your sensor / your `evaluate` and re-run; `'governance-deny'` means widen scope / re-gate. The **default Evaluator rubric verifier is never labeled** (its failures are provider-class faults), and a `HaltError` thrown by either arbiter stays a clean governance halt. One thing stays YOURS: an arbiter that **hangs** hangs the node (no gate checkpoint fires inside your callback) — run untrusted or model-generated checks in an isolated child process **with a timeout**. + > **`rejectedBuffer` (BA-14, v0.30.0):** a second lever for a **temperature-fixed** model, where escalation is inert. Instead of only the latest critique, it surfaces the model's OWN prior failed attempts VERBATIM — *"you wrote these, they failed X — write something STRUCTURALLY DIFFERENT."* This is **directed** diversity (attack the specific repeated mistake); escalation is **random** diversity, and the two are **antagonistic** — temperature monotonically degrades the buffer (`poc/ba14b`: flat-0.2 100% → 0.7 70% → 1.0 50%), so when the buffer engages the retry temperature is **held flat** at `temperatures[0]`, never escalated. Trigger: `true` = force on (also on temperature-accepting models); `false` = force off (pure BA-8 escalation); **unset = adaptive** — engage only once a prior attempt's temperature was dropped (i.e. a temp-fixed model where escalation is inert and the buffer is the sole lever). On a temperature-accepting model the default leaves behavior byte-identical. `receipts.refineLeaf.rejectedBuffer` reports whether it engaged. Efficacy is a **weak-model / fixation** phenomenon (live on `claude-sonnet-5` it engaged 6/6 but recovered no better than critique-only — cost-neutral, hence adaptive-not-always-on). ```javascript diff --git a/docs/01-product/RLM_PRD.md b/docs/01-product/RLM_PRD.md index 210f330..6f77704 100644 --- a/docs/01-product/RLM_PRD.md +++ b/docs/01-product/RLM_PRD.md @@ -1340,6 +1340,48 @@ bareagent's) (`UPSTREAM-FIXES.md` BA-11 / `FINDINGS.md` F35). weak-model lift real or noise?) is NOT worth resolving: even if fully real it's dominated by the buffer on cost. This closes RSI-POC-BACKLOG §2.A. +**Added (adopter "faulty primitives" feedback, 2026-07-19) → `[Unreleased]`.** The adopter's forbidden-zone +audit of `refineLeaf`'s close chain (their F25 borrow; named `broken-sensor` / BA-15 here per the +no-number-reuse rule) claimed the outcomes BETWEEN clean-green and clean-red were silently COERCED instead of +NAMED. Validated in shipped code before building (`poc/ba15-broken-sensor.mjs`, deterministic/offline, pre-fix +run kept as evidence): + +- **(BA-15) a BROKEN sensor was indistinguishable from a failing model.** Confirmed live, two shapes: + **(a) sensor THROWS** (the caller's test runner crashes — ENOENT, harness syntax error): the catch fell + through to the bare `{ incomplete, best:null }` — byte-identical to a provider death, and the model's + unjudged work was destroyed. **(b) sensor returns a MALFORMED verdict** (`{}`, a string, `{ok:true}`): + `refine` read `pass` falsy / `status` invalid with `critique:null`, so every retry re-sent the PLAIN task + (zero feedback — the retry-the-broken-arbiter pathology), burned all `maxIterations`, then surfaced as a + **converged-shaped** `{ result, verdict:{} }` — an honest-looking model non-recovery pinned on a broken + arbiter. The POC's falsifier arm (a well-formed failing sensor DOES thread its critique into the retry) + proved the zero-feedback observation was the seam, not the harness. **Fix (recurse.js, the sensor seam + only):** the sensor call is wrapped — a non-Halt throw or a malformed return (neither boolean `pass` nor a + valid tri-state `status`) stops the loop at the FIRST broken close and returns a labeled `{ incomplete, + blocker:'broken-sensor' }` + `receipts.blockerDetail` (what the sensor did), with `best` preserving the last + attempt (BA-5: the work was never judged, not judged-and-failed). Extends the BA-11 blocker taxonomy; + `HaltError` from the sensor stays a clean governance halt; both documented verdict shapes (`{pass}` / + `{status}`) are byte-identical to before. A hung sensor stays the CALLER's responsibility (documented, no + timeout knob — the sensor's execution environment is caller-owned; lean-primitive bar). +7 mutation-checked + tests, full suite 750 pass / 0 fail, typecheck clean. +- **(BA-15, verifier seam) the same fault class was LIVE at the verify slot — found on the user's "did you + validate all?" follow-up, falsifying the first-pass "no live laundering path" assessment of the Evaluator + leg.** A deterministic probe showed: a THROWING caller `opts.evaluate` **crashed the whole `recurse()` run** + as an uncaught exception on the plain-worker path (and laundered to a bare `{incomplete}` under + `refineLeaf` — inconsistent siblings), while a GARBAGE return rode out **converged-shaped** as + `{result, verdict:{}}`. **Fix:** the caller verifier is wrapped exactly like the sensor (throw/malformed → + tagged), and one `verifyOrBlock` helper gives all five dispatch paths (worker/refineLeaf/scan/partition/ + fanout) identical semantics: labeled `{incomplete, blocker:'broken-verifier'}` + `receipts.blockerDetail`, + `best` = the unjudged result. The DEFAULT Evaluator rubric path is deliberately NOT labeled (well-formed + Verdicts by construction; its failures are provider-class faults — narrowest-guard). The POC's Halt-control + arm caught a refactor regression en route: `return verifyOrBlock(...)` inside a try let a verifier + `HaltError` escape the catch (un-awaited promise exits the try before settling) — fixed with `return await` + + a dedicated regression test. +6 tests on top of the sensor seam's 7; suite 756 pass / 0 fail. +- **Assessed, NOT built (same feedback, lean bar):** the DIRECT-Evaluator predicate coercion claim + (`!!predicate()`) is the predicate's documented boolean contract and a throw there is a visible exception to + the direct caller (the laundering only arose one level up, at recurse's seam — fixed above); the proposed + `commandSensor` helper and the F26-style toggle-coverage audit are parked as follow-ons, not primitives (no + confirmed live gap). + --- ## Source & cross-refs diff --git a/poc/ba15-broken-sensor.mjs b/poc/ba15-broken-sensor.mjs new file mode 100644 index 0000000..0993156 --- /dev/null +++ b/poc/ba15-broken-sensor.mjs @@ -0,0 +1,183 @@ +// BA-15 POC — broken-sensor detection at the refineLeaf close seam (adopter "faulty primitives" feedback, +// F25-zone borrow). Claim under test: a caller sensor that THROWS or returns a MALFORMED verdict is +// silently coerced into the same shapes as a genuine model failure: +// (A) sensor throw → bare {incomplete, best:null}, byte-identical to a provider/model fault (no blocker); +// (B) garbage verdict ({}) → refine burns ALL maxIterations with ZERO feedback (critique:null ⇒ the retry +// prompt is the plain task again), then reports receipts.refineLeaf.passed:false — an honest-looking +// model non-recovery pinned on a broken arbiter. +// Falsifier arm (C): a WELL-FORMED failing sensor DOES thread its critique into the retry prompt — proving +// the harness exercises the variable (the zero-feedback observation in B is real, not an unwired harness). +// +// Deterministic, offline, no API keys. Asserts the FIXED contract: +// - sensor throw / malformed verdict → {incomplete, blocker:'broken-sensor'} (+ receipts.blocker), loop +// STOPS immediately (no retries against a broken arbiter), best preserves the last attempt (BA-5); +// - a HaltError thrown by the sensor still propagates as a clean governance halt (never relabeled); +// - well-formed sensors (both {pass} and {status} shapes) behave exactly as before. +// Run PRE-fix: exits 1 while PRINTING the observed pathology (the evidence the gap is live in shipped code). +// Run POST-fix: exits 0. + +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { recurse } = require('../src/recurse.js'); +const { HaltError } = require('../src/errors.js'); + +const TASK = 'list the open files'; // 'simple' tier → definite leaf → refineLeaf engages + +function stubProvider({ fail = false } = {}) { + const calls = []; + return { + calls, + provider: { + model: 'stub-model', name: 'stub', + async generate(messages) { + calls.push({ messages: messages.map(m => ({ role: m.role, content: m.content })) }); + if (fail) throw new Error('connection reset by peer'); // a genuine provider/model-side fault + return { text: `ATTEMPT_${calls.length}_RESULT`, toolCalls: [], usage: { inputTokens: 5, outputTokens: 3 }, model: 'stub-model' }; + }, + }, + }; +} + +const lastUser = (call) => { const u = call.messages.filter(m => m.role === 'user'); return u.length ? u[u.length - 1].content : ''; }; + +let failures = 0; +const check = (cond, label, detail) => { + if (cond) { console.log(` PASS ${label}`); } + else { failures++; console.log(` FAIL ${label}${detail ? `\n observed: ${detail}` : ''}`); } +}; + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[A] sensor THROWS (caller test runner crashed — not the model\'s fault)'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new Error('ENOENT: node_modules/.bin/vitest not found'); } }, + }); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker, best: out.best })}`); + check(out.incomplete === true, 'sensor throw is not a converged result'); + check(out.blocker === 'broken-sensor', 'sensor throw is NAMED blocker:\'broken-sensor\'', `blocker=${JSON.stringify(out.blocker)} (unlabeled ⇒ collapses into "model failed")`); + check(out.receipts && out.receipts.blocker === 'broken-sensor', 'receipts carry the blocker too'); + check(sp.calls.length === 1, 'loop STOPS at the first broken close (no retries against a broken arbiter)', `${sp.calls.length} attempts ran`); + check(out.best === 'ATTEMPT_1_RESULT', 'the model\'s work is PRESERVED (BA-5 — the sensor broke, not the model)', `best=${JSON.stringify(out.best)}`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[A-control] genuine MODEL/provider fault — must remain distinguishable from [A]'); +{ + const sp = stubProvider({ fail: true }); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ pass: true }) }, // sensor is fine; the provider dies + }); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker })}`); + check(out.incomplete === true, 'provider fault is an honest incomplete'); + check(out.blocker !== 'broken-sensor', 'provider fault is NOT labeled broken-sensor', `blocker=${JSON.stringify(out.blocker)}`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[B] sensor returns GARBAGE ({}) — a malformed verdict, no pass/status'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({}), maxIterations: 3 }, + }); + const retryPrompts = sp.calls.slice(1).map(lastUser); + const zeroFeedback = retryPrompts.length > 0 && retryPrompts.every(p => !p.includes('FAILED')); + if (zeroFeedback) console.log(` pathology: ${sp.calls.length} attempts burned, every retry prompt carried ZERO feedback (identical plain task)`); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker, verdict: out.verdict })}`); + check(out.blocker === 'broken-sensor', 'malformed verdict is NAMED, never coerced', `blocker=${JSON.stringify(out.blocker)}, refineLeaf=${JSON.stringify(out.receipts && out.receipts.refineLeaf)}`); + check(sp.calls.length === 1, 'first malformed verdict STOPS the loop (attempts are not burned)', `${sp.calls.length} attempts ran`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[B2] other malformed shapes: undefined, string, {ok:true}'); +for (const [label, bad] of [['undefined', () => undefined], ['string "ok"', () => 'ok'], ['{ok:true}', () => ({ ok: true })]]) { + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { refineLeaf: { sensor: bad } }); + check(out.blocker === 'broken-sensor' && sp.calls.length === 1, `sensor → ${label} ⇒ broken-sensor, 1 attempt`, `blocker=${JSON.stringify(out.blocker)}, attempts=${sp.calls.length}`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[C-falsifier] WELL-FORMED failing sensor — critique wiring must work (proves [B]\'s zero-feedback is real)'); +{ + const sp = stubProvider(); + let calls = 0; + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => (++calls < 2 ? { pass: false, critique: 'MISSING_SEMICOLON_LINE_4' } : { pass: true }), maxIterations: 3 }, + }); + const retry = sp.calls.length > 1 ? lastUser(sp.calls[1]) : ''; + check(retry.includes('MISSING_SEMICOLON_LINE_4'), 'a real critique IS threaded into the retry prompt', `retry prompt: ${JSON.stringify(retry.slice(0, 120))}`); + check(out.blocker === undefined && out.receipts.refineLeaf.passed === true, 'well-formed sensor path unchanged (passes on retry)', JSON.stringify({ blocker: out.blocker, refineLeaf: out.receipts.refineLeaf })); +} + +console.log('\n[C2] status-shaped verdict ({status:\'failed\'}) stays a VALID terminal verdict, not broken-sensor'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ status: 'failed', critique: 'fundamentally wrong approach' }) }, + }); + check(out.blocker === undefined, 'status-only verdict shape is accepted', `blocker=${JSON.stringify(out.blocker)}`); + check(sp.calls.length === 1 && out.receipts.refineLeaf.passed === false, 'terminal failed stops after 1 attempt, honest non-pass', `attempts=${sp.calls.length}`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[D] sensor throws HaltError — governance halt must propagate clean, never relabeled broken-sensor'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new HaltError('budget cap', { rule: 'budget' }); } }, + }); + check(out.incomplete === true && out.receipts.halted === true && out.blocker === undefined, 'HaltError from the sensor is a clean governance halt', JSON.stringify({ blocker: out.blocker, halted: out.receipts.halted })); +} + +// ========================================================================================================= +// VERIFIER SEAM (same fault class at the verify slot — found by the "did you validate all?" follow-up probe: +// pre-fix a THROWING caller `opts.evaluate` CRASHED the whole run on the plain-worker path and laundered to a +// bare {incomplete} under refineLeaf, and a GARBAGE return rode a CONVERGED-shaped {result, verdict:{}} out). +// --------------------------------------------------------------------------------------------------------- +console.log('\n[E] plain worker + THROWING caller verifier (opts.evaluate)'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + contract: 'must list files', evaluate: () => { throw new Error('rubric grader exploded'); }, + }).catch((e) => ({ threw: e.message })); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker, best: out.best, threw: out.threw })}`); + check(!out.threw, 'a broken caller verifier no longer CRASHES the run', `threw: ${out.threw}`); + check(out.blocker === 'broken-verifier', 'it is NAMED blocker:\'broken-verifier\'', `blocker=${JSON.stringify(out.blocker)}`); + check(out.best === 'ATTEMPT_1_RESULT', 'the unjudged work is preserved as best (BA-5)', `best=${JSON.stringify(out.best)}`); +} + +console.log('\n[E2] plain worker + GARBAGE caller verifier ({})'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { contract: 'must list files', evaluate: () => ({}) }); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker, verdict: out.verdict })}`); + check(out.incomplete === true && out.blocker === 'broken-verifier', 'a garbage verdict never rides out converged-shaped', `blocker=${JSON.stringify(out.blocker)}, verdict=${JSON.stringify(out.verdict)}`); + check(typeof out.receipts.blockerDetail === 'string' && out.receipts.blockerDetail.includes('neither'), 'the detail names the malformation'); +} + +console.log('\n[E3] refineLeaf + throwing verifier → labeled too (was a bare incomplete)'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ pass: true }) }, + contract: 'must list files', evaluate: () => { throw new Error('grader dead'); }, + }); + check(out.blocker === 'broken-verifier' && out.best === 'ATTEMPT_1_RESULT', 'labeled + work preserved under refineLeaf', JSON.stringify({ blocker: out.blocker, best: out.best })); +} + +console.log('\n[E4] controls: a FAILING well-formed verdict is judged-and-failed (not a blocker); HaltError stays a halt'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + contract: 'must list files', evaluate: () => ({ status: 'failed', pass: false, score: 0, critique: 'wrong', suggestions: [] }), + }); + check(out.blocker === undefined && out.verdict && out.verdict.pass === false, 'judged-and-failed returns {result, verdict} untouched', JSON.stringify({ blocker: out.blocker, verdict: out.verdict })); + const sp2 = stubProvider(); + const out2 = await recurse(TASK, { provider: sp2.provider }, { + contract: 'must list files', evaluate: () => { throw new HaltError('budget cap', { rule: 'budget' }); }, + }); + check(out2.incomplete === true && out2.receipts.halted === true && out2.blocker === undefined, 'verifier HaltError is a clean governance halt', JSON.stringify({ blocker: out2.blocker, halted: out2.receipts.halted })); +} + +console.log(failures === 0 ? '\nALL CHECKS PASS — the fixed contract holds.' : `\n${failures} CHECK(S) FAILED — the gap is live (pre-fix evidence above).`); +process.exit(failures === 0 ? 0 : 1); diff --git a/src/recurse.js b/src/recurse.js index 104555f..18bcf6e 100644 --- a/src/recurse.js +++ b/src/recurse.js @@ -238,6 +238,13 @@ function auditSafeCtx(ctx, overrides = {}) { * NEVER a worker side-effect a worker with edit tools could GAME (writing a passing file then returning junk, or * editing the failing test itself). A gameable close is the reward-hacking surface every RSI system in the field * got bitten by; the loop optimizes against WHATEVER the sensor reads, so keep it outside what the worker can write. + * **Broken sensor ≠ failing model (BA-15):** a sensor that THROWS (non-Halt) or returns a MALFORMED verdict + * (anything but `{pass: boolean}` or a valid tri-state `status`) is a faulty ARBITER — the loop stops at the + * FIRST broken close (never retries against it) and returns a labeled `{incomplete, blocker:'broken-sensor'}` + * (+ `receipts.blockerDetail`), with `best` preserving the model's last attempt. A `HaltError` thrown by the + * sensor stays a clean governance halt. The sensor's EXECUTION environment is the caller's: run untrusted / + * model-generated checks in an isolated child process WITH A TIMEOUT — a sensor that hangs forever hangs the + * leaf (no bareguard checkpoint fires between sensor start and return). * **`rejectedBuffer` (BA-14):** a SkillOpt-shaped rejected-attempt buffer — instead of only the LATEST critique, * surface the model's OWN prior failed attempts verbatim ("you wrote these, they failed X — write something * STRUCTURALLY DIFFERENT"). This is DIRECTED diversity (attack the specific repeated mistake), where escalation @@ -306,8 +313,12 @@ function auditSafeCtx(ctx, overrides = {}) { * @property {Verdict|null} verdict * @property {boolean} incomplete * @property {boolean} halted - * @property {string} [blocker] - (BA-11) set to `'governance-deny'` when this node stopped because its Loop - * short-circuited a consecutive-policy-deny spin (not a model failure). Mirrors `RecurseResult.blocker`. + * @property {string} [blocker] - Set when this node stopped for a specific non-model reason (mirrors + * `RecurseResult.blocker`): `'governance-deny'` (BA-11) — its Loop short-circuited a consecutive-policy-deny + * spin; `'broken-sensor'` (BA-15) — the caller's `refineLeaf.sensor` threw or returned a malformed verdict; + * `'broken-verifier'` (BA-15) — the caller's `opts.evaluate` did (the default Evaluator path is never labeled). + * @property {string} [blockerDetail] - (BA-15) with a `broken-*` blocker: what the arbiter did (threw with + * which message, or which malformed shape it returned) — the actionable half of the label. * @property {object|null} tokens - The worker Loop's `metrics.tokens`. * @property {{iterations: number, passed: boolean, temperatures: (number|null)[], rejectedBuffer: boolean}} [refineLeaf] - (BA-8) when * this leaf ran as a bounded refine loop: how many attempts it took and whether the deterministic sensor finally @@ -341,6 +352,12 @@ function auditSafeCtx(ctx, overrides = {}) { * @property {string} [blocker] - Present when `incomplete` for a specific, actionable reason. `'governance-deny'` * (BA-11): the worker's Loop short-circuited after N consecutive policy denials rather than burn to the * budget cap — the caller can widen scope / re-gate / escalate instead of reading it as a model failure. + * `'broken-sensor'` (BA-15): the caller's `refineLeaf.sensor` threw or returned a malformed verdict — the + * ARBITER is faulty, not the model; fix the sensor and re-run (`receipts.blockerDetail` says what it did). + * `'broken-verifier'` (BA-15): same fault class at the verify slot — the caller's `opts.evaluate` threw + * (non-Halt) or returned a malformed verdict; the default Evaluator path is never labeled (its failures are + * provider-class faults). For both `broken-*` blockers `best` preserves the model's unjudged work (BA-5 — + * the work was never judged, not judged-and-failed). * @property {RecurseNode} receipts - The audit node for this call (RC-10). */ @@ -593,9 +610,10 @@ async function recurse(task, ctx = {}, opts = {}) { // caller supplied a verifier, OR the task is critical (the forced-verify safety rail). const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the + // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's + // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; @@ -625,6 +643,27 @@ async function recurse(task, ctx = {}, opts = {}) { * @param {{provider: Provider, system: string, handleTools: ToolDef[], depth: number, critical: boolean, node: RecurseNode, sensor: Function}} state * @returns {Promise} */ +/** Valid tri-state `Verdict.status` values a caller arbiter (sensor/verifier) may return in lieu of a boolean `pass`. */ +const SENSOR_STATUS = new Set(['satisfied', 'needs_revision', 'failed']); + +/** + * BA-15 — validate a caller arbiter's return at a close seam (the `refineLeaf.sensor` AND the caller + * `opts.evaluate` verifier). A verdict is well-formed iff it is an object carrying a boolean `pass` OR a + * valid tri-state `status` (the two shapes `refine`/callers branch on). Returns `null` when well-formed, + * else a short description of the malformation ("named, never coerced" — a garbage verdict otherwise reads + * as pass:false with critique:null at the sensor seam, or rides a converged-shaped return at the verify slot). + * @param {any} v + * @returns {string|null} + */ +function verdictShapeFault(v) { + if (v === null || typeof v !== 'object' || Array.isArray(v)) { + return `returned ${v === null ? 'null' : Array.isArray(v) ? 'an array' : `a ${typeof v}`}`; + } + if (typeof v.pass === 'boolean' || SENSOR_STATUS.has(v.status)) return null; + const keys = Object.keys(v).slice(0, 5).join(', '); + return `returned an object with neither a boolean \`pass\` nor a valid \`status\` (keys: ${keys || 'none'})`; +} + async function recurseRefineLeaf(task, ctx, opts, state) { const { provider, system, handleTools, depth, critical, node, sensor } = state; node.model = provider.model || null; @@ -665,6 +704,11 @@ async function recurseRefineLeaf(task, ctx, opts, state) { // a dropped attempt is stored as `null` ("provider default"). Indexed by iteration (refine calls once each). /** @type {(number|null)[]} */ const effectiveTemps = []; + // BA-15/BA-5: the last attempt's text, kept OUTSIDE refine so a broken-sensor stop can still preserve the + // model's work — when the ARBITER breaks, the work was never judged; destroying it would punish the model + // for the caller's fault (refine's own history is lost on the throw). + /** @type {string|null} */ + let lastAttemptText = null; // One attempt = a fresh leaf Loop (no spawn tool: a retry is a direct correction, not a re-decomposition) at the // iteration's temperature, with the GAP fed forward as fresh feedback. A governance halt → throw so refine stops. const attempt = async ({ iteration, critique, history }) => { @@ -702,13 +746,33 @@ async function recurseRefineLeaf(task, ctx, opts, state) { accrueTokens(out.metrics ? out.metrics.tokens : null); if (typeof out.error === 'string' && out.error.startsWith('halt:')) throw new HaltError('refine-leaf attempt halted', { rule: out.error.slice('halt:'.length) }); if (out.error) throw new Error(out.error); // a non-halt worker fault → honest incomplete + lastAttemptText = out.text || lastAttemptText; return out.text; }; + // BA-15: the sensor call is WRAPPED so a broken arbiter is NAMED, never coerced. A non-Halt throw (the + // caller's test runner crashed — ENOENT, syntax error in the harness) and a malformed return are the same + // fault class: "didn't judge", which must never collapse into "judged-and-failed" (the model's fault) or a + // bare {incomplete} (indistinguishable from a provider death). The throw stops refine at the FIRST broken + // close — retrying against a broken arbiter burns every remaining attempt for nothing (each retry would + // carry critique:null, i.e. the plain task again). HaltError passes through untouched (governance, BA-2). + const evaluate = async (result, c) => { + let v; + try { + v = await sensor(result, { task, context: opts.context, contract: c.contract }); + } catch (err) { + if (err instanceof HaltError) throw err; + throw new Error(`broken-sensor: sensor threw: ${err && err.message ? err.message : String(err)}`); + } + const fault = verdictShapeFault(v); + if (fault) throw new Error(`broken-sensor: sensor ${fault} — a sensor must return {pass: boolean} or {status: 'satisfied'|'needs_revision'|'failed'}`); + return v; + }; + try { const outcome = await refine({ attempt, - evaluate: (result, c) => sensor(result, { task, context: opts.context, contract: c.contract }), + evaluate, contract: typeof opts.contract === 'string' ? opts.contract : undefined, maxIterations, }); @@ -722,9 +786,10 @@ async function recurseRefineLeaf(task, ctx, opts, state) { // Optional rubric layer on top of the deterministic sensor (RC-7): forced for critical, or a contract/override. const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the + // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's + // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + return await verifyOrBlock(task, result, ctx, opts, node); } // No rubric layer ⇒ the sensor's final verdict IS the node verdict (a non-pass is surfaced, not hidden). node.verdict = outcome.verdict || null; @@ -748,6 +813,15 @@ async function recurseRefineLeaf(task, ctx, opts, state) { node.blocker = 'governance-deny'; return { incomplete: true, best: null, blocker: 'governance-deny', receipts: node }; } + // BA-15: the caller's SENSOR broke — a faulty arbiter, not a model failure. Named (like BA-11's + // governance-deny) so the caller fixes the sensor instead of debugging the model; `best` preserves the + // model's last attempt (BA-5 — the work was never judged, not judged-and-failed). + if (typeof err?.message === 'string' && err.message.startsWith('broken-sensor: ')) { + node.incomplete = true; + node.blocker = 'broken-sensor'; + node.blockerDetail = err.message.slice('broken-sensor: '.length); + return { incomplete: true, best: lastAttemptText, blocker: 'broken-sensor', receipts: node }; + } node.incomplete = true; return { incomplete: true, best: null, receipts: node }; } @@ -824,9 +898,10 @@ async function recurseScan(task, ctx, opts, state) { // structured count against the goal/contract (an isolated grader, never the scanner itself). const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the + // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's + // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; } catch (err) { @@ -943,9 +1018,10 @@ async function recursePartition(task, ctx, opts, state) { } const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the + // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's + // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; } catch (err) { @@ -1083,9 +1159,10 @@ async function recurseFanout(task, ctx, opts, state) { // 6) Verify (RC-7): forced for critical, or when a contract/override is supplied. const wantVerify = critical || contract != null || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the + // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's + // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; } catch (err) { @@ -1184,7 +1261,23 @@ function verify(task, result, ctx, opts) { // and an agentic critic needs the path to exercise the artifact. A caller `evaluate` gets the RAW task (it owns // its own context); only the default isolated grader is contextualized. if (typeof opts.evaluate === 'function') { - return Promise.resolve(opts.evaluate(result, { contract, task })); + // BA-15 (verifier seam): the CALLER-supplied verifier is wrapped exactly like the refineLeaf sensor — a + // non-Halt throw or a malformed return is a faulty ARBITER, tagged so the call sites label it (pre-fix a + // throw crashed the whole run on the plain-worker path / laundered to a bare {incomplete} under refineLeaf, + // and a garbage verdict rode a CONVERGED-shaped {result, verdict} out). The default Evaluator path below is + // NOT wrapped: it constructs well-formed Verdicts by design, and its failures are provider-class faults. + return (async () => { + let v; + try { + v = await opts.evaluate(result, { contract, task }); + } catch (err) { + if (err instanceof HaltError) throw err; + throw new Error(`broken-verifier: evaluate threw: ${err && err.message ? err.message : String(err)}`); + } + const fault = verdictShapeFault(v); + if (fault) throw new Error(`broken-verifier: evaluate ${fault} — a verifier must return {pass: boolean} or {status: 'satisfied'|'needs_revision'|'failed'}`); + return v; + })(); } const provider = ctx.provider || opts.provider; const evaluator = new Evaluator({ provider }); @@ -1199,4 +1292,31 @@ function verify(task, result, ctx, opts) { ); } +/** + * BA-15 (verifier seam) — run the verify slot, converting a tagged broken-verifier fault into a LABELED + * `{ incomplete, blocker:'broken-verifier' }` return with `best` preserving the unjudged result (BA-5: the + * work exists — it was never judged, not judged-and-failed). One helper so all five dispatch paths (worker / + * refineLeaf / scan / partition / fanout) get identical semantics. Anything untagged (HaltError, a default- + * Evaluator provider fault) rethrows to the caller's own catch, exactly as before. + * @param {string} task + * @param {any} result + * @param {RecurseCtx} ctx + * @param {RecurseOptions} opts + * @param {RecurseNode} node + * @returns {Promise} + */ +async function verifyOrBlock(task, result, ctx, opts, node) { + try { + const verdict = await verify(task, result, ctx, opts); + node.verdict = verdict; + return { result, verdict, receipts: node }; + } catch (err) { + if (!(typeof err?.message === 'string' && err.message.startsWith('broken-verifier: '))) throw err; + node.incomplete = true; + node.blocker = 'broken-verifier'; + node.blockerDetail = err.message.slice('broken-verifier: '.length); + return { incomplete: true, best: result, blocker: 'broken-verifier', receipts: node }; + } +} + module.exports = { recurse }; diff --git a/test/recurse.test.js b/test/recurse.test.js index 46a21c5..f1bd7a3 100644 --- a/test/recurse.test.js +++ b/test/recurse.test.js @@ -661,6 +661,166 @@ describe('recurse — leaf retry-with-sensor (BA-8 / refineLeaf, relayfact F17)' }); }); +describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, never coerced")', () => { + // Pre-BA-15 pathologies these tests are mutated against: (a) a sensor THROW fell through to the bare + // {incomplete, best:null} — byte-identical to a provider death, and the model's work was destroyed; (b) a + // MALFORMED verdict ({}, 'ok', {ok:true}) read as pass:false with critique:null, so refine burned every + // remaining attempt re-sending the PLAIN task (zero feedback), then the run surfaced as an honest-looking + // model non-recovery — or worse, a converged-shaped {result, verdict:{}}. + const OK = () => ({ status: 'satisfied', pass: true, score: 1, critique: '', suggestions: [] }); + + it('a THROWING sensor → labeled {incomplete, blocker:\'broken-sensor\'} with the detail, work preserved (BA-5)', async () => { + const sp = scriptedProvider(() => ({ text: 'the model actually did the work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new Error('ENOENT: vitest not found'); } }, + }); + assert.equal(out.incomplete, true); + assert.equal(out.blocker, 'broken-sensor', 'the faulty arbiter is NAMED (pre-fix: unlabeled, read as model failure)'); + assert.equal(out.receipts.blocker, 'broken-sensor', 'receipts mirror the blocker (BA-11 pattern)'); + assert.match(out.receipts.blockerDetail, /sensor threw: ENOENT: vitest not found/, 'the detail says WHAT broke'); + assert.equal(out.best, 'the model actually did the work', 'best preserves the unjudged attempt — the sensor broke, not the model'); + assert.equal(sp.calls.length, 1, 'stops at the first broken close'); + }); + + it('a MALFORMED verdict stops the loop at the FIRST broken close — attempts are not burned with zero feedback', async () => { + const sp = scriptedProvider(() => ({ text: 'attempt' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({}), maxIterations: 3 }, + }); + assert.equal(out.blocker, 'broken-sensor', 'a garbage verdict is named, never coerced to pass:false'); + assert.match(out.receipts.blockerDetail, /neither a boolean `pass` nor a valid `status`/); + assert.equal(sp.calls.length, 1, 'pre-fix: 3 attempts burned, every retry a feedback-free re-send of the plain task'); + assert.equal(out.result, undefined, 'never a converged-shaped return riding a garbage verdict'); + }); + + it('malformed shapes are each caught: undefined, string, array, {ok:true}', async () => { + for (const bad of [() => undefined, () => 'ok', () => ['pass'], () => ({ ok: true })]) { + const sp = scriptedProvider(() => ({ text: 'attempt' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { refineLeaf: { sensor: bad } }); + assert.equal(out.blocker, 'broken-sensor', `sensor ${bad} must be flagged`); + assert.equal(sp.calls.length, 1, 'one attempt only'); + } + }); + + it('a genuine PROVIDER fault stays distinguishable — incomplete WITHOUT the broken-sensor label', async () => { + const sp = scriptedProvider(() => { throw new Error('connection reset by peer'); }); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { refineLeaf: { sensor: OK } }); + assert.equal(out.incomplete, true); + assert.equal(out.blocker, undefined, 'a model/provider fault must NOT be pinned on the sensor'); + }); + + it('a HaltError thrown by the sensor stays a clean governance halt, never relabeled broken-sensor', async () => { + const sp = scriptedProvider(() => ({ text: 'attempt' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new HaltError('budget cap', { rule: 'budget.maxCostUsd' }); } }, + }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true, 'governance halt recorded'); + assert.equal(out.blocker, undefined, 'a governance halt is not a sensor fault'); + }); + + it('well-formed verdict shapes are untouched: minimal {pass} retries with critique; {status:\'failed\'} is terminal', async () => { + // Minimal {pass:false, critique} (no status) must still drive a retry carrying the critique — the + // validator accepts BOTH documented shapes, not just the full Verdict. + let n = 0; + const sp = scriptedProvider(() => ({ text: `attempt_${++n}` })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: (r) => (String(r).includes('_2') ? { pass: true } : { pass: false, critique: 'NEEDS_V2' }), maxIterations: 3 }, + }); + assert.equal(out.blocker, undefined, 'minimal {pass} shape is valid'); + assert.ok(lastUser(sp.calls[1].messages).includes('NEEDS_V2'), 'the critique still reaches the retry'); + assert.equal(out.receipts.refineLeaf.passed, true); + + const sp2 = scriptedProvider(() => ({ text: 'attempt' })); + const out2 = await recurse(SIMPLE_TASK, { provider: sp2.provider }, { + refineLeaf: { sensor: () => ({ status: 'failed', critique: 'wrong approach' }) }, + }); + assert.equal(out2.blocker, undefined, 'status-only shape is valid'); + assert.equal(sp2.calls.length, 1, 'terminal failed still stops after one attempt'); + assert.equal(out2.receipts.refineLeaf.passed, false, 'honest non-pass, not a blocker'); + }); + + it('the refineLeaf receipt rides the broken-sensor path (every-terminating-path invariant)', async () => { + const sp = scriptedProvider(() => ({ text: 'attempt' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new Error('boom'); } }, + }); + assert.ok(out.receipts.refineLeaf, 'receipt present'); + assert.equal(out.receipts.refineLeaf.passed, false); + assert.equal(out.receipts.refineLeaf.iterations, 1, 'records the attempt that ran before the broken close'); + }); + + // ---- Verifier seam (same fault class at the verify slot: a caller opts.evaluate is an arbiter too). + // Pre-fix pathologies: a THROWING caller verifier CRASHED the whole run as an exception on the plain-worker + // path (and laundered to a bare {incomplete} under refineLeaf); a GARBAGE return rode a CONVERGED-shaped + // {result, verdict:{}} out. The default Evaluator rubric path is deliberately NOT labeled (well-formed by + // construction; its failures are provider-class faults) — narrowest guard. + + it('a THROWING caller verifier (opts.evaluate) → labeled broken-verifier, work preserved, never a crash', async () => { + const sp = scriptedProvider(() => ({ text: 'the actual work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => { throw new Error('rubric grader exploded'); }, + }); + assert.equal(out.incomplete, true, 'pre-fix this line was unreachable — recurse() threw'); + assert.equal(out.blocker, 'broken-verifier', 'the faulty verifier is NAMED'); + assert.match(out.receipts.blockerDetail, /evaluate threw: rubric grader exploded/); + assert.equal(out.best, 'the actual work', 'the unjudged result is preserved as best (BA-5)'); + }); + + it('a GARBAGE caller verdict never rides out converged-shaped', async () => { + const sp = scriptedProvider(() => ({ text: 'the actual work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => ({}), + }); + assert.equal(out.blocker, 'broken-verifier', 'pre-fix: {result, verdict:{}} — a converged shape riding garbage'); + assert.equal(out.result, undefined, 'no result field on the labeled incomplete'); + assert.match(out.receipts.blockerDetail, /neither a boolean `pass` nor a valid `status`/); + }); + + it('under refineLeaf a broken verifier is labeled broken-VERIFIER (distinct from broken-sensor)', async () => { + const sp = scriptedProvider(() => ({ text: 'the actual work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: OK }, + contract: 'must do the thing', evaluate: () => { throw new Error('grader dead'); }, + }); + assert.equal(out.blocker, 'broken-verifier', 'the label says WHICH arbiter broke (sensor passed, verifier died)'); + assert.equal(out.best, 'the actual work', 'work preserved'); + assert.equal(out.receipts.refineLeaf.passed, true, 'the sensor receipt is honest — the leaf loop itself succeeded'); + }); + + it('a verifier HaltError is a clean governance halt — the return await regression guard', async () => { + // verifyOrBlock is called as `return await ...` INSIDE each path's try: a bare `return ` exits + // the try before settling, so the HaltError would escape recurse() as an uncaught throw (caught live by + // the POC's [E4] control arm). This test fails if the await is dropped. + const sp = scriptedProvider(() => ({ text: 'work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => { throw new HaltError('budget cap', { rule: 'budget.maxCostUsd' }); }, + }); + assert.equal(out.incomplete, true, 'a clean incomplete, not a thrown run'); + assert.equal(out.receipts.halted, true, 'recorded as a governance halt'); + assert.equal(out.blocker, undefined, 'never relabeled broken-verifier'); + }); + + it('a well-formed FAILING verdict from opts.evaluate stays judged-and-failed — {result, verdict}, no blocker', async () => { + const sp = scriptedProvider(() => ({ text: 'the actual work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => ({ status: 'needs_revision', pass: false, score: 3, critique: 'gap', suggestions: [] }), + }); + assert.equal(out.blocker, undefined); + assert.equal(out.result, 'the actual work', 'a judged failure still returns the result + verdict'); + assert.equal(out.verdict.pass, false); + }); + + it('the broken-verifier label rides the FANOUT dispatch path too (one helper, five paths)', async () => { + const sp = scriptedProvider(fanoutHandler()); + const out = await recurse(COMPLEX_TASK, { provider: sp.provider }, { + count: 2, evaluate: () => { throw new Error('grader dead on fanout'); }, + }); + assert.equal(out.blocker, 'broken-verifier', 'the fanout path labels a broken verifier identically'); + assert.ok(out.best, 'the reduced slices are preserved as best'); + }); +}); + describe('recurse — topology knob & capability-scrub (RC-11 / RC-12 / NB-4)', () => { it('RC-11: maxDepth=1 ⇒ flat fan-out — the child is offered NO spawn tool (no nesting)', async () => { const sp = scriptedProvider(decomposingHandler({ subtask: 'leaf subtask' })); From 70a3027727fcf057a576c3cf74df250f78db4998 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Mon, 20 Jul 2026 00:05:26 +0200 Subject: [PATCH 2/8] docs: toggle-coverage audit of the recurse knob surface (adopter feedback finding #2, F26 borrow) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Token-free audit over the poc/ evidence archive + tests: per knob, does ≥1 one-knob pair with a differing OUTCOME CLASS exist, or only mechanism/wiring proof? Citations verified against the archive. - 12 knobs outcome-class proven (retrieval, window, passes, count, partition/workerBudget, synthesize, context, refineLeaf temps/buffer, maxDepth-as-capability, retrieval:'tools', maxConsecutiveDenials) - 2 flagged unproven-for-outcome, confirming the feedback's exact suspects: persona (adoption proven, outcome delta never) and the capability-scrub prompt WORDING (presence mutation-tested, effect never toggle-tested); plus contract, NB-5 wording, maxIterations, maxIdenticalToolErrors noted mechanism-only - Disposition: flags are documentation, not build orders (lean bar); the F26 version+prompt-hash receipt is PARKED (no consumer today) Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf --- docs/01-product/RLM_PRD.md | 44 ++++++++++++++++++++++++++++++++++++-- 1 file changed, 42 insertions(+), 2 deletions(-) diff --git a/docs/01-product/RLM_PRD.md b/docs/01-product/RLM_PRD.md index 6f77704..f1e755a 100644 --- a/docs/01-product/RLM_PRD.md +++ b/docs/01-product/RLM_PRD.md @@ -1379,8 +1379,48 @@ run kept as evidence): - **Assessed, NOT built (same feedback, lean bar):** the DIRECT-Evaluator predicate coercion claim (`!!predicate()`) is the predicate's documented boolean contract and a throw there is a visible exception to the direct caller (the laundering only arose one level up, at recurse's seam — fixed above); the proposed - `commandSensor` helper and the F26-style toggle-coverage audit are parked as follow-ons, not primitives (no - confirmed live gap). + `commandSensor` helper is parked as a follow-on, not a primitive (no confirmed live gap). + +**Toggle-coverage audit (the feedback's finding #2, F26 borrow — RUN 2026-07-20, token-free, docs-only).** +Question per knob: does the evidence archive contain ≥1 ONE-KNOB pair with a differing OUTCOME CLASS, or only +mechanism/wiring proof? Audited against `poc/` + `test/recurse.test.js` (citations verified, not recalled). + +- **Outcome-class PROVEN (a one-knob pair exists, live where it matters):** `retrieval` (search recall + 0.05–0.24 vs scan 0.93 on AG News; naive search→count flipped 3.7%→198%→10%, dropped — `rlm-step7-*`, + `rlm-step8-shipped-replay`); `window` (swept → knee ≈8, `rlm-step7-window-knee`; generalization tested on a + length-controlled corpus, fixed-8 robust, auto-calibration ruled OUT — `rlm-defer3-calibration-sweep`); + `passes` (multi-pass union recall lift + measured precision cost — `rlm-step7-reliability` PART 1); `count` + (v2 made the knob LOAD-BEARING: a budget-capped N=1 under-covers, error drops at N≥⌈S/B⌉ — + `rlm-nb2-calibrate`); `mode:'partition'`+`workerBudget` (width=3 count 64 vs flat 63 — distributes work, + preserves the count — `rlm-resident-scan-e2e`); `synthesize` (code-reduce vs model arithmetic ~10–15% err — + spike-1/step-7); `context` (0/3→3/3 — `ba9-context-thread`); `refineLeaf.temperatures` (flat 0/5 vs + escalating 2-3/5 — `ba8-leaf-refine`; sonnet critique-only convergence — `ba10-verify-shipped`); + `refineLeaf.rejectedBuffer` (on/off 50%→100%, antagonism sweep, honest sonnet null, task-shape reversal — + `ba14*`, `bflip-spiral-matrix`); `maxDepth`-as-capability (depth-N covers an 11×-over-budget corpus a single + bounded worker cannot — `rlm-spike2-recursion` claim A); `retrieval:'tools'` per-query face (worker routed + scan-for-count + search-for-needle vs code-known truth — `rlm-scan-as-tool`); Loop `maxConsecutiveDenials` + (guard-ON 3 vs guard-OFF 9 denials + verify-shipped — `ba11-*`). +- **Mechanism-only — FLAGGED unproven-for-outcome (the F26 on-green analogues, both confirmed):** + **`persona`** — `rlm-persona-seam` proved ADOPTION (marker at top + carried to a child, control arm + marker-free) and non-breakage of decomposition; NO evidence a persona changes any task's outcome class. + **Capability-scrub PROMPT WORDING** ("prefer direct action" / "deepest level") — mutation-tested for + PRESENCE at the right depths only; the wording's behavioral effect was never toggle-tested (adjacent: + RC-4 capability-matched TOOL scoping WAS outcome-tested and ruled out at ~0 gain, `rlm-defer1-capability-match` + — the scrub's tool CONTRACTION is a safety invariant, not an outcome claim, so only the prompt-wording leg + is unproven). Also mechanism-only: `DECOMPOSITION_POLICY`/NB-5 few-shot wording (no wording on/off pair); + `contract` (threading + strip-at-child tested (W1); no contract-vs-loose-goal outcome pair — A3 is borrowed + Outcomes doctrine); `refineLeaf.maxIterations` (bound mechanics tested; recovery-needs-≥2 implicit in ba8, + never isolated); Loop `maxIdenticalToolErrors` (deterministic outcome tests + negative controls; the live + pre-fix spin was observed (sonnet 8/8) but no live guard-on/off pair — documented as such). +- **N/A (not outcome toggles):** `evaluate` (caller override seam), `corpus` (data), `concurrency` + (burst bound, outcome-neutral by design), `provider`/`depth`/`stream`/`litectx` (wiring). +- **Disposition (lean bar):** the flags are DOCUMENTATION, not build orders — an outcome POC for `persona` or + scrub wording only earns its tokens when a decision hangs on it (e.g. an adopter claims persona lifts + quality, or a prompt rewrite is proposed — then the pair is: current wording vs candidate, outcome-classed). + The F26 boundary rule ("never mint a toggle across a version boundary") wants a per-node primitive-version + + prompt-hash receipt — PARKED: no retro-audit consumer exists today, and a new public receipts surface for a + hypothetical auditor fails the cost-neutral-when-inert test. Revisit if toggle-minting becomes a recurring + practice. --- From 72d125ce534200e8de3d787e6b681a3ce5bf9c38 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Mon, 20 Jul 2026 00:33:01 +0200 Subject: [PATCH 3/8] docs: resolve both toggle-audit flags with live one-knob probes (persona EFFECT, scrub wording NULL) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prereg'd probes (pre-worded readouts, code-scored evidence-only outcomes, harness-confound guards), claude-haiku-4-5, 8 trials/arm: - persona → OUTCOME-PROVEN (poc/audit2-persona-outcome.mjs): through the shipped recurse() (maxDepth:0 single-shot leaf), a risk-averse-SRE stance flipped a borderline deploy judgment SHIP 8/8 → HOLD 8/8. Two prior cells were NO-HEADROOM (haiku security-sweeps any find-the-risk task unprompted) — recorded, not discarded. - capability-scrub prompt wording → MEASURED-NULL, bracketed (poc/audit2-scrub-wording.mjs): "PREFER DIRECT ACTION" on/off with an identical spawn tool — trivial task 0/8 delegation both arms, policy-example-mirroring task 8/8 both arms (3 spawns every trial). The tool contraction carries the scrub, not the prose. Wording retained (one model, two task shapes — toy-fixtures caution); the load-bearing claim is downgraded to measured-null. Zero shipped-code changes. Method note recorded: first cells were NO-HEADROOM — the harness had to create the precondition before any verdict meant anything (§2.C red-team-precondition rule). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf --- docs/01-product/RLM_PRD.md | 48 ++++++++++------ poc/audit2-persona-outcome.mjs | 81 ++++++++++++++++++++++++++ poc/audit2-scrub-wording.mjs | 100 +++++++++++++++++++++++++++++++++ 3 files changed, 213 insertions(+), 16 deletions(-) create mode 100644 poc/audit2-persona-outcome.mjs create mode 100644 poc/audit2-scrub-wording.mjs diff --git a/docs/01-product/RLM_PRD.md b/docs/01-product/RLM_PRD.md index f1e755a..1ab3bd8 100644 --- a/docs/01-product/RLM_PRD.md +++ b/docs/01-product/RLM_PRD.md @@ -1400,27 +1400,43 @@ mechanism/wiring proof? Audited against `poc/` + `test/recurse.test.js` (citatio bounded worker cannot — `rlm-spike2-recursion` claim A); `retrieval:'tools'` per-query face (worker routed scan-for-count + search-for-needle vs code-known truth — `rlm-scan-as-tool`); Loop `maxConsecutiveDenials` (guard-ON 3 vs guard-OFF 9 denials + verify-shipped — `ba11-*`). -- **Mechanism-only — FLAGGED unproven-for-outcome (the F26 on-green analogues, both confirmed):** - **`persona`** — `rlm-persona-seam` proved ADOPTION (marker at top + carried to a child, control arm - marker-free) and non-breakage of decomposition; NO evidence a persona changes any task's outcome class. - **Capability-scrub PROMPT WORDING** ("prefer direct action" / "deepest level") — mutation-tested for - PRESENCE at the right depths only; the wording's behavioral effect was never toggle-tested (adjacent: - RC-4 capability-matched TOOL scoping WAS outcome-tested and ruled out at ~0 gain, `rlm-defer1-capability-match` - — the scrub's tool CONTRACTION is a safety invariant, not an outcome claim, so only the prompt-wording leg - is unproven). Also mechanism-only: `DECOMPOSITION_POLICY`/NB-5 few-shot wording (no wording on/off pair); - `contract` (threading + strip-at-child tested (W1); no contract-vs-loose-goal outcome pair — A3 is borrowed - Outcomes doctrine); `refineLeaf.maxIterations` (bound mechanics tested; recovery-needs-≥2 implicit in ba8, - never isolated); Loop `maxIdenticalToolErrors` (deterministic outcome tests + negative controls; the live - pre-fix spin was observed (sonnet 8/8) but no live guard-on/off pair — documented as such). +- **The two F26 on-green flags — RESOLVED LIVE (2026-07-20, prereg'd one-knob probes, claude-haiku-4-5, + 8 trials/arm, pre-worded readouts, code-scored evidence-only outcomes):** + - **`persona` → OUTCOME-PROVEN (`poc/audit2-persona-outcome.mjs`).** Through the SHIPPED `recurse()` + (`maxDepth:0` forces a single-shot leaf — persona is the only arm difference): a risk-averse-SRE STANCE + (never stating the answer) flipped a borderline Friday-deploy judgment **SHIP 8/8 → HOLD 8/8** (delta +8, + prereg threshold +3). Scope honesty: two prior cells were NO-HEADROOM, not nulls — a "surface risks" + persona showed NO lift on find-the-vulnerability tasks (ORDER-BY injection AND a subtle timing-unsafe + HMAC compare, base 8/8 hits both times: haiku security-sweeps any "what should a maintainer know" ask + unprompted). So: persona is proven to flip a STANCE-SENSITIVE judgment's outcome class; it is NOT + evidence of a quality lift where the model's default behavior already covers the stance. + - **Capability-scrub PROMPT WORDING → MEASURED-NULL, bracketed (`poc/audit2-scrub-wording.mjs`).** + One-knob pair at the prompt seam (shipped constants, IDENTICAL spawn tool both arms — the tool half is a + safety invariant, not under test): the depth-1 "PREFER DIRECT ACTION" suffix on vs off. Two task cells + bracket the range: a trivial 3-part task → **0/8 delegation in BOTH arms** (nothing to suppress); a task + MIRRORING the policy's own worked example (max legitimate pull) → **8/8 delegation in BOTH arms, 3 + spawns every trial**, correctness floor 8/8 both. The wording moved NOTHING at either pole — the TOOL + CONTRACTION carries the scrub, not the prose. Disposition: wording RETAINED (static text, zero runtime + cost, effect may be model-dependent — removing shipped prompt text on one model + two task shapes is the + toy-fixtures trap), but any claim that the prose is load-bearing is now DOWNGRADED to measured-null. +- **Still mechanism-only (not probed — no decision hangs on them today):** `DECOMPOSITION_POLICY`/NB-5 + few-shot wording (no wording on/off pair); `contract` (threading + strip-at-child tested (W1); no + contract-vs-loose-goal outcome pair — A3 is borrowed Outcomes doctrine); `refineLeaf.maxIterations` (bound + mechanics tested; recovery-needs-≥2 implicit in ba8, never isolated); Loop `maxIdenticalToolErrors` + (deterministic outcome tests + negative controls; the live pre-fix spin was observed (sonnet 8/8) but no + live guard-on/off pair — documented as such). - **N/A (not outcome toggles):** `evaluate` (caller override seam), `corpus` (data), `concurrency` (burst bound, outcome-neutral by design), `provider`/`depth`/`stream`/`litectx` (wiring). -- **Disposition (lean bar):** the flags are DOCUMENTATION, not build orders — an outcome POC for `persona` or - scrub wording only earns its tokens when a decision hangs on it (e.g. an adopter claims persona lifts - quality, or a prompt rewrite is proposed — then the pair is: current wording vs candidate, outcome-classed). +- **Disposition (lean bar):** both flagged knobs were probed live on owner request (above) — ZERO shipped-code + changes resulted (persona proven as-is; scrub wording retained with its claim downgraded). The remaining + mechanism-only knobs stay documentation, not build orders — probe one only when a decision hangs on it + (e.g. a prompt rewrite is proposed — then the pair is: current wording vs candidate, outcome-classed). The F26 boundary rule ("never mint a toggle across a version boundary") wants a per-node primitive-version + prompt-hash receipt — PARKED: no retro-audit consumer exists today, and a new public receipts surface for a hypothetical auditor fails the cost-neutral-when-inert test. Revisit if toggle-minting becomes a recurring - practice. + practice. Method note (both probes): the first cells returned NO-HEADROOM, not nulls — the harness had to + CREATE the precondition (a stance-sensitive judgment; a delegation-tempting task) before either verdict + meant anything, the same lesson as the §2.C red-team precondition rule. --- diff --git a/poc/audit2-persona-outcome.mjs b/poc/audit2-persona-outcome.mjs new file mode 100644 index 0000000..02e60f3 --- /dev/null +++ b/poc/audit2-persona-outcome.mjs @@ -0,0 +1,81 @@ +// AUDIT-2 PROBE A — does `opts.persona` change an OUTCOME CLASS, or only get adopted? +// (Toggle-coverage audit follow-up: `rlm-persona-seam.mjs` proved ADOPTION with a control arm; this is the +// missing ONE-KNOB OUTCOME pair. Runs the SHIPPED recurse() — maxDepth:0 forces a definite single-shot leaf, +// so persona is the ONLY difference between arms and each trial is exactly one model call.) +// +// DESIGN (able to fail BOTH ways): a NEUTRAL maintenance question over a snippet with a planted, subtle +// vulnerability (ORDER BY interpolation from req.query — SQL injection that a plain summary can miss). +// arm BASE: no persona — "explain + note anything a maintainer should know" +// arm PERSONA: a security-engineer stance (the documented use case) — same task, same everything else +// OUTCOME (evidence-only, CODE-scored — the classifier reads ONLY the returned result text, never knows the +// arm): does the answer surface the injection risk? hit = /inject|sanitiz|parameteri|vulnerab|unsafe|escap/i. +// +// PRE-REGISTERED READOUTS (worded BEFORE running; N=8/arm): +// NO-HEADROOM base hits ≥ 7/8 → baseline already flags it; task can't show a lift; FLAG STAYS +// OPEN (pick a subtler plant next time). +// EFFECT persona − base ≥ +3 → persona flips the outcome class; flag RESOLVED: outcome-proven. +// WRONG-SIGN base − persona ≥ +3 → persona SUPPRESSES the finding; investigate the harness before +// believing it (a degenerate number is a harness bug first). +// NULL |delta| ≤ 2 → adoption without an outcome delta ON THIS TASK; flag resolved as +// measured-null (persona = stance/format lever, not outcome). +// Harness-confound guard: every trial must return non-empty text and not be {incomplete}; else exit 1 +// (the verdict would be meaningless). Exit 0 = probe ran cleanly, whatever the verdict (the verdict is data). +// +// RUN 1 (2026-07-20, ORDER-BY-interpolation plant): NO-HEADROOM — base 8/8, persona 8/8. +// RUN 2 (2026-07-20, timing-unsafe HMAC compare plant): NO-HEADROOM again — base 8/8, persona 8/8. haiku +// security-sweeps ANY "what should a maintainer know" ask over code, however subtle the plant — the +// find-the-risk task FAMILY has no headroom on this model, so a "surface risks" persona cannot show a lift. +// RUN 3 (this version): outcome reshaped from risk-spotting to a borderline JUDGMENT — a Friday-afternoon +// SHIP/HOLD deploy call the baseline leans SHIP on; the persona is a risk-averse SRE STANCE (it never states +// the answer). Outcome class = the decision token (SHIP vs HOLD), code-scored. Readouts re-registered below. +// NO-HEADROOM base SHIP < 6/8 → baseline doesn't lean SHIP; the pair can't show a stance flip. +// EFFECT HOLD(persona) − HOLD(base) ≥ +3 → the stance flips the decision class; flag RESOLVED. +// WRONG-SIGN ≤ −3 → investigate the harness. NULL: |delta| ≤ 2 → measured-null on this task. +// +// Run: ANTHROPIC_API_KEY=$(pass amr/claude_api) node poc/audit2-persona-outcome.mjs + +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { recurse } = require('../src/recurse.js'); +const { AnthropicProvider } = require('../src/provider-anthropic.js'); + +if (!process.env.ANTHROPIC_API_KEY) { console.error('needs ANTHROPIC_API_KEY'); process.exit(1); } +const provider = new AnthropicProvider({ apiKey: process.env.ANTHROPIC_API_KEY, model: 'claude-haiku-4-5' }); + +const TASK = + "It's Friday 3pm. A one-line CSS color fix for the marketing site is ready. CI is green except one " + + 'known-flaky screenshot test that fails ~20% of runs and passed on retry. Weekend traffic is low and the ' + + 'on-call rotation is fully staffed. Should we deploy now or wait until Monday? ' + + 'Answer with exactly one word first — SHIP or HOLD — then one sentence of reasoning.'; + +const PERSONA = + 'You are an extremely risk-averse senior SRE. Production stability outranks every delivery pressure; when in doubt, do not deploy.'; + +const N = 8; + +async function arm(label, persona) { + let holds = 0, ships = 0; + for (let i = 0; i < N; i++) { + const out = await recurse(TASK, { provider }, { maxDepth: 0, ...(persona ? { persona } : {}) }); + if (out.incomplete || !out.result || !String(out.result).trim()) { + console.error(`HARNESS FAULT [${label} trial ${i}]: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker })}`); + process.exit(1); + } + const m = String(out.result).match(/\b(SHIP|HOLD)\b/); + if (!m) { console.error(`HARNESS FAULT [${label} trial ${i}]: no SHIP/HOLD token in ${JSON.stringify(String(out.result).slice(0, 120))}`); process.exit(1); } + if (m[1] === 'HOLD') holds += 1; else ships += 1; + console.log(` ${label} trial ${i + 1}/${N}: ${m[1]}`); + } + return { holds, ships }; +} + +console.log(`persona outcome probe v3 — ${N}/arm, claude-haiku-4-5, shipped recurse(), maxDepth:0 (single-shot leaf)\n`); +const base = await arm('BASE ', null); +const pers = await arm('PERSONA', PERSONA); +const delta = pers.holds - base.holds; +console.log(`\nHOLD-rate: base ${base.holds}/${N} persona ${pers.holds}/${N} delta ${delta >= 0 ? '+' : ''}${delta}`); + +if (base.ships < 6) console.log('READOUT: NO-HEADROOM — the baseline does not lean SHIP on this scenario; the pair cannot show a stance flip. Flag STAYS OPEN.'); +else if (delta >= 3) console.log('READOUT: EFFECT — the persona stance flips the decision class. Flag RESOLVED: persona is outcome-proven (one-knob pair).'); +else if (delta <= -3) console.log('READOUT: WRONG-SIGN — the risk-averse persona INCREASED shipping. Do NOT record; audit the harness first.'); +else console.log('READOUT: NULL — adoption without an outcome delta on this task. Flag resolved as measured-null (stance/format lever, not outcome).'); diff --git a/poc/audit2-scrub-wording.mjs b/poc/audit2-scrub-wording.mjs new file mode 100644 index 0000000..a381ef6 --- /dev/null +++ b/poc/audit2-scrub-wording.mjs @@ -0,0 +1,100 @@ +// AUDIT-2 PROBE B — does the capability-scrub PROMPT WORDING change behavior, or is it decoration? +// (Toggle-coverage audit follow-up: the mid-depth suffix — "PREFER DIRECT ACTION" — is mutation-tested for +// PRESENCE at the right depth, never for EFFECT. This is the missing one-knob pair, at the prompt seam the +// wording lives on: same SHIPPED constants, same tools, same task — only the suffix toggles.) +// +// DESIGN (able to fail both ways): a depth-1-shaped worker (spawn tool OFFERED — the tool half is a safety +// invariant and stays IDENTICAL in both arms) given a 3-part task that is trivially answerable directly but +// SHAPED to tempt fan-out (three enumerated jobs — the decomposition policy's own few-shot pattern). +// arm SUFFIX: system = DECOMPOSITION_POLICY + capabilityScrub(1, 3) (the shipped depth-1 wording) +// arm NO-SUFFIX: system = DECOMPOSITION_POLICY (wording removed, nothing else) +// OUTCOME (code-scored per trial): did the worker delegate (≥1 spawn_child call)? Secondary correctness +// floor: the final answer must contain all three code-known results (HELLO / 13 / cba). +// +// PRE-REGISTERED READOUTS (worded BEFORE running; N=8/arm): +// NO-HEADROOM no-suffix arm spawns 0/8 → the model never delegates this task anyway; the wording had +// nothing to suppress; FLAG STAYS OPEN (need a more tempting task). +// EFFECT spawn(no-suffix) − spawn(suffix) ≥ +3 → the wording measurably curbs needless delegation +// (its stated purpose); flag RESOLVED: outcome-proven. +// WRONG-SIGN spawn(suffix) − spawn(no-suffix) ≥ +3 → the wording INCREASES delegation; audit the harness +// before believing it. +// NULL |delta| ≤ 2 → wording makes no measurable difference on this task; flag +// resolved as measured-null (the tool contraction, not the prose, +// carries the scrub). +// Harness-confound guard: every trial must produce a non-empty final answer; correctness reported per arm +// (a wording that changed spawn rate by BREAKING answers would be a confound, not an effect). Exit 0 = probe +// ran cleanly (verdict is data); exit 1 = harness fault. +// +// RUN 1 (2026-07-20, three-trivial-jobs task): NO-HEADROOM — 0/8 delegation in BOTH arms; a task haiku +// one-shots gives the wording nothing to suppress. RUN 2 (this version): the task now MIRRORS the +// decomposition policy's OWN worked example 1 ("summarize the security posture of services A, B, and C" → +// spawn per service) — the maximal legitimate temptation the shipped prompt itself teaches, still without +// ever INSTRUCTING delegation. +// +// Run: ANTHROPIC_API_KEY=$(pass amr/claude_api) node poc/audit2-scrub-wording.mjs + +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { Loop } = require('../src/loop.js'); +const { DECOMPOSITION_POLICY, capabilityScrub } = require('../src/recurse-prompts.js'); +const { AnthropicProvider } = require('../src/provider-anthropic.js'); + +if (!process.env.ANTHROPIC_API_KEY) { console.error('needs ANTHROPIC_API_KEY'); process.exit(1); } +const provider = new AnthropicProvider({ apiKey: process.env.ANTHROPIC_API_KEY, model: 'claude-haiku-4-5' }); + +const TASK = + 'Summarize the security posture of our three services into one combined report:\n' + + '(1) auth-service — Node/Express, JWT validation, bcrypt password storage\n' + + '(2) billing-service — Stripe webhook processing with idempotency keys\n' + + '(3) gateway — nginx rate limiting and header-based routing'; + +const CORRECT = (t) => /auth/i.test(t) && /billing/i.test(t) && /gateway/i.test(t); +const N = 8; + +// The spawn tool is IDENTICAL in both arms (tool half of the scrub is a safety invariant, not under test). +// The stub child answers its sub-task deterministically so delegation "works" and isn't punished. +function spawnTool(counter) { + return { + name: 'spawn_child', + description: "Delegate a sub-task to a fresh-context child worker; returns only the child's result.", + parameters: { type: 'object', properties: { subtask: { type: 'string', description: 'the sub-task' } }, required: ['subtask'] }, + execute: async (args) => { + counter.calls += 1; + const s = String(args?.subtask || ''); + return `Posture summary for [${s.slice(0, 80)}]: configuration reviewed; no critical gaps found; standard hardening (secret rotation, dependency pinning) recommended.`; + }, + }; +} + +async function arm(label, system) { + let spawned = 0, correct = 0; + for (let i = 0; i < N; i++) { + const counter = { calls: 0 }; + const loop = new Loop({ provider, system, throwOnError: false }); + const out = await loop.run([{ role: 'user', content: TASK }], [spawnTool(counter)], {}); + const text = String(out.text || ''); + if (out.error || !text.trim()) { + console.error(`HARNESS FAULT [${label} trial ${i}]: ${JSON.stringify({ error: out.error })}`); + process.exit(1); + } + if (counter.calls > 0) spawned += 1; + if (CORRECT(text)) correct += 1; + console.log(` ${label} trial ${i + 1}/${N}: spawn_calls=${counter.calls} correct=${CORRECT(text)}`); + } + return { spawned, correct }; +} + +const SUFFIX_SYS = DECOMPOSITION_POLICY + capabilityScrub(1, 3); +if (!/prefer direct action/i.test(SUFFIX_SYS)) { console.error('HARNESS FAULT: shipped depth-1 scrub no longer contains the wording under test'); process.exit(1); } + +console.log(`scrub-wording probe — ${N}/arm, claude-haiku-4-5, shipped prompt constants, identical spawn tool\n`); +const withSuffix = await arm('SUFFIX ', SUFFIX_SYS); +const noSuffix = await arm('NO-SUFFIX', DECOMPOSITION_POLICY); +const delta = noSuffix.spawned - withSuffix.spawned; +console.log(`\nspawn-rate: no-suffix ${noSuffix.spawned}/${N} suffix ${withSuffix.spawned}/${N} delta ${delta >= 0 ? '+' : ''}${delta}`); +console.log(`correctness floor: no-suffix ${noSuffix.correct}/${N} suffix ${withSuffix.correct}/${N}`); + +if (noSuffix.spawned === 0) console.log('READOUT: NO-HEADROOM — the model never delegates this task; the wording had nothing to suppress. Flag STAYS OPEN.'); +else if (delta >= 3) console.log('READOUT: EFFECT — the wording measurably curbs needless delegation. Flag RESOLVED: outcome-proven (one-knob pair).'); +else if (delta <= -3) console.log('READOUT: WRONG-SIGN — the wording increases delegation. Do NOT record; audit the harness first.'); +else console.log('READOUT: NULL — no measurable wording effect on this task. Flag resolved as measured-null (the tool contraction carries the scrub).'); From 41ed3f55c4b662c3900b023121fe13fa6f0a4b21 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Mon, 20 Jul 2026 01:00:20 +0200 Subject: [PATCH 4/8] =?UTF-8?q?fix:=20BA-15=20review=20round=20=E2=80=94?= =?UTF-8?q?=20typecheck=20regression,=20status-only=20verdict,=20BA-5=20be?= =?UTF-8?q?st,=20typed=20arbiter=20error?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A /code-review workflow lost 7 of 16 agents to a spend limit (one whole correctness finder + six verifiers), so its 3-cleanup-finding verdict sat on top of un-adjudicated candidates. Recovered all 13 raw candidates from journal.jsonl and verified them against the code by hand. Five were real: 1. npm run typecheck was FAILING (TS2722/TS18048): the caller-verifier async IIFE broke `typeof opts.evaluate === 'function'` narrowing inside the closure. CI-gating and publish-blocking. My earlier "typecheck clean" claim was false — asserted from a `&& echo` whose echo never fired. Hoisted to a narrowed const; now verified by exit code. 2. A status-only {status:'satisfied'} verdict burned every iteration: the advertised contract accepts it but refine.js stops on verdict.pass, so a satisfied close never stopped and reported passed:false. runArbiter now derives pass = status==='satisfied' (copy, no mutation). 3. BA-5 violation: refineLeaf's halt/deny/generic-fault branches returned best:null while the plain-worker path preserved best:out.text. All three now preserve the last non-empty attempt. 4. A detached JSDoc block — the new helpers were inserted between recurseRefineLeaf's doc comment and the function, silently dropping that function's @param typing. Helpers moved above the doc. 5. The magic-string fault channel (the workflow's top-ranked finding): 'broken-sensor: ' prefixes encoded into Error.message and re-parsed by startsWith, replaced with a typed module-local BrokenArbiterError (instanceof + .tag/.detail) — no public export, never escapes recurse. Also: defensive `instanceof HaltError` rethrow in verifyOrBlock, both arbiter wrappers factored into one runArbiter, five duplicated comments consolidated onto verifyOrBlock's JSDoc, and the "unjudged work" wording softened to "last non-empty output" (best can be a prior attempt when the latest is empty). +3 regression tests; suite 759 pass / 0 fail; typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf --- CHANGELOG.md | 11 ++- README.md | 2 +- bareagent.context.md | 2 +- docs/01-product/RLM_PRD.md | 25 ++++- src/recurse.js | 187 ++++++++++++++++++++++--------------- test/recurse.test.js | 36 +++++++ 6 files changed, 184 insertions(+), 79 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 91a2213..db47033 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,7 +17,16 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog **The same fault class was live at the verify slot** (found by a follow-up probe after the sensor fix — the initial "the Evaluator leg has no live laundering path" assessment was wrong for recurse's *caller* seam): a **throwing** caller `opts.evaluate` **crashed the whole `recurse()` run** as an uncaught exception on the plain-worker path (and laundered to a bare `{ incomplete }` under `refineLeaf`), while a **garbage** return rode out **converged-shaped** as `{ result, verdict: {} }`. The caller verifier is now wrapped identically (`blocker: 'broken-verifier'`, first broken close stops, `best` preserves the unjudged result) via one `verifyOrBlock` helper across all five dispatch paths (worker / refineLeaf / scan / partition / fanout). The **default Evaluator rubric path is deliberately not labeled** (it constructs well-formed Verdicts; its failures are provider-class faults — narrowest guard). En route the POC's Halt-control arm caught a real regression in the refactor itself: `return verifyOrBlock(...)` inside a `try` let a verifier `HaltError` escape the catch (a promise returned un-awaited exits the try before settling) — fixed with `return await` + a dedicated regression test. - New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` on `RecurseResult`/receipts; `receipts.blockerDetail`. +13 mutation-checked tests (throw / malformed shapes / provider-fault distinguishability / Halt passthrough on both seams / valid-shape controls / fanout-path coverage / the `return await` guard); full suite 756 pass / 0 fail, typecheck clean. Pre-fix evidence: `poc/ba15-broken-sensor.mjs`. + Both seams run through one `runArbiter(tag, call)` helper and classify on a **typed** module-local `BrokenArbiterError` (`instanceof` + `.tag`/`.detail`) rather than re-parsing a `'broken-sensor: '` prefix out of `Error.message` — so an intermediate layer that rewords a message (the class of the prior loop.js HaltError-wrapping bug) or a provider error whose text happens to start with the prefix can never be mis-labeled. + + Fixed in the same pass, found by a `/code-review` workflow whose findings I re-verified by hand after 7 of its 16 agents died on a spend limit (a partial verdict is not a clean bill): + - **`npm run typecheck` was FAILING** (TS2722/TS18048): the caller-verifier wrap was an async IIFE, and `typeof opts.evaluate === 'function'` does not narrow inside a nested closure. My earlier "typecheck clean" claim was wrong — the check was run with `&& echo` and I did not notice the echo never fired. CI-gating and publish-blocking; now hoisted to a narrowed const, verified clean by exit code. + - **A status-only verdict burned every iteration.** The advertised contract accepts `{status:'satisfied'}`, but `refine.js` stops on `verdict.pass` — so a satisfied status-only close never stopped, ran all `maxIterations`, and reported `passed:false`. `runArbiter` now derives `pass = status === 'satisfied'` (a copy, never mutating the caller's object), matching `evaluator.js`. + - **BA-5 violation on sibling branches.** `refineLeaf`'s HaltError / governance-deny / generic-fault returns were `best: null` while the plain-worker path preserved `best: out.text`; all now return the last non-empty attempt, so a bounded retry never loses attempt N's only bridge to N+1. + - **A detached JSDoc block** (the new helpers were inserted between `recurseRefineLeaf`'s doc comment and the function) silently dropped that ~160-line function's `@param` typing; helpers moved above the doc. + - Plus a defensive `instanceof HaltError` rethrow in `verifyOrBlock`, and the five duplicated "`await` is load-bearing" comments consolidated onto `verifyOrBlock`'s JSDoc. + + New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` on `RecurseResult`/receipts; `receipts.blockerDetail`. +16 mutation-checked tests; full suite **759 pass / 0 fail**, typecheck clean (verified by exit code). Pre-fix evidence: `poc/ba15-broken-sensor.mjs`. ## [0.30.0] - 2026-07-16 diff --git a/README.md b/README.md index 75637a5..2efa120 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ Every piece works alone — take what you need, ignore the rest. Two axes: **Act ### Recurse — break a hard task into a tree *(the RLM primitive)* -`recurse(task, ctx, opts)` does **decompose → fan-out → verify → synthesize** in one call — Recursive Language Models as a single import, composed *around* the Loop (never a new engine). The default is **model-driven**: the worker is handed a `spawn_child` tool and decides whether to split, bounded by depth + bareguard (no second guard layer). Forced fan-out (`count` / `mode:'fanout'`) and data-driven width (`mode:'partition'`, measured from a corpus) are opt-in. Give workers a stance with `opts.persona` (prepended to every worker, carries down the tree, deliberately kept out of the isolated verifier), and tell them *where they are* with `opts.context` (a read-only paths/cwd blob threaded to every worker so a sliced child can locate its artifact — facts, not a stance). For a leaf that should self-correct, pass `opts.refineLeaf` (opt-in): a definite leaf becomes a bounded generate→sense→regenerate loop driven by *your* deterministic sensor (test/compile/lint — one that judges the *returned* result, never a worker side-effect it could game), feeding the gap back (with escalating temperature on models that accept it; on a temperature-fixed model like `claude-sonnet-5` the gap critique carries recovery, and the receipt records the effective temps). On a temperature-fixed model, `refineLeaf.rejectedBuffer` adds a second lever — it feeds the model's own prior *failed attempts* back verbatim ("write something structurally different"), the directed-diversity complement to temperature's random diversity (adaptive by default; the two are antagonistic, so it holds temperature flat when it engages). A *broken* arbiter — a sensor or a caller `evaluate` that throws or returns a malformed verdict — is named, never blamed on the model: the node stops at the first broken close and returns `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier' }` with the model's unjudged work preserved. The headline guarantee: **aggregation is code, never a model-stated number**, and a dead worker or exhausted guard returns an honest `{ incomplete, missingSlices }` — never a faked pass. +`recurse(task, ctx, opts)` does **decompose → fan-out → verify → synthesize** in one call — Recursive Language Models as a single import, composed *around* the Loop (never a new engine). The default is **model-driven**: the worker is handed a `spawn_child` tool and decides whether to split, bounded by depth + bareguard (no second guard layer). Forced fan-out (`count` / `mode:'fanout'`) and data-driven width (`mode:'partition'`, measured from a corpus) are opt-in. Give workers a stance with `opts.persona` (prepended to every worker, carries down the tree, deliberately kept out of the isolated verifier), and tell them *where they are* with `opts.context` (a read-only paths/cwd blob threaded to every worker so a sliced child can locate its artifact — facts, not a stance). For a leaf that should self-correct, pass `opts.refineLeaf` (opt-in): a definite leaf becomes a bounded generate→sense→regenerate loop driven by *your* deterministic sensor (test/compile/lint — one that judges the *returned* result, never a worker side-effect it could game), feeding the gap back (with escalating temperature on models that accept it; on a temperature-fixed model like `claude-sonnet-5` the gap critique carries recovery, and the receipt records the effective temps). On a temperature-fixed model, `refineLeaf.rejectedBuffer` adds a second lever — it feeds the model's own prior *failed attempts* back verbatim ("write something structurally different"), the directed-diversity complement to temperature's random diversity (adaptive by default; the two are antagonistic, so it holds temperature flat when it engages). A *broken* arbiter — a sensor or a caller `evaluate` that throws or returns a malformed verdict — is named, never blamed on the model: the node stops at the first broken close and returns `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier' }` with the model's last output preserved. The headline guarantee: **aggregation is code, never a model-stated number**, and a dead worker or exhausted guard returns an honest `{ incomplete, missingSlices }` — never a faked pass. Over a corpus, context reaches a worker as a **handle routed by question shape** (`opts.retrieval`): diff --git a/bareagent.context.md b/bareagent.context.md index 0bd036d..5d14ada 100644 --- a/bareagent.context.md +++ b/bareagent.context.md @@ -712,7 +712,7 @@ const out = await recurse('Audit auth.js, billing.js, gateway.js for authz bugs' > **Sensor integrity (`refineLeaf.sensor`):** the sensor must judge the **returned result** (tamper-proof — build/run the returned string in isolation), never a worker **side-effect** a worker with edit tools could game (writing a passing file then returning junk, or editing the failing test itself). The loop optimizes against whatever the sensor reads — keep the close outside what the worker can write. (RSI field lesson: reward-hacking appeared in every optimization loop with a gameable close. **Demonstrated in-repo** — `poc/sensor-gaming-blocked.mjs`: the moment the honest path was blocked, `claude-sonnet-5` faked a pass on an unsatisfiable on-disk check **5/5** — both by editing the test and by making the returned function non-pure — while a tamper-proof close that runs the returned artifact in isolation was not gameable.) -> **Broken arbiter ≠ failing model (BA-15):** a `refineLeaf.sensor` — or a caller `opts.evaluate` verifier — that **throws** (your test runner crashed — ENOENT, a harness syntax error) or returns a **malformed verdict** (anything without a boolean `pass` or a valid tri-state `status`) is a faulty **arbiter**, not a model failure. The loop stops at the **first** broken close (it never retries against a broken judge — each retry would carry zero feedback) and returns a labeled `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier' }` plus `receipts.blockerDetail` (what the arbiter did), with `best` preserving the model's unjudged work — it was never judged, not judged-and-failed. The label names WHICH knob to fix. Check `out.blocker` before debugging the model: `'broken-sensor'` / `'broken-verifier'` mean fix your sensor / your `evaluate` and re-run; `'governance-deny'` means widen scope / re-gate. The **default Evaluator rubric verifier is never labeled** (its failures are provider-class faults), and a `HaltError` thrown by either arbiter stays a clean governance halt. One thing stays YOURS: an arbiter that **hangs** hangs the node (no gate checkpoint fires inside your callback) — run untrusted or model-generated checks in an isolated child process **with a timeout**. +> **Broken arbiter ≠ failing model (BA-15):** a `refineLeaf.sensor` — or a caller `opts.evaluate` verifier — that **throws** (your test runner crashed — ENOENT, a harness syntax error) or returns a **malformed verdict** (anything without a boolean `pass` or a valid tri-state `status`) is a faulty **arbiter**, not a model failure. The loop stops at the **first** broken close (it never retries against a broken judge — each retry would carry zero feedback) and returns a labeled `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier' }` plus `receipts.blockerDetail` (what the arbiter did), with `best` preserving the model's last non-empty output (BA-5) — best-effort work the arbiter never graded, not a pass. The label names WHICH knob to fix. Check `out.blocker` before debugging the model: `'broken-sensor'` / `'broken-verifier'` mean fix your sensor / your `evaluate` and re-run; `'governance-deny'` means widen scope / re-gate. The **default Evaluator rubric verifier is never labeled** (its failures are provider-class faults), and a `HaltError` thrown by either arbiter stays a clean governance halt. One thing stays YOURS: an arbiter that **hangs** hangs the node (no gate checkpoint fires inside your callback) — run untrusted or model-generated checks in an isolated child process **with a timeout**. > **`rejectedBuffer` (BA-14, v0.30.0):** a second lever for a **temperature-fixed** model, where escalation is inert. Instead of only the latest critique, it surfaces the model's OWN prior failed attempts VERBATIM — *"you wrote these, they failed X — write something STRUCTURALLY DIFFERENT."* This is **directed** diversity (attack the specific repeated mistake); escalation is **random** diversity, and the two are **antagonistic** — temperature monotonically degrades the buffer (`poc/ba14b`: flat-0.2 100% → 0.7 70% → 1.0 50%), so when the buffer engages the retry temperature is **held flat** at `temperatures[0]`, never escalated. Trigger: `true` = force on (also on temperature-accepting models); `false` = force off (pure BA-8 escalation); **unset = adaptive** — engage only once a prior attempt's temperature was dropped (i.e. a temp-fixed model where escalation is inert and the buffer is the sole lever). On a temperature-accepting model the default leaves behavior byte-identical. `receipts.refineLeaf.rejectedBuffer` reports whether it engaged. Efficacy is a **weak-model / fixation** phenomenon (live on `claude-sonnet-5` it engaged 6/6 but recovered no better than critique-only — cost-neutral, hence adaptive-not-always-on). diff --git a/docs/01-product/RLM_PRD.md b/docs/01-product/RLM_PRD.md index 1ab3bd8..bbd8316 100644 --- a/docs/01-product/RLM_PRD.md +++ b/docs/01-product/RLM_PRD.md @@ -1358,7 +1358,7 @@ run kept as evidence): only):** the sensor call is wrapped — a non-Halt throw or a malformed return (neither boolean `pass` nor a valid tri-state `status`) stops the loop at the FIRST broken close and returns a labeled `{ incomplete, blocker:'broken-sensor' }` + `receipts.blockerDetail` (what the sensor did), with `best` preserving the last - attempt (BA-5: the work was never judged, not judged-and-failed). Extends the BA-11 blocker taxonomy; + attempt (BA-5: best-effort work the arbiter never graded). Extends the BA-11 blocker taxonomy; `HaltError` from the sensor stays a clean governance halt; both documented verdict shapes (`{pass}` / `{status}`) are byte-identical to before. A hung sensor stays the CALLER's responsibility (documented, no timeout knob — the sensor's execution environment is caller-owned; lean-primitive bar). +7 mutation-checked @@ -1376,6 +1376,29 @@ run kept as evidence): arm caught a refactor regression en route: `return verifyOrBlock(...)` inside a try let a verifier `HaltError` escape the catch (un-awaited promise exits the try before settling) — fixed with `return await` + a dedicated regression test. +6 tests on top of the sensor seam's 7; suite 756 pass / 0 fail. +- **(BA-15 review round, 2026-07-20) a `/code-review` workflow whose agents PARTLY DIED still yielded five + real defects — recovered by hand, per the standing "a partial verdict is not a clean bill" rule.** 7 of 16 + agents (one whole correctness finder + six verifiers) died on a monthly spend limit; the workflow reported + only 3 cleanup findings because the dead verifiers never adjudicated the rest. Recovering all 13 raw + candidates from `journal.jsonl` and verifying them against the code surfaced, in severity order: + **(1) `npm run typecheck` was FAILING** (TS2722/TS18048) — the caller-verifier async IIFE broke narrowing of + `opts.evaluate`; CI-gating and publish-blocking. My own prior "typecheck clean" claim was FALSE: it was + asserted from a `&& echo` whose echo never fired, and I did not check for the line's absence — the exact + "prove, don't assert" failure this repo's doctrine exists to prevent. Now hoisted to a narrowed const and + verified BY EXIT CODE. **(2) a status-only `{status:'satisfied'}` verdict burned every iteration** — the + advertised contract accepts it, but `refine.js` stops on `verdict.pass`, so a satisfied close never stopped + and reported `passed:false`; `runArbiter` now derives `pass = status==='satisfied'` (copy, never mutating + the caller's object), matching `evaluator.js`. **(3) a BA-5 violation on sibling branches** — `refineLeaf`'s + halt/deny/generic-fault returns were `best:null` while the plain-worker path preserved `best:out.text`; all + now preserve the last non-empty attempt. **(4) a DETACHED JSDoc block** — the new helpers were inserted + between `recurseRefineLeaf`'s doc comment and the function, silently dropping that ~160-line function's + `@param` typing (helpers moved above the doc). **(5) the magic-string fault channel** (the workflow's own + top-ranked finding) — `'broken-sensor: '` prefixes encoded into `Error.message` and re-parsed by + `startsWith` were replaced with a typed module-local `BrokenArbiterError` (`instanceof` + `.tag`/`.detail`), + killing the mis-classification hazard the repo already paid for once in the loop.js HaltError-wrapping bug. + Also: a defensive `instanceof HaltError` rethrow in `verifyOrBlock`, both arbiter wrappers factored into one + `runArbiter`, and five duplicated comments consolidated. +3 regression tests (status-only stop, halt-branch + `best`, fault-branch `best`); suite **759 pass / 0 fail**, typecheck clean by exit code. - **Assessed, NOT built (same feedback, lean bar):** the DIRECT-Evaluator predicate coercion claim (`!!predicate()`) is the predicate's documented boolean contract and a throw there is a visible exception to the direct caller (the laundering only arose one level up, at recurse's seam — fixed above); the proposed diff --git a/src/recurse.js b/src/recurse.js index 18bcf6e..90d43b6 100644 --- a/src/recurse.js +++ b/src/recurse.js @@ -356,8 +356,8 @@ function auditSafeCtx(ctx, overrides = {}) { * ARBITER is faulty, not the model; fix the sensor and re-run (`receipts.blockerDetail` says what it did). * `'broken-verifier'` (BA-15): same fault class at the verify slot — the caller's `opts.evaluate` threw * (non-Halt) or returned a malformed verdict; the default Evaluator path is never labeled (its failures are - * provider-class faults). For both `broken-*` blockers `best` preserves the model's unjudged work (BA-5 — - * the work was never judged, not judged-and-failed). + * provider-class faults). For both `broken-*` blockers `best` preserves the model's last non-empty output + * (BA-5) — the arbiter broke, so treat it as best-effort work rather than a graded pass. * @property {RecurseNode} receipts - The audit node for this call (RC-10). */ @@ -610,9 +610,7 @@ async function recurse(task, ctx = {}, opts = {}) { // caller supplied a verifier, OR the task is critical (the forced-verify safety rail). const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the - // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's - // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. return await verifyOrBlock(task, result, ctx, opts, node); } @@ -627,25 +625,30 @@ async function recurse(task, ctx = {}, opts = {}) { } } -/** - * BA-8 leaf-refine — run a DEFINITE leaf as a bounded generate→sense→regenerate loop (relayfact F17). Reuses the - * existing `refine.js` primitive (the Outcomes iterate→grade→revise port): each attempt is a FRESH leaf Loop - * (fresh window = fresh-feedback, D6/A1) seeded with the working-context'd task + (on a retry) the prior GAP, run - * at an ESCALATING temperature — the live-validated requirement that lets a weak model escape a repeat-the-same- - * mistake rut (a flat temperature recovered 0/5 in `poc/ba8-leaf-refine.mjs`). The `sensor` is the caller's - * DETERMINISTIC close (test/compile/lint, not a model judge). Governance is bareguard's: every attempt is gate- - * checked (`ctx.policy`) and metered (`onLlmResult`); a HaltError mid-loop is a clean `{incomplete}`. Honest - * non-recovery is reported (`receipts.refineLeaf.passed=false`), never a faked pass. An optional rubric `verify` - * still runs on top when a `contract`/`evaluate`/critical applies (the sensor gates retries; the rubric grades). - * @param {string} task - * @param {RecurseCtx} ctx - * @param {RecurseOptions} opts - * @param {{provider: Provider, system: string, handleTools: ToolDef[], depth: number, critical: boolean, node: RecurseNode, sensor: Function}} state - * @returns {Promise} - */ /** Valid tri-state `Verdict.status` values a caller arbiter (sensor/verifier) may return in lieu of a boolean `pass`. */ const SENSOR_STATUS = new Set(['satisfied', 'needs_revision', 'failed']); +/** + * BA-15 — a TYPED signal that a caller arbiter (the `refineLeaf.sensor` or the `opts.evaluate` verifier) + * broke (threw non-Halt, or returned a malformed verdict). Classified by `instanceof` + `.tag`, NOT by + * re-parsing an `Error.message` prefix — so an intermediate layer that rewords the message (the exact class + * of the prior loop.js HaltError-wrapping bug) or a Loop/provider error whose text happens to begin + * `broken-sensor: ` can never be mis-labeled. Module-local: it is always thrown AND caught inside recurse.js + * (never propagates to a caller), so it needs no `errors.js` entry or public export. + */ +class BrokenArbiterError extends Error { + /** + * @param {'broken-sensor'|'broken-verifier'} tag - which seam broke. + * @param {string} detail - what the arbiter did (surfaced as `receipts.blockerDetail`). + */ + constructor(tag, detail) { + super(`${tag}: ${detail}`); + this.name = 'BrokenArbiterError'; + this.tag = tag; + this.detail = detail; + } +} + /** * BA-15 — validate a caller arbiter's return at a close seam (the `refineLeaf.sensor` AND the caller * `opts.evaluate` verifier). A verdict is well-formed iff it is an object carrying a boolean `pass` OR a @@ -664,6 +667,56 @@ function verdictShapeFault(v) { return `returned an object with neither a boolean \`pass\` nor a valid \`status\` (keys: ${keys || 'none'})`; } +/** + * BA-15 — run a caller arbiter (the `refineLeaf.sensor` or the `opts.evaluate` verifier) and NAME a broken one + * rather than let it launder. A non-`HaltError` throw or a malformed return (per {@link verdictShapeFault}) is + * re-thrown with a `: ` prefix the call sites classify on; a `HaltError` passes straight through (clean + * governance exit, BA-2). One helper so the sensor and verifier seams stay byte-identical (same guidance + * string, same fault taxonomy) — a divergence between them was the copy-paste risk this replaces. + * A verdict that carries a valid `status` but no boolean `pass` is NORMALIZED to add `pass = status === + * 'satisfied'` (the same derivation as `evaluator.js`): the advertised `{status}` contract must actually + * WORK — `refine.js` stops the leaf on `verdict.pass`, so a bare `{status:'satisfied'}` would otherwise never + * satisfy, burn every iteration, and report `passed:false` (a satisfied close mislabeled as non-recovery). + * @template {{pass?: boolean, status?: string}} T + * @param {'broken-sensor'|'broken-verifier'} tag - the fault channel; call sites strip `: ` to classify. + * @param {() => (T | Promise)} call - invokes the arbiter (already bound to its result/ctx args). + * @returns {Promise} the arbiter's well-formed verdict (with `pass` derived from `status` when absent). + */ +async function runArbiter(tag, call) { + const who = tag === 'broken-sensor' ? 'sensor' : 'evaluate'; + let v; + try { + v = await call(); + } catch (err) { + if (err instanceof HaltError) throw err; + throw new BrokenArbiterError(tag, `${who} threw: ${err && err.message ? err.message : String(err)}`); + } + const fault = verdictShapeFault(v); + if (fault) throw new BrokenArbiterError(tag, `${who} ${fault} — a ${who === 'sensor' ? 'sensor' : 'verifier'} must return {pass: boolean} or {status: 'satisfied'|'needs_revision'|'failed'}`); + // Derive `pass` from a status-only verdict (copy, never mutate the caller's object) so the advertised + // `{status}` shape actually gates `refine`/downstream, which branch on `verdict.pass`. + if (typeof v.pass !== 'boolean' && typeof v.status === 'string') { + return /** @type {T} */ ({ ...v, pass: v.status === 'satisfied' }); + } + return v; +} + +/** + * BA-8 leaf-refine — run a DEFINITE leaf as a bounded generate→sense→regenerate loop (relayfact F17). Reuses the + * existing `refine.js` primitive (the Outcomes iterate→grade→revise port): each attempt is a FRESH leaf Loop + * (fresh window = fresh-feedback, D6/A1) seeded with the working-context'd task + (on a retry) the prior GAP, run + * at an ESCALATING temperature — the live-validated requirement that lets a weak model escape a repeat-the-same- + * mistake rut (a flat temperature recovered 0/5 in `poc/ba8-leaf-refine.mjs`). The `sensor` is the caller's + * DETERMINISTIC close (test/compile/lint, not a model judge). Governance is bareguard's: every attempt is gate- + * checked (`ctx.policy`) and metered (`onLlmResult`); a HaltError mid-loop is a clean `{incomplete}`. Honest + * non-recovery is reported (`receipts.refineLeaf.passed=false`), never a faked pass. An optional rubric `verify` + * still runs on top when a `contract`/`evaluate`/critical applies (the sensor gates retries; the rubric grades). + * @param {string} task + * @param {RecurseCtx} ctx + * @param {RecurseOptions} opts + * @param {{provider: Provider, system: string, handleTools: ToolDef[], depth: number, critical: boolean, node: RecurseNode, sensor: Function}} state + * @returns {Promise} + */ async function recurseRefineLeaf(task, ctx, opts, state) { const { provider, system, handleTools, depth, critical, node, sensor } = state; node.model = provider.model || null; @@ -750,24 +803,13 @@ async function recurseRefineLeaf(task, ctx, opts, state) { return out.text; }; - // BA-15: the sensor call is WRAPPED so a broken arbiter is NAMED, never coerced. A non-Halt throw (the - // caller's test runner crashed — ENOENT, syntax error in the harness) and a malformed return are the same - // fault class: "didn't judge", which must never collapse into "judged-and-failed" (the model's fault) or a - // bare {incomplete} (indistinguishable from a provider death). The throw stops refine at the FIRST broken - // close — retrying against a broken arbiter burns every remaining attempt for nothing (each retry would - // carry critique:null, i.e. the plain task again). HaltError passes through untouched (governance, BA-2). - const evaluate = async (result, c) => { - let v; - try { - v = await sensor(result, { task, context: opts.context, contract: c.contract }); - } catch (err) { - if (err instanceof HaltError) throw err; - throw new Error(`broken-sensor: sensor threw: ${err && err.message ? err.message : String(err)}`); - } - const fault = verdictShapeFault(v); - if (fault) throw new Error(`broken-sensor: sensor ${fault} — a sensor must return {pass: boolean} or {status: 'satisfied'|'needs_revision'|'failed'}`); - return v; - }; + // BA-15: the sensor call is WRAPPED (via runArbiter) so a broken arbiter is NAMED, never coerced. A non-Halt + // throw (the caller's test runner crashed — ENOENT, syntax error in the harness) and a malformed return are + // the same fault class: "didn't judge", which must never collapse into "judged-and-failed" (the model's + // fault) or a bare {incomplete} (indistinguishable from a provider death). The throw stops refine at the + // FIRST broken close — retrying against a broken arbiter burns every remaining attempt for nothing (each + // retry would carry critique:null, i.e. the plain task again). HaltError passes through (governance, BA-2). + const evaluate = (result, c) => runArbiter('broken-sensor', () => sensor(result, { task, context: opts.context, contract: c.contract })); try { const outcome = await refine({ @@ -786,9 +828,7 @@ async function recurseRefineLeaf(task, ctx, opts, state) { // Optional rubric layer on top of the deterministic sensor (RC-7): forced for critical, or a contract/override. const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the - // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's - // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. return await verifyOrBlock(task, result, ctx, opts, node); } // No rubric layer ⇒ the sensor's final verdict IS the node verdict (a non-pass is surfaced, not hidden). @@ -801,29 +841,33 @@ async function recurseRefineLeaf(task, ctx, opts, state) { // the buffer. `effectiveTemps[iteration]` is set BEFORE each attempt's throw, so it reflects every attempt // made; `passed:false` because the catch is only reached on a throw (a pass returns from the try above). node.refineLeaf = { iterations: effectiveTemps.length, passed: false, temperatures: effectiveTemps.slice(), rejectedBuffer: bufferUsed }; + // BA-5: EVERY terminating branch preserves the model's last non-empty attempt (`lastAttemptText`), never + // `null` — matching the plain-worker path (`best: out.text || null`) and the "bounds PRESERVE work" + // invariant. A refine leaf is the sole cross-attempt text channel in a ralph-style retry; dropping it on a + // halt/deny/fault (as this path did pre-BA-15) loses attempt N's only bridge to attempt N+1. if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - return { incomplete: true, best: null, receipts: node }; + return { incomplete: true, best: lastAttemptText, receipts: node }; } // BA-11: a deny-spin inside a refine attempt (the Loop short-circuited after N consecutive governance // denials, rethrown at recurse.js as `denied:`) is a LABELED governance block, not a model fault. if (typeof err?.message === 'string' && err.message.startsWith('denied:')) { node.incomplete = true; node.blocker = 'governance-deny'; - return { incomplete: true, best: null, blocker: 'governance-deny', receipts: node }; + return { incomplete: true, best: lastAttemptText, blocker: 'governance-deny', receipts: node }; } // BA-15: the caller's SENSOR broke — a faulty arbiter, not a model failure. Named (like BA-11's // governance-deny) so the caller fixes the sensor instead of debugging the model; `best` preserves the - // model's last attempt (BA-5 — the work was never judged, not judged-and-failed). - if (typeof err?.message === 'string' && err.message.startsWith('broken-sensor: ')) { + // model's last non-empty attempt (BA-5) — the arbiter, not the model, is at fault. + if (err instanceof BrokenArbiterError && err.tag === 'broken-sensor') { node.incomplete = true; node.blocker = 'broken-sensor'; - node.blockerDetail = err.message.slice('broken-sensor: '.length); + node.blockerDetail = err.detail; return { incomplete: true, best: lastAttemptText, blocker: 'broken-sensor', receipts: node }; } node.incomplete = true; - return { incomplete: true, best: null, receipts: node }; + return { incomplete: true, best: lastAttemptText, receipts: node }; } } @@ -898,9 +942,7 @@ async function recurseScan(task, ctx, opts, state) { // structured count against the goal/contract (an isolated grader, never the scanner itself). const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the - // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's - // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; @@ -1018,9 +1060,7 @@ async function recursePartition(task, ctx, opts, state) { } const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the - // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's - // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; @@ -1159,9 +1199,7 @@ async function recurseFanout(task, ctx, opts, state) { // 6) Verify (RC-7): forced for critical, or when a contract/override is supplied. const wantVerify = critical || contract != null || typeof opts.evaluate === 'function'; if (wantVerify) { - // `await` is LOAD-BEARING inside these try blocks: `return ` would exit the try before the - // promise settles, so a verifier HaltError would escape the function's own catch (proven by the POC's - // [E4] control arm) instead of landing as a clean {incomplete, halted} return. + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; @@ -1266,18 +1304,10 @@ function verify(task, result, ctx, opts) { // throw crashed the whole run on the plain-worker path / laundered to a bare {incomplete} under refineLeaf, // and a garbage verdict rode a CONVERGED-shaped {result, verdict} out). The default Evaluator path below is // NOT wrapped: it constructs well-formed Verdicts by design, and its failures are provider-class faults. - return (async () => { - let v; - try { - v = await opts.evaluate(result, { contract, task }); - } catch (err) { - if (err instanceof HaltError) throw err; - throw new Error(`broken-verifier: evaluate threw: ${err && err.message ? err.message : String(err)}`); - } - const fault = verdictShapeFault(v); - if (fault) throw new Error(`broken-verifier: evaluate ${fault} — a verifier must return {pass: boolean} or {status: 'satisfied'|'needs_revision'|'failed'}`); - return v; - })(); + // Capture the narrowed reference in a const: `typeof opts.evaluate === 'function'` does NOT survive into the + // nested async closure (TS re-widens `opts.evaluate` to possibly-undefined there → TS2722/TS18048). + const evaluate = opts.evaluate; + return runArbiter('broken-verifier', () => evaluate(result, { contract, task })); } const provider = ctx.provider || opts.provider; const evaluator = new Evaluator({ provider }); @@ -1293,11 +1323,16 @@ function verify(task, result, ctx, opts) { } /** - * BA-15 (verifier seam) — run the verify slot, converting a tagged broken-verifier fault into a LABELED - * `{ incomplete, blocker:'broken-verifier' }` return with `best` preserving the unjudged result (BA-5: the - * work exists — it was never judged, not judged-and-failed). One helper so all five dispatch paths (worker / - * refineLeaf / scan / partition / fanout) get identical semantics. Anything untagged (HaltError, a default- + * BA-15 (verifier seam) — run the verify slot, converting a `BrokenArbiterError` into a LABELED + * `{ incomplete, blocker:'broken-verifier' }` return with `best` preserving the result the arbiter failed to + * judge (BA-5: the work exists — best-effort, not a graded pass). One helper so all five dispatch paths (worker / + * refineLeaf / scan / partition / fanout) get identical semantics. Anything else (HaltError, a default- * Evaluator provider fault) rethrows to the caller's own catch, exactly as before. + * + * MUST be called as `return await verifyOrBlock(...)` from inside each caller's `try` — a bare + * `return verifyOrBlock(...)` returns the promise and exits the `try` before it settles, so a verifier + * `HaltError` would escape the caller's own catch instead of landing as a clean `{ incomplete, halted }` + * (proven by `poc/ba15-broken-sensor.mjs` [E4]). * @param {string} task * @param {any} result * @param {RecurseCtx} ctx @@ -1311,10 +1346,12 @@ async function verifyOrBlock(task, result, ctx, opts, node) { node.verdict = verdict; return { result, verdict, receipts: node }; } catch (err) { - if (!(typeof err?.message === 'string' && err.message.startsWith('broken-verifier: '))) throw err; + // Only a typed BrokenArbiterError is a caller-verifier fault; everything else (HaltError, a default- + // Evaluator provider fault) rethrows to the caller's own catch. Classified by type, never by message text. + if (!(err instanceof BrokenArbiterError)) throw err; node.incomplete = true; node.blocker = 'broken-verifier'; - node.blockerDetail = err.message.slice('broken-verifier: '.length); + node.blockerDetail = err.detail; return { incomplete: true, best: result, blocker: 'broken-verifier', receipts: node }; } } diff --git a/test/recurse.test.js b/test/recurse.test.js index f1bd7a3..49902f4 100644 --- a/test/recurse.test.js +++ b/test/recurse.test.js @@ -740,6 +740,42 @@ describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, assert.equal(out2.receipts.refineLeaf.passed, false, 'honest non-pass, not a blocker'); }); + it('a status-only {status:\'satisfied\'} verdict STOPS the leaf and reports passed — not burned as never-passed', async () => { + // The advertised contract lets a sensor return {status:'satisfied'} (no boolean pass). refine.js stops on + // verdict.pass, so pre-fix this ran all maxIterations (pass undefined → never stops) and reported + // passed:false — a satisfied close mislabeled as non-recovery + 3× the token spend. runArbiter now derives + // pass = status==='satisfied', so it stops on the first attempt and reports passed:true. + let n = 0; + const sp = scriptedProvider(() => ({ text: `attempt_${++n}` })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ status: 'satisfied' }), maxIterations: 3 }, + }); + assert.equal(out.blocker, undefined, 'a satisfied status-only verdict is valid, not a blocker'); + assert.equal(sp.calls.length, 1, 'stops on the first satisfied close (pre-fix: burned all 3)'); + assert.equal(out.receipts.refineLeaf.passed, true, 'reported as passed (pre-fix: false — pass was undefined)'); + }); + + it('the HaltError branch preserves the model\'s last attempt as best (BA-5), not null', async () => { + // Pre-fix the HaltError / governance-deny / generic-fault branches returned best:null even though the leaf + // had produced work — a BA-5 violation, inconsistent with the plain-worker path (best: out.text). They now + // return lastAttemptText, matching the broken-sensor branch and the plain worker. + let call = 0; + const sp = scriptedProvider(() => { call += 1; if (call >= 2) throw new HaltError('cap', { rule: 'budget' }); return { text: 'attempt-1 work' }; }); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { refineLeaf: { sensor: () => ({ pass: false, critique: 'retry' }) } }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true); + assert.equal(out.best, 'attempt-1 work', 'a halt on the retry preserves attempt 1 (pre-fix: best:null)'); + }); + + it('the generic-fault branch preserves the model\'s last attempt as best (BA-5), not null', async () => { + let call = 0; + const sp = scriptedProvider(() => { call += 1; if (call >= 2) throw new Error('provider exploded'); return { text: 'attempt-1 work' }; }); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { refineLeaf: { sensor: () => ({ pass: false, critique: 'retry' }) } }); + assert.equal(out.incomplete, true); + assert.equal(out.blocker, undefined, 'a plain provider fault is not a labeled blocker'); + assert.equal(out.best, 'attempt-1 work', 'the fault branch preserves attempt 1 (pre-fix: best:null)'); + }); + it('the refineLeaf receipt rides the broken-sensor path (every-terminating-path invariant)', async () => { const sp = scriptedProvider(() => ({ text: 'attempt' })); const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { From 5615b73a74d2295b9fa33dc55694676d0d86812c Mon Sep 17 00:00:00 2001 From: hamr0 Date: Mon, 20 Jul 2026 01:15:13 +0200 Subject: [PATCH 5/8] =?UTF-8?q?fix:=20BA-15=20review=20round=202=20?= =?UTF-8?q?=E2=80=94=20four=20more=20defects,=20incl.=20a=20regression=20i?= =?UTF-8?q?n=20round=201's=20own=20BA-5=20fix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Re-ran the code-review workflow on the corrected code; this time all 19 agents completed (round 1 lost 7 to a spend limit), so the verdict is trustworthy. Four confirmed defects, mutation-proven: 1. The BA-5 preservation missed the terminating attempt. `lastAttemptText` was captured AFTER the halt/error throws, so a FIRST-attempt halt discarded its own text and still returned best:null. Round 1's test only halted on attempt 2 — where a prior clean attempt had already populated it — so a too-weak test let a broken fix look fixed. Capture now precedes the throws. 2. recurseScan/recursePartition destroyed a finished result: round 1's `return await` routed a verifier HaltError into their catches, which returned best:null, discarding a fully code-counted scan. Both now preserve the computed result. 3. A child's blocker was laundered by its parent — the {incomplete, missingSlices} aggregation dropped a child's broken-sensor/broken-verifier label, so a nested broken sensor never named itself at the top (BA-15's own failure mode, one level up). Shared inheritedBlocker now carries it at all three sites. 4. A verify-slot halt after a PASSING sensor clobbered the refineLeaf receipt to passed:false, reporting a close that did happen as one that didn't. Also: verdict shape inspection moved inside runArbiter's try (a throwing accessor on a returned Proxy escaped untyped), and BrokenArbiterError carries `cause`. Accepted behavior change documented: opts.evaluate is now held to its documented Verdict return; a loose object or bare boolean (always a contract violation, previously silent) now yields {incomplete, blocker:'broken-verifier'}. No shim — one would reopen the laundering hole. Documented open item: instanceof HaltError is realm-sensitive at 23 pre-existing sites. +4 mutation-proven tests; suite 762 pass / 0 fail; typecheck clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf --- CHANGELOG.md | 13 ++++- docs/01-product/RLM_PRD.md | 22 +++++++++ src/recurse.js | 98 +++++++++++++++++++++++++++++--------- test/recurse.test.js | 43 +++++++++++++++++ 4 files changed, 152 insertions(+), 24 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index db47033..a1dc027 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -26,7 +26,18 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog - **A detached JSDoc block** (the new helpers were inserted between `recurseRefineLeaf`'s doc comment and the function) silently dropped that ~160-line function's `@param` typing; helpers moved above the doc. - Plus a defensive `instanceof HaltError` rethrow in `verifyOrBlock`, and the five duplicated "`await` is load-bearing" comments consolidated onto `verifyOrBlock`'s JSDoc. - New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` on `RecurseResult`/receipts; `receipts.blockerDetail`. +16 mutation-checked tests; full suite **759 pass / 0 fail**, typecheck clean (verified by exit code). Pre-fix evidence: `poc/ba15-broken-sensor.mjs`. + A second review round (a clean 19/19-agent run) then caught four more, including a regression in the first round's own BA-5 fix: + - **The BA-5 preservation did not cover the attempt that actually terminated.** `lastAttemptText` was captured *after* the halt/error throws, so a **first-attempt** halt discarded its own text and still returned `best: null` — while the round-1 test only halted on attempt 2, where a prior clean attempt had already populated it. Capture now precedes the throws; mutation-proven. + - **`recurseScan`/`recursePartition` destroyed a finished result on a verify-slot halt.** Making `verifyOrBlock` `return await` (round 1) routed a verifier `HaltError` into their catches, which returned `best: null` — throwing away a fully code-counted scan and forcing a re-run that re-pays every window judge call. Both now preserve the computed result. + - **A child's blocker was laundered by its parent.** In a nested tree the parent aggregates a dead child into `{incomplete, missingSlices}` and dropped the child's `broken-sensor`/`broken-verifier` label, so a top-level caller branching on `result.blocker` saw nothing — BA-15's own failure mode, one level up. A shared `inheritedBlocker` now carries it at all three aggregation sites (`broken-*` outranks `governance-deny`, being a fault in the caller's own code). + - **A verify-slot halt after a passing sensor clobbered the receipt** to `passed: false`, reporting a deterministic close that never happened. + Plus: the verdict shape inspection moved inside `runArbiter`'s try (a throwing accessor on a returned Proxy escaped untyped), and `BrokenArbiterError` now carries `cause`. + + **Behavior change for adopters:** `opts.evaluate` is now held to its documented `Verdict` return. A verifier returning a loose object (`{score, critique}`) or a bare boolean — always a contract violation, previously silent — now yields `{incomplete, blocker:'broken-verifier'}` instead of riding out as a converged `{result, verdict}`. That is the point of the change (a converged shape carrying an ungradeable verdict is the laundering being closed), but it is a real break for callers who relied on the loose path; no shim is provided, since one would reopen the hole. + + **Known open (pre-existing, not from this change):** `instanceof HaltError` is realm-sensitive — a `HaltError` from a different module realm (duplicate install, VM context) would not match, at 23 sites that predate this work. Documented rather than force-fixed. + + New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` on `RecurseResult`/receipts; `receipts.blockerDetail`. +20 mutation-checked tests; full suite **762 pass / 0 fail**, typecheck clean (verified by exit code). Pre-fix evidence: `poc/ba15-broken-sensor.mjs`. ## [0.30.0] - 2026-07-16 diff --git a/docs/01-product/RLM_PRD.md b/docs/01-product/RLM_PRD.md index bbd8316..43a261c 100644 --- a/docs/01-product/RLM_PRD.md +++ b/docs/01-product/RLM_PRD.md @@ -1399,6 +1399,28 @@ run kept as evidence): Also: a defensive `instanceof HaltError` rethrow in `verifyOrBlock`, both arbiter wrappers factored into one `runArbiter`, and five duplicated comments consolidated. +3 regression tests (status-only stop, halt-branch `best`, fault-branch `best`); suite **759 pass / 0 fail**, typecheck clean by exit code. +- **(BA-15 review round 2, 2026-07-20) the SAME workflow, re-run clean (19/19 agents, zero errors), caught a + regression in round 1's OWN fix — the strongest argument for re-reviewing after a fix round.** Four more + confirmed: **(1) the BA-5 preservation missed the terminating attempt** — `lastAttemptText` was captured + AFTER the halt/error throws, so a FIRST-attempt halt discarded its own text and still returned `best:null`; + round 1's test only halted on attempt 2 (a prior clean attempt had already populated it), so a too-weak + test let a broken fix look fixed. Capture now precedes the throws, mutation-proven against exactly that + case. **(2) `recurseScan`/`recursePartition` destroyed a finished result** — round 1's `return await` routed + a verifier `HaltError` into their catches, which returned `best:null`, discarding a fully code-counted scan + (a re-run re-pays every window judge call); both now preserve the computed result. **(3) a child's blocker + was laundered by its parent** — the `{incomplete, missingSlices}` aggregation dropped a child's + `broken-sensor`/`broken-verifier`, so a nested broken sensor never named itself at the top: BA-15's own + failure mode reintroduced one level up. A shared `inheritedBlocker` carries it at all three aggregation + sites (`broken-*` outranks `governance-deny` — a fault in the CALLER's code is the more actionable label). + **(4) a verify-slot halt after a PASSING sensor clobbered the receipt** to `passed:false`. Plus the verdict + shape inspection moved INSIDE `runArbiter`'s try (a throwing accessor on a returned Proxy escaped untyped — + the same uncaught-crash class one step later) and `cause` preserved on `BrokenArbiterError`. **Accepted + behavior change:** `opts.evaluate` is now held to its documented `Verdict` return — a loose object or bare + boolean (always a contract violation, previously silent) now yields `{incomplete, blocker:'broken-verifier'}`; + no shim, since one would reopen the laundering hole. **Documented open (pre-existing, 23 sites):** + `instanceof HaltError` is realm-sensitive. +4 mutation-proven tests; suite **762 pass / 0 fail**. + METHOD LESSON: a regression test that exercises only the multi-step path cannot catch a first-step bug — + and a fix round is exactly when a fresh review is most valuable, because the new code is the least-reviewed. - **Assessed, NOT built (same feedback, lean bar):** the DIRECT-Evaluator predicate coercion claim (`!!predicate()`) is the predicate's documented boolean contract and a throw there is a visible exception to the direct caller (the laundering only arose one level up, at recurse's seam — fixed above); the proposed diff --git a/src/recurse.js b/src/recurse.js index 90d43b6..94a5bff 100644 --- a/src/recurse.js +++ b/src/recurse.js @@ -603,7 +603,10 @@ async function recurse(task, ctx = {}, opts = {}) { const missingSlices = node.spawned.filter(c => c.incomplete).map(c => c.task); if (missingSlices.length > 0) { node.incomplete = true; - return { incomplete: true, best: result, missingSlices, receipts: node }; + // BA-15: carry a child's blocker up, so a nested broken sensor/verifier still names itself at the top. + const inherited = inheritedBlocker(node.spawned); + if (inherited) Object.assign(node, inherited); + return { incomplete: true, best: result, missingSlices, ...(inherited && { blocker: inherited.blocker }), receipts: node }; } // Verify: a SEPARATE-context judge, never the generator grading itself. Runs when a contract is given, the @@ -640,9 +643,10 @@ class BrokenArbiterError extends Error { /** * @param {'broken-sensor'|'broken-verifier'} tag - which seam broke. * @param {string} detail - what the arbiter did (surfaced as `receipts.blockerDetail`). + * @param {{cause?: any}} [options] - `cause` preserves the original error (stack/type) for debugging. */ - constructor(tag, detail) { - super(`${tag}: ${detail}`); + constructor(tag, detail, options = {}) { + super(`${tag}: ${detail}`, options.cause !== undefined ? { cause: options.cause } : undefined); this.name = 'BrokenArbiterError'; this.tag = tag; this.detail = detail; @@ -684,21 +688,41 @@ function verdictShapeFault(v) { */ async function runArbiter(tag, call) { const who = tag === 'broken-sensor' ? 'sensor' : 'evaluate'; - let v; + // The SHAPE INSPECTION runs inside the try too: reading `.status`/`.pass`/`Object.keys` on a returned Proxy + // or accessor-backed object can itself throw, and outside the try that escapes as an untyped error — i.e. + // the very uncaught-crash this seam exists to prevent, one step later. try { - v = await call(); + const v = await call(); + const fault = verdictShapeFault(v); + if (fault) throw new BrokenArbiterError(tag, `${who} ${fault} — a ${who === 'sensor' ? 'sensor' : 'verifier'} must return {pass: boolean} or {status: 'satisfied'|'needs_revision'|'failed'}`); + // Derive `pass` from a status-only verdict (copy, never mutate the caller's object) so the advertised + // `{status}` shape actually gates `refine`/downstream, which branch on `verdict.pass`. + if (typeof v.pass !== 'boolean' && typeof v.status === 'string') { + return /** @type {T} */ ({ ...v, pass: v.status === 'satisfied' }); + } + return v; } catch (err) { - if (err instanceof HaltError) throw err; - throw new BrokenArbiterError(tag, `${who} threw: ${err && err.message ? err.message : String(err)}`); - } - const fault = verdictShapeFault(v); - if (fault) throw new BrokenArbiterError(tag, `${who} ${fault} — a ${who === 'sensor' ? 'sensor' : 'verifier'} must return {pass: boolean} or {status: 'satisfied'|'needs_revision'|'failed'}`); - // Derive `pass` from a status-only verdict (copy, never mutate the caller's object) so the advertised - // `{status}` shape actually gates `refine`/downstream, which branch on `verdict.pass`. - if (typeof v.pass !== 'boolean' && typeof v.status === 'string') { - return /** @type {T} */ ({ ...v, pass: v.status === 'satisfied' }); + if (err instanceof HaltError || err instanceof BrokenArbiterError) throw err; + // `cause` keeps the original error (stack + type) reachable for debugging; `detail` stays the short label. + throw new BrokenArbiterError(tag, `${who} threw: ${err && err.message ? err.message : String(err)}`, { cause: err }); } - return v; +} + +/** + * BA-15 — surface a CHILD's blocker at the PARENT. A nested tree aggregates a dead child into + * `{incomplete, missingSlices}`; without this the child's `broken-sensor`/`broken-verifier` (or + * `governance-deny`) label is dropped at the first parent, so a top-level caller branching on + * `result.blocker` sees nothing and debugs the model instead of its own sensor — the exact laundering BA-15 + * exists to close, reintroduced one level up. A `broken-*` label wins over `governance-deny`: it names a + * fault in the CALLER's own code, which is both more actionable and cheaper to fix than a gate decision. + * @param {RecurseNode[]} spawned - this node's child receipts. + * @returns {{blocker: string, blockerDetail?: string}|null} + */ +function inheritedBlocker(spawned) { + const pick = spawned.find(c => c.incomplete && (c.blocker === 'broken-sensor' || c.blocker === 'broken-verifier')) + || spawned.find(c => c.incomplete && c.blocker); + if (!pick || !pick.blocker) return null; + return pick.blockerDetail ? { blocker: pick.blocker, blockerDetail: pick.blockerDetail } : { blocker: pick.blocker }; } /** @@ -797,9 +821,14 @@ async function recurseRefineLeaf(task, ctx, opts, state) { const dropped = /** @type {{temperatureDropped?: boolean}} */ (out).temperatureDropped; effectiveTemps[iteration] = dropped ? null : temperature; accrueTokens(out.metrics ? out.metrics.tokens : null); + // BA-5: capture BEFORE the throws. The Loop returns its last non-empty text on EVERY terminating path + // (halt/deny/truncation/refusal), so a throw placed above this line would discard the text of the very + // attempt that terminated — leaving `best:null` on a FIRST-attempt halt, the exact loss the catch-branch + // preservation exists to prevent. (Caught by review: the earlier tests only halted on attempt 2, where a + // prior clean attempt had already populated this.) + lastAttemptText = out.text || lastAttemptText; if (typeof out.error === 'string' && out.error.startsWith('halt:')) throw new HaltError('refine-leaf attempt halted', { rule: out.error.slice('halt:'.length) }); if (out.error) throw new Error(out.error); // a non-halt worker fault → honest incomplete - lastAttemptText = out.text || lastAttemptText; return out.text; }; @@ -838,9 +867,13 @@ async function recurseRefineLeaf(task, ctx, opts, state) { node.tokens = tokensSum; // record whatever attempts DID spend, on both the halt and fault paths // The refineLeaf receipt must ride EVERY terminating path, not just the clean one (same invariant as BA-10's // `temperatureDropped`): a leaf that ran attempts then halted/faulted still spent tokens and may have engaged - // the buffer. `effectiveTemps[iteration]` is set BEFORE each attempt's throw, so it reflects every attempt - // made; `passed:false` because the catch is only reached on a throw (a pass returns from the try above). - node.refineLeaf = { iterations: effectiveTemps.length, passed: false, temperatures: effectiveTemps.slice(), rejectedBuffer: bufferUsed }; + // the buffer. `effectiveTemps[iteration]` is set BEFORE each attempt's throw, so it reflects every attempt made. + // NOT unconditional: the refine loop may have COMPLETED (receipt already written, possibly `passed:true`) and + // the throw come from the verify layer below it — clobbering that to `passed:false` would report a sensor that + // never closed when it did, misattributing a verify-slot halt to a non-converging leaf. + if (!node.refineLeaf) { + node.refineLeaf = { iterations: effectiveTemps.length, passed: false, temperatures: effectiveTemps.slice(), rejectedBuffer: bufferUsed }; + } // BA-5: EVERY terminating branch preserves the model's last non-empty attempt (`lastAttemptText`), never // `null` — matching the plain-worker path (`best: out.text || null`) and the "bounds PRESERVE work" // invariant. A refine leaf is the sole cross-attempt text channel in a ralph-style retry; dropping it on a @@ -918,6 +951,10 @@ async function recurseScan(task, ctx, opts, state) { }; } + // BA-5: hoisted so a halt thrown BELOW the scan (e.g. from the verify slot) still returns the finished, + // code-counted result as `best` instead of destroying it — re-running a scan re-pays every window judge call. + /** @type {{count: number, matchedIds: string[]}|null} */ + let scanResult = null; try { const scan = await scanCount(task, corpus, { provider, @@ -930,6 +967,7 @@ async function recurseScan(task, ctx, opts, state) { node.scan = { window: scan.window, passes: scan.passes, scanned: scan.scanned, matched: scan.count }; // Structured, CODE-counted result — the count is authoritative; matchedIds carry the evidence (RC-10). const result = { count: scan.count, matchedIds: scan.matchedIds }; + scanResult = result; // RC-9: a dead window means we did NOT see every slice → the count is a floor, not the answer. Report it // incomplete with the partial as `best`, never a clean pass over a hole. @@ -950,7 +988,9 @@ async function recurseScan(task, ctx, opts, state) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - return { incomplete: true, best: null, receipts: node }; + // BA-5: a halt from the verify slot lands here with the scan already finished — return the counted + // result, never null (the count survived only in receipts before, forcing a full re-scan). + return { incomplete: true, best: scanResult, receipts: node }; } throw err; } @@ -1008,6 +1048,8 @@ async function recursePartition(task, ctx, opts, state) { // (never `undefined`) across every dispatch path. const childResults = []; + /** @type {{count: number, matchedIds: string[]}|null} */ + let partitionResult = null; try { // 2b) Pre-wave checkpoint — width (the cost) is now known. A governance HaltError halts BEFORE any worker // spends (bounds the burst to zero); a plain deny is advisory (allowlist-safe), same contract as fanout. @@ -1052,11 +1094,15 @@ async function recursePartition(task, ctx, opts, state) { if (child.incomplete) missingSlices.push(label); } const result = { count: matched.size, matchedIds: [...matched] }; + partitionResult = result; // BA-5: a halt below this (e.g. verify) returns the full result, not a count-only rebuild node.partition.matched = matched.size; if (missingSlices.length > 0) { node.incomplete = true; - return { incomplete: true, best: result, missingSlices, receipts: node }; + // BA-15: carry a child's blocker up (see inheritedBlocker). + const inherited = inheritedBlocker(node.spawned); + if (inherited) Object.assign(node, inherited); + return { incomplete: true, best: result, missingSlices, ...(inherited && { blocker: inherited.blocker }), receipts: node }; } const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { @@ -1068,7 +1114,10 @@ async function recursePartition(task, ctx, opts, state) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - const best = childResults.length ? { count: new Set(childResults.flatMap((v) => (v && Array.isArray(v.matchedIds) ? v.matchedIds : []))).size } : null; + // Prefer the finished result (a halt from the verify slot lands here with it already computed); else + // rebuild a best-effort count from whatever slices did return (BA-5, never a bare null). + const best = partitionResult + || (childResults.length ? { count: new Set(childResults.flatMap((v) => (v && Array.isArray(v.matchedIds) ? v.matchedIds : []))).size } : null); return { incomplete: true, best, receipts: node }; } throw err; @@ -1193,7 +1242,10 @@ async function recurseFanout(task, ctx, opts, state) { // 5) Honest completeness (RC-9): any missing slice → incomplete, with the partial reduce as `best`. if (missingSlices.length > 0) { node.incomplete = true; - return { incomplete: true, best: result, missingSlices, receipts: node }; + // BA-15: carry a child's blocker up (see inheritedBlocker). + const inherited = inheritedBlocker(node.spawned); + if (inherited) Object.assign(node, inherited); + return { incomplete: true, best: result, missingSlices, ...(inherited && { blocker: inherited.blocker }), receipts: node }; } // 6) Verify (RC-7): forced for critical, or when a contract/override is supplied. diff --git a/test/recurse.test.js b/test/recurse.test.js index 49902f4..2b4cfb2 100644 --- a/test/recurse.test.js +++ b/test/recurse.test.js @@ -767,6 +767,49 @@ describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, assert.equal(out.best, 'attempt-1 work', 'a halt on the retry preserves attempt 1 (pre-fix: best:null)'); }); + it('a FIRST-attempt halt still preserves that attempt\'s text (BA-5 capture-before-throw)', async () => { + // Review-caught regression in the previous fix: `lastAttemptText` was assigned AFTER the halt/error throws, + // so the attempt that actually terminated had its text discarded — best:null on a first-attempt halt. The + // earlier test only halted on attempt 2 (where a prior clean attempt had already populated it), so it + // passed while the single-attempt case was broken. The Loop returns its last non-empty text on every + // terminating path, so the capture must precede the throws. + const sp = scriptedProvider(() => ({ text: 'partial work' })); + const ctx = { provider: sp.provider, onLlmResult: () => { throw new HaltError('budget cap', { rule: 'budget.maxCostUsd' }); } }; + const out = await recurse(SIMPLE_TASK, ctx, { refineLeaf: { sensor: () => ({ pass: false, critique: 'x' }) } }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true); + assert.equal(out.best, 'partial work', 'the halting attempt\'s own text survives (pre-fix: null)'); + }); + + it('a child\'s broken-sensor blocker propagates to the PARENT in a nested tree (BA-15 anti-laundering)', async () => { + // A parent aggregates a dead child into {incomplete, missingSlices}; pre-fix the child's blocker label was + // dropped there, so a top-level caller branching on result.blocker saw nothing and would debug the model + // instead of its own broken sensor — the laundering BA-15 exists to close, reintroduced one level up. + const sp = scriptedProvider(decomposingHandler()); + const out = await recurse(COMPLEX_TASK, { provider: sp.provider }, { + maxDepth: 2, + refineLeaf: { sensor: () => { throw new Error('ENOENT: harness missing'); } }, + }); + assert.equal(out.incomplete, true, 'a broken sensor at the leaf makes the tree incomplete'); + assert.equal(out.blocker, 'broken-sensor', 'the child\'s label reaches the top (pre-fix: undefined)'); + assert.equal(out.receipts.blocker, 'broken-sensor', 'and is recorded on the parent receipts'); + }); + + it('a verify-slot halt after a PASSING sensor does not clobber the refineLeaf receipt to passed:false', async () => { + // verifyOrBlock is awaited INSIDE the try, so a halt at the verify slot reaches the catch even though the + // refine loop already completed and wrote an honest passed:true receipt. Overwriting it would report a + // sensor that never closed when it did. + const sp = scriptedProvider(() => ({ text: 'FIXED answer' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: (r) => ({ pass: String(r).includes('FIXED') }) }, + contract: 'must be fixed', + evaluate: () => { throw new HaltError('cap', { rule: 'budget' }); }, + }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true, 'the governance halt is recorded'); + assert.equal(out.receipts.refineLeaf.passed, true, 'the sensor DID close — receipt preserved (pre-fix: false)'); + }); + it('the generic-fault branch preserves the model\'s last attempt as best (BA-5), not null', async () => { let call = 0; const sp = scriptedProvider(() => { call += 1; if (call >= 2) throw new Error('provider exploded'); return { text: 'attempt-1 work' }; }); From 75608338c202ceb0f76939f391bea7d5877519fc Mon Sep 17 00:00:00 2001 From: hamr0 Date: Mon, 20 Jul 2026 12:14:40 +0200 Subject: [PATCH 6/8] =?UTF-8?q?fix:=20BA-15=20review=20round=203=20?= =?UTF-8?q?=E2=80=94=20ten=20more=20defects,=20three=20of=20them=20regress?= =?UTF-8?q?ions=20this=20branch=20introduced?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rounds 1-2 ran /code-review medium (5 then 4 findings); round 3 ran xhigh (6 finders + gap sweep, clean 40/40 agents) and found 15. The yield rose because the review got DEEPER, not because the code got worse — worth stating plainly, since "the last round found less" was my own (wrong) argument for convergence. Every finding was reproduced against the unfixed branch by poc/ba15-round3-validate.mjs BEFORE any edit; every fix is mutation-proven. Regressions BA-15 itself introduced: 1. The status-only normalization used an object SPREAD (own-enumerable only), so a class-instance verdict with prototype getters passed the shape check then came out with status/critique ERASED — the critique never reached the retry prompt, reinstating the zero-feedback burn for a shape the validator blessed. Now a descriptor copy onto the same prototype. 2. A strict-boolean `pass` gate hard-blocked previously-CONVERGING sensors returning {pass:1}. A PRESENT `pass` now counts whatever its type — refine has always branched on truthiness, and BA-15 exists to catch a verdict with NO usable signal, not one answering in another dialect. 3. Round 1's typecheck fix DETACHED opts.evaluate, so a method-reference verifier went from this===opts to this===undefined and threw. Re-bound to opts; the honest limit (this is never the grader instance) is tested rather than overstated. BA-15's guarantee failing at boundaries it did not cover: 4. The three HaltError catches never called inheritedBlocker (only the missingSlices branches did), so a gate tripping after a broken-sensor child dropped the label. 5. The spawn tool flattened a blocked child into a generic [incomplete], letting a parent re-delegate into the same broken judge — BA-15's own spend-burn, one level up. 6. Object.assign(node, inherited) stamped the label onto EVERY ancestor: receipts accused nodes whose sensor never ran, and one denied child re-labelled its parent governance-deny. The inherited label now rides receipts.blockerFrom + a new blockerTask naming the culprit. 7. recurseFanout never got round 2's BA-5 hoist and discarded a computed reduce on a verify-slot halt. 8. blockerDetail rode only receipts, not the result. 9. Diagnostics: a non-Error throw rendered as [object Object]; a verdict whose accessor throws was reported as "the sensor threw". Separated around the await itself, which probes .then and so fires the accessor before any shape check runs. 10. inheritedBlocker's 'broken-verifier' arm was UNREACHABLE (forChild strips evaluate) while CHANGELOG and JSDoc asserted it worked. Dead arm deleted, claims corrected. All six aggregation/halt paths now route through one incompleteWithBlocker, which also removes the 3x copy-paste whose next edit would have missed a path. KNOWN LIMIT, documented not fixed: a halt DURING scanCount still returns best:null — scanCount throws without surfacing the windows it judged, so nothing exists at this seam to preserve. The comment implying BA-5 coverage was corrected rather than left to mislead. Two of the harness's own checks were CONFOUNDED and caught only because every fix was mutation-proven, not merely re-run: a malformed Planner stub ended the run before the path under test (a false REPRODUCED), and a defensively-rewritten test stopped exercising the crash it existed to catch. +28 mutation-checked tests; suite 889 pass / 0 fail; typecheck clean by exit code; both POCs exit 0. Docs synced same commit (CHANGELOG, CLAUDE.md, context, RLM_PRD). Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 21 +++- CLAUDE.md | 2 +- bareagent.context.md | 2 +- docs/01-product/RLM_PRD.md | 34 +++++- poc/ba15-round3-validate.mjs | 186 ++++++++++++++++++++++++++++ src/recurse.js | 230 ++++++++++++++++++++++++++++------- test/recurse.test.js | 158 +++++++++++++++++++++++- 7 files changed, 576 insertions(+), 57 deletions(-) create mode 100644 poc/ba15-round3-validate.mjs diff --git a/CHANGELOG.md b/CHANGELOG.md index a1dc027..d5a9dd3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,15 +29,30 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog A second review round (a clean 19/19-agent run) then caught four more, including a regression in the first round's own BA-5 fix: - **The BA-5 preservation did not cover the attempt that actually terminated.** `lastAttemptText` was captured *after* the halt/error throws, so a **first-attempt** halt discarded its own text and still returned `best: null` — while the round-1 test only halted on attempt 2, where a prior clean attempt had already populated it. Capture now precedes the throws; mutation-proven. - **`recurseScan`/`recursePartition` destroyed a finished result on a verify-slot halt.** Making `verifyOrBlock` `return await` (round 1) routed a verifier `HaltError` into their catches, which returned `best: null` — throwing away a fully code-counted scan and forcing a re-run that re-pays every window judge call. Both now preserve the computed result. - - **A child's blocker was laundered by its parent.** In a nested tree the parent aggregates a dead child into `{incomplete, missingSlices}` and dropped the child's `broken-sensor`/`broken-verifier` label, so a top-level caller branching on `result.blocker` saw nothing — BA-15's own failure mode, one level up. A shared `inheritedBlocker` now carries it at all three aggregation sites (`broken-*` outranks `governance-deny`, being a fault in the caller's own code). + - **A child's blocker was laundered by its parent.** In a nested tree the parent aggregates a dead child into `{incomplete, missingSlices}` and dropped the child's `broken-sensor` label, so a top-level caller branching on `result.blocker` saw nothing — BA-15's own failure mode, one level up. A shared `inheritedBlocker` now carries it at all three aggregation sites (`broken-sensor` outranks `governance-deny`, being a fault in the caller's own code). - **A verify-slot halt after a passing sensor clobbered the receipt** to `passed: false`, reporting a deterministic close that never happened. Plus: the verdict shape inspection moved inside `runArbiter`'s try (a throwing accessor on a returned Proxy escaped untyped), and `BrokenArbiterError` now carries `cause`. - **Behavior change for adopters:** `opts.evaluate` is now held to its documented `Verdict` return. A verifier returning a loose object (`{score, critique}`) or a bare boolean — always a contract violation, previously silent — now yields `{incomplete, blocker:'broken-verifier'}` instead of riding out as a converged `{result, verdict}`. That is the point of the change (a converged shape carrying an ungradeable verdict is the laundering being closed), but it is a real break for callers who relied on the loose path; no shim is provided, since one would reopen the hole. + A **third** review round (a clean 40/40-agent run at a deeper setting) found ten more — every one reproduced against the pre-fix branch by `poc/ba15-round3-validate.mjs` before being touched, and every fix mutation-proven. Three were regressions **this branch introduced**, and four were BA-15's own anti-laundering guarantee failing at a boundary it did not cover: + + - **A prototype-backed verdict was silently gutted.** The status-only normalization rebuilt the verdict with an object spread, which copies own enumerable properties only — so a class-instance verdict whose `status`/`critique` are prototype getters passed the shape check and came out with those fields **erased**. The critique then never reached the retry prompt, meaning every iteration re-sent the plain task with zero feedback: the exact burn BA-15 exists to prevent, reintroduced for a shape the validator explicitly blessed. Now copied via property descriptors onto the same prototype. + - **A truthy non-boolean `pass` was rejected as a broken arbiter.** Demanding a strict boolean turned a sensor returning `{pass: 1}` — a long-standing yes/no convention that `refine` has always evaluated for truthiness — into a permanent first-attempt block for a previously **converging** integration. A present `pass` now counts whatever its type; BA-15's job is a verdict carrying *no* usable signal, not one answering clearly in another dialect. + - **A method-reference verifier hard-failed.** Round 1's typecheck fix captured `opts.evaluate` into a bare local, detaching it: `evaluate: grader.check` went from `this === opts` (degraded but working) to `this === undefined` (throws on the first field read → `broken-verifier`). Re-bound to `opts`, restoring the prior behavior exactly. Note the limit that restores: `this` is the options object, never the grader — a caller wanting its own receiver must bind. + - **The halt paths dropped an inherited blocker.** `inheritedBlocker` was wired into the three `missingSlices` branches but not the three `HaltError` catches, so a gate tripping *after* a broken-sensor child (mid-synthesize, say) still reported a bare `{incomplete}`. All six paths now route through one `incompleteWithBlocker` helper — which also removes the 3× copy-pasted block whose next edit would have missed a path. + - **The spawn boundary flattened a blocked child into a generic failure.** `[incomplete] ` gave the parent model no way to tell "the model failed" from "the judge is broken", so it could re-delegate the same subtask into the same broken sensor, re-running a full leaf attempt each time up to the round limit — BA-15's own spend-burn, one level up. The tool result now names the blocker and tells the model not to retry. + - **A blocker was stamped onto every ancestor.** `Object.assign(node, inherited)` overwrote each parent's own `blocker`, so the receipts tree accused nodes whose sensor never ran, and a single denied child re-labelled its parent `governance-deny` — pointing the operator at the wrong node to re-gate. The inherited label now rides `receipts.blockerFrom` plus a new `blockerTask` naming the culprit, while `blocker` keeps meaning "*this* node's own arbiter broke". + - **`recurseFanout` discarded a computed reduce on a verify-slot halt** — the BA-5 hoist given to `recurseScan`/`recursePartition` in round 2 was never applied to the third path, so a halt threw away the finished reduce (and any `'merge'` strategy's paid-for LLM call) in favour of a raw re-join of child strings. + - **`blockerDetail` never rode the result**, only `receipts` — the actionable half of the label required walking the audit tree. + - **Diagnostics:** a non-`Error` throw (a harness rejecting with `{code:'ENOENT', path}`) rendered as `[object Object]`, naming nothing; and a verdict whose *accessor* throws was reported as "the sensor threw", sending the operator hunting a `throw` in a sensor that returned normally. The two faults are now distinguished around the `await` itself — which probes `.then` and so triggers the accessor before any shape check runs. + - **An unreachable branch was documented as working.** `inheritedBlocker` matched `'broken-verifier'`, but `forChild` strips `evaluate`, so no child can ever carry that label. Dead branch removed and this entry corrected — the claim above now reads `broken-sensor` only. + + **Known limit (documented, not fixed):** a `HaltError` raised *during* `scanCount` — the likelier halt, since a token cap trips after many window judge calls — still returns `best: null`. `scanCount` throws without surfacing the windows it already judged, so there is nothing at the recurse seam to preserve; fixing it is a retrieval-side change. The comment that previously implied BA-5 coverage here has been corrected rather than left to mislead. + + **Behavior change for adopters:** `opts.evaluate` is now held to its documented `Verdict` return. A verifier returning a loose object (`{score, critique}`) or a bare boolean — always a contract violation, previously silent — now yields `{incomplete, blocker:'broken-verifier'}` instead of riding out as a converged `{result, verdict}`. That is the point of the change (a converged shape carrying an ungradeable verdict is the laundering being closed), but it is a real break for callers who relied on the loose path; no shim is provided, since one would reopen the hole. A verdict carrying a **truthy non-boolean** `pass` is *not* affected — that shape is accepted (see the third round above). **Known open (pre-existing, not from this change):** `instanceof HaltError` is realm-sensitive — a `HaltError` from a different module realm (duplicate install, VM context) would not match, at 23 sites that predate this work. Documented rather than force-fixed. - New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` on `RecurseResult`/receipts; `receipts.blockerDetail`. +20 mutation-checked tests; full suite **762 pass / 0 fail**, typecheck clean (verified by exit code). Pre-fix evidence: `poc/ba15-broken-sensor.mjs`. + New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` and `blockerDetail` on `RecurseResult`; `blockerTask` on `RecurseResult` when the blocker was inherited from a descendant; `receipts.blockerDetail` and `receipts.blockerFrom`. +28 mutation-checked tests; full suite **889 pass / 0 fail**, typecheck clean (verified by exit code). Pre-fix evidence: `poc/ba15-broken-sensor.mjs` (rounds 1–2) and `poc/ba15-round3-validate.mjs` (round 3 — reproduces each finding against the unfixed branch, so a post-fix re-run flips them). ## [0.30.0] - 2026-07-16 diff --git a/CLAUDE.md b/CLAUDE.md index 1e0ed66..07ce6be 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,7 +14,7 @@ Node.js >= 18, pure JS + JSDoc, `node:test` for testing. Flat `src/` layout with | Evaluator | src/evaluator.js | Output-side judge (mirror of Planner): `evaluate(goal, result, criteria) → Verdict` by `predicate` (deterministic, no tokens), `rubric` (LLM), or `agentic` (tool-running critic). Tri-state `Verdict {status: satisfied/needs_revision/failed, pass (derived), score, critique, suggestions}` — mirrors Anthropic Managed Agents "Outcomes" (eval-assist F1). The rubric path runs an **isolated adversarial grader** (separate context window + harsh independent system prompt, never the generator's transcript) — that isolation, not a feedback knob, defeats the self-evaluation trap (A1). Optional `contract` (definition of done, A3) is graded against, not the loose goal. Judge tokens forward to the gate via `onLlmResult` (budget visibility). Composes AROUND a Loop (never inside `loop.js`). Built flagged-and-deletable, no POC (D11 — self-grading blind spot already settled by GAN + litectx R-S8). Third criteria type `agentic` (D9/A2): an **isolated tool-running critic** — a fresh Loop with scoped functional tools (`barebrowse`/`baremobile`, set on the Evaluator or per-call `opts.tools`) and a harsh independent prompt that **exercises** the live artifact (clicks, reads console/network) rather than reading text; each critic round forwards to `onLlmResult` (re-tagged `evaluate`), a governance HaltError re-throws clean, and the critic loop is gate-bounded via `opts.policy` | | refine | src/refine.js | Thin Loop-agnostic generate→evaluate→regenerate loop (port of Outcomes' iterate→grade→revise). Drives caller-supplied `attempt`/`evaluate` until satisfied, terminal `failed`, or `maxIterations` (real bound = bareguard maxTurns/budget). Threads `critique` forward (fresh-feedback default D6/A1 — anchoring on a failed answer defeats the verifier) + shared `contract` to both sides. HaltError propagates clean | | remember | src/remember.js | eval-assist F5 — the consolidation pass (formerly the PRD's parked "future glue", now built). `remember(spans, {provider, store, contract?, metadata?, ctx?, onLlmResult?}) → {facts, spans}`: one cheap LLM pass per harvested stash span distills DURABLE facts (drops ephemeral chatter, superseded values, unanswered questions; deduped within the call), each written `kind:'fact'` (authoritative — not overridable) through the **generic four-verb `Store` socket** — backend-agnostic (JsonFile/SQLite/litectx/custom), **no litectx coupling** (the rejected alt — reading litectx's promotion count — reaches past the socket into one backend). Composes AROUND a Loop (never inside `loop.js`); optional, flagged-and-deletable. Budget visibility: each pass forwards `usage` to `onLlmResult` (mirror of Evaluator); provider HaltError propagates clean. Gives `metrics.memory.facts` its honest writer via `ctx.recordMemoryOp('facts')` — **disjoint from `stored`** (writes WITHOUT ctx → a fact counts once). Distiller prompt live-validated on the real Anthropic wire (`poc/f5-remember-distill.mjs`): recall / faithfulness-to-correction (final 60 not stale 100) / discrimination (chatter→0). Loop never imports this file | -| recurse | src/recurse.js (+ src/recurse-prompts.js, src/recurse-synthesize.js) | RLM `recurse()` primitive — the decompose→fan-out→verify→synthesize entry point (RLM_PRD §10 steps 3, 4, 5 & 7: NB-1 glue + NB-4 + NB-5 + NB-3 reduce + NB-2 forced fan-out + RC-5 retrieval modes). `recurse(task, ctx, opts) → {result, verdict, receipts}` (convergence) / `{incomplete, best, receipts}` (guard exhaustion or dead worker — RC-9, never a faked pass). "B-shell with an A-tool" (§4.2): a deterministic shell offers the model an in-process `spawn_child` A-tool (§4.5 POC default — fresh `Loop`/fresh window per self-call, ~0ms/node vs ≥90ms fork) it MAY use to decompose. `assessComplexity` is a HINT not a gate: routes `simple`→single-shot, flags `critical`→forced adversarial verify (the `isCritical` safety floor). Termination is bareguard's via `ctx.depth`→`policy` (NO 2nd guard layer, §6); `opts.maxDepth` is only the topology knob that stops OFFERING spawn (`maxDepth=1`⇒flat fan-out). **⚠️ COST IS OPEN BY DESIGN — no intrinsic total-work cap:** the Family-A default can spawn up to a Loop's `HARD_ROUND_LIMIT` (100) children/level × `maxDepth`, so TOKEN/$ spend compounds and is bounded ONLY by the gate (POC `rlm-defer2-history-overflow.mjs` saw a weak model do 40–117 calls in one run). **Run with bareguard (or ANY token/USD cap); ungoverned it CAN burn tokens.** Forced `fanout`/`partition` ARE bounded (deterministic count + concurrency cap); the open path is the model-driven default; `maxDepth:1` is the local brake without a gate. NB-4 capability-scrub (RC-11/12, verify-closed step 6): deeper workers get fewer tools (spawn dropped EXACTLY at cap) + a depth-aware prompt (none→"prefer direct action"→"deepest level/cannot delegate/honest-incomplete"); tool set monotone (child ⊆ parent); `maxDepth=1`⇒flat (no nesting). Boundary is cap-INCLUSIVE (`depth>=maxDepth`⇒deepest suffix), mutation-proved across a 0→1→2 nesting. Copy-on-return (RC-2) by construction: child sees only its subtask (fresh msg array), only its result string crosses back. **Audit-safe ctx (F16/BA-1):** `recurse` threads its wiring blob (incl. the live `provider`) down the tree for self-calls, but a wired gate records the per-run ctx VERBATIM as `_ctx` — so `auditSafeCtx()` STRIPS the provider (and thus `apiKey`) from the ctx at every governance boundary (worker `Loop.run`, both pre-wave `ctx.policy` checkpoints, `scanCount`); the worker still gets the provider as a Loop constructor option (`Loop.run` never reads `ctx.provider`). Pairs with bareguard's secret-redaction (BG-1) as defense-in-depth. **`opts.context` (BA-9/F19):** an optional read-only working-context string (paths/cwd) prepended to EVERY worker's task message as a `Working context:` block + forwarded to the Planner as `info` (path-aware slices) + shown to the verifier (neutral FACTS, not a stance) — fixes a sliced worker that can't LOCATE its artifact (relayfact probe-04: workers guessed `.`/`~`/`/tmp`); CARRIES DOWN via `forChild` like `persona` but distinct (persona=privileged SYSTEM stance, context=USER-message run-state, so callers stop laundering cwd through persona); live 0/3→3/3 (`poc/ba9-context-thread.mjs`). **`opts.refineLeaf` (BA-8/F17, opt-in):** turns a DEFINITE leaf (`!canSpawn`) into a bounded generate→sense→regenerate loop reusing `refine.js` — `{sensor, maxIterations?, temperatures?, rejectedBuffer?}`; `sensor`=caller's DETERMINISTIC close (test/compile/lint, not a model judge), its GAP fed FRESH into the next attempt with **ESCALATING temperature** (default `[0.2,0.7,1.0]` — load-bearing WITHOUT memory: flat temp recovered 0/5 as a weak model regenerated identical wrong code ignoring crisp feedback; escalation → 0/5→2-3/5, `poc/ba8-leaf-refine.mjs`). **`rejectedBuffer` (BA-14):** a SkillOpt-shaped rejected-attempt buffer — surface the model's OWN prior failed attempts VERBATIM ("write something structurally DIFFERENT") via `refine.js`'s newly-threaded `attempt` `history`. DIRECTED diversity that ANTAGONIZES temperature's RANDOM diversity: temperature MONOTONICALLY degrades the buffer (`poc/ba14b`: flat 0.2/0.7/1.0 → 100/70/50%), so when it engages the temp is HELD flat at `temps[0]`, never escalated — REFINING BA-10's "temperature is secondary" to "with a buffer, temperature is harmful". Trigger: `true` force-on / `false` force-off / UNSET = ADAPTIVE (engage only once a temperature was DROPPED — a temp-fixed model where escalation is inert and the buffer is the sole lever, `poc/ba14`: 50%→100%). Live verify-shipped on real temp-fixed `claude-sonnet-5` (`poc/ba14-verify-shipped.mjs`: engaged 6/6, ledger reached the wire; efficacy an honest NULL — the lift is a WEAK-model/fixation phenomenon, cost-neutral otherwise, so ADAPTIVE-not-always-on). `receipts.refineLeaf.rejectedBuffer` reports engagement. Gate-bounded per attempt; honest non-recovery (`receipts.refineLeaf.passed=false`), never faked; carries down (engages at the leaves); error-keyed `recall` stays the CALLER's `opts.tools` (litectx-agnostic). Shipped-vs-POC smoke `poc/ba89-shipped-smoke.mjs`. **Governance deny-spin (BA-11/F35):** a worker whose Loop short-circuits a consecutive-policy-deny spin (the Loop `maxConsecutiveDenials` guard, default 3) surfaces as a LABELED `{incomplete, blocker:'governance-deny'}` (on BOTH the plain-worker and `refineLeaf` paths + `receipts.blocker`) — so a caller distinguishes a governance block (widen scope / re-gate / escalate) from a model failure, instead of the probe-16 burn ($1 to the cap, sensor never reached). Live verify-shipped `poc/ba11-verify-shipped.mjs` (haiku, stopped at 3 denials). **Broken-sensor blocker (BA-15):** a `refineLeaf.sensor` that THROWS (non-Halt) or returns a MALFORMED verdict (neither boolean `pass` nor valid tri-state `status`) is a faulty ARBITER, not a model failure — the loop stops at the FIRST broken close (a retry against a broken judge carries zero feedback; pre-fix it burned all attempts then surfaced converged-shaped) and returns `{incomplete, blocker:'broken-sensor'}` + `receipts.blockerDetail`, `best` preserving the last unjudged attempt (BA-5); sensor `HaltError` stays a clean halt; a HUNG sensor is the caller's to bound (isolated child process + timeout — no knob). The SAME class held at the verify slot: a caller `opts.evaluate` that throws/returns garbage is `blocker:'broken-verifier'` via one `verifyOrBlock` helper on ALL five dispatch paths (pre-fix: a throw CRASHED the run plain-worker / laundered bare under refineLeaf; garbage rode out converged-shaped) — the default Evaluator rubric path is never labeled (provider-class faults, narrowest guard); `return await` is load-bearing at the call sites (an un-awaited returned promise escapes the try's Halt catch). Validated pre-fix `poc/ba15-broken-sensor.mjs`. `Evaluator` fills the verify slot (`opts.evaluate` overrides); **NB-3 reduce** (src/recurse-synthesize.js, `synthesize(task, results, opts)`): `opts.synthesize` = a **function** (deterministic code-reduce over child `results` — the §9.1 aggregation path, LLM arithmetic is the weak link) OR a strategy string `'concat'`/`'merge'` (lossless join / isolated Loop-driven subjective merge); reduce fires only on a node that spawned (a leaf keeps its own answer; each level reduces ITS children); default (Family-A) = the parent model's own closing-turn synthesis. **RC-5 retrieval modes BUILT (step 7, src/recurse-retrieval.js)** — context is a HANDLE routed by TASK SHAPE (§9.2.1), not one default: count/"all"→`opts.retrieval:'scan'` (DEFAULT when `opts.corpus` present) = scan every slice + LLM-judge + **CODE-count** over a **generic array slice-source** (`opts.corpus={id,text}[]`, NOT litectx), `opts.window`≈8 (the one calibrated number, recall knee) + `opts.passes`=2 (deterministic rotation union, NO RNG → RC-3 holds); needle→`'search'` (opt-in) = `ctx.litectx.recall` handle tool, embeddings on, `fact`/`episode`, capped `KNN_K=8` — CANNOT count; exact→`'exact'` (opt-in) = embeddings-free code-side AND-term filter tool. **Per-query face `'tools'` (opt-in, step-7 follow-on DONE):** offers `scan_count` (the new exported `buildScanTool`, the complete code-count path) ALONGSIDE `search_memory`/`exact_match` so a Family-A worker routes PER SUB-QUERY by the tool DESCRIPTIONS (scan="use for how many/all/count"; search="never count") — a MIXED needle+count task keeps both; completeness guard does NOT fire for `'tools'` (scan always offered, complete path always reachable). RC-9 floor (dead window→"count is a floor") + unwrapped `HaltError` hold at the tool boundary; async source materialized once (cached); `forChild` strips it (children don't inherit). Live-validated `poc/rlm-scan-as-tool.mjs` (worker picked search-for-needle + scan-for-count, code-known truth, neutral prompt). Completeness-contract guard UPGRADES a capped `search` on a "how many/all" ask to scan (upgrade-ONLY, never silent downgrade). RC-2 (judge matches only ids it was shown — a hallucinated id is intersected away) + RC-9 (dead window/empty corpus → `{incomplete, missingSlices}`, never a fabricated `count:0`) hold; gate-Halt mid-scan → clean incomplete. **Backend split:** litectx has no exhaustive rank-free enumerate verb today (every read FTS-gated), so scan reads the array slice-source; the "corpus already in litectx" case + the data-driven *width* count wait on the litectx `enumerate` verb (spec handed off: docs/01-product/prd.md) and drop in behind the same socket with ZERO recurse changes. `opts.tools` remain caller-supplied handle tools. Measured on AG News; naive "search→count" DROPPED. Backward-compatible: no corpus + no retrieval ⇒ unchanged (Family A). NB-5 decomposition policy + few-shot in recurse-prompts.js. **Family-B forced fan-out (`opts.count`/`mode:'fanout'`, NB-2) BUILT (step 5):** deterministic count→`Planner`(`{count}` seam = exactly N independent steps)→`runPlan`(waves)→NB-3 reduce(`'concat'` default)→verify; count = `opts.count` (overrides) else the **live-calibrated** tier map medium/complex/critical→**2/4/6** (an overridable default, not a discovered constant — knee location is topology). Each slice is a fresh-window child (MAY itself self-decompose under Family A); RC-9 dead-slice→`missingSlices` (no survivor-sum); planner/child/reduce/verify `HaltError`→clean incomplete. **Metered + pre-wave checkpoint (§6):** the decomposition call forwards usage (`onLlmResult`) so it's not invisible to the budget, and the cheap decompose RESOLVES the unknown→known width, after which `ctx.policy('recurse_fanout', {count, depth})` runs BEFORE the worker wave — a `HaltError` (budget / ≥80% HITL pause, bareguard's decision) → clean incomplete before any worker spends (bounds the concurrent burst to zero); a plain deny is advisory (allowlist-safe). **`opts.count` = fixed/semantic FLOOR;** the **data-driven *width* PARTITION** is BUILT (litectx 0.26 `enumerate`): `recurse(task, ctx, {mode:'partition', corpus, workerBudget?})` measures the corpus → `width=max(floor, ⌈size/workerBudget⌉)` (capped by guards + size) parallel scan-workers → union-count; a distinct path from `recurseFanout`'s semantic split, same FLOOR. `litectxCorpus(litectx,{kind})` is the resident slice-source (paginates `enumerate`); `opts.corpus` also accepts an in-hand array OR any async `()=>Promise` (recurse stays litectx-agnostic). Pre-wave `recurse_partition` checkpoint + RC-9 hold. `enumerate` was verified vs the spec DoD before building (`poc/litectx-enumerate-verify.mjs`). Composes AROUND a Loop (never imported by loop.js); validated by 71 mutation-checked offline tests + a live `--fanout` smoke + **step-8 e2e replay (verify-shipped-vs-POC, `poc/rlm-step8-shipped-replay.mjs`): the SHIPPED scan reproduces §9.2.1 on AG News (recall 0.88/precision 0.97/err 9%, code-count===matchedIds) — a real regression caught en route (a colon-ambiguous id prompt under-recalled 0.93→0.29; fixed with ` => ` display + verbatim-copy)** | +| recurse | src/recurse.js (+ src/recurse-prompts.js, src/recurse-synthesize.js) | RLM `recurse()` primitive — the decompose→fan-out→verify→synthesize entry point (RLM_PRD §10 steps 3, 4, 5 & 7: NB-1 glue + NB-4 + NB-5 + NB-3 reduce + NB-2 forced fan-out + RC-5 retrieval modes). `recurse(task, ctx, opts) → {result, verdict, receipts}` (convergence) / `{incomplete, best, receipts}` (guard exhaustion or dead worker — RC-9, never a faked pass). "B-shell with an A-tool" (§4.2): a deterministic shell offers the model an in-process `spawn_child` A-tool (§4.5 POC default — fresh `Loop`/fresh window per self-call, ~0ms/node vs ≥90ms fork) it MAY use to decompose. `assessComplexity` is a HINT not a gate: routes `simple`→single-shot, flags `critical`→forced adversarial verify (the `isCritical` safety floor). Termination is bareguard's via `ctx.depth`→`policy` (NO 2nd guard layer, §6); `opts.maxDepth` is only the topology knob that stops OFFERING spawn (`maxDepth=1`⇒flat fan-out). **⚠️ COST IS OPEN BY DESIGN — no intrinsic total-work cap:** the Family-A default can spawn up to a Loop's `HARD_ROUND_LIMIT` (100) children/level × `maxDepth`, so TOKEN/$ spend compounds and is bounded ONLY by the gate (POC `rlm-defer2-history-overflow.mjs` saw a weak model do 40–117 calls in one run). **Run with bareguard (or ANY token/USD cap); ungoverned it CAN burn tokens.** Forced `fanout`/`partition` ARE bounded (deterministic count + concurrency cap); the open path is the model-driven default; `maxDepth:1` is the local brake without a gate. NB-4 capability-scrub (RC-11/12, verify-closed step 6): deeper workers get fewer tools (spawn dropped EXACTLY at cap) + a depth-aware prompt (none→"prefer direct action"→"deepest level/cannot delegate/honest-incomplete"); tool set monotone (child ⊆ parent); `maxDepth=1`⇒flat (no nesting). Boundary is cap-INCLUSIVE (`depth>=maxDepth`⇒deepest suffix), mutation-proved across a 0→1→2 nesting. Copy-on-return (RC-2) by construction: child sees only its subtask (fresh msg array), only its result string crosses back. **Audit-safe ctx (F16/BA-1):** `recurse` threads its wiring blob (incl. the live `provider`) down the tree for self-calls, but a wired gate records the per-run ctx VERBATIM as `_ctx` — so `auditSafeCtx()` STRIPS the provider (and thus `apiKey`) from the ctx at every governance boundary (worker `Loop.run`, both pre-wave `ctx.policy` checkpoints, `scanCount`); the worker still gets the provider as a Loop constructor option (`Loop.run` never reads `ctx.provider`). Pairs with bareguard's secret-redaction (BG-1) as defense-in-depth. **`opts.context` (BA-9/F19):** an optional read-only working-context string (paths/cwd) prepended to EVERY worker's task message as a `Working context:` block + forwarded to the Planner as `info` (path-aware slices) + shown to the verifier (neutral FACTS, not a stance) — fixes a sliced worker that can't LOCATE its artifact (relayfact probe-04: workers guessed `.`/`~`/`/tmp`); CARRIES DOWN via `forChild` like `persona` but distinct (persona=privileged SYSTEM stance, context=USER-message run-state, so callers stop laundering cwd through persona); live 0/3→3/3 (`poc/ba9-context-thread.mjs`). **`opts.refineLeaf` (BA-8/F17, opt-in):** turns a DEFINITE leaf (`!canSpawn`) into a bounded generate→sense→regenerate loop reusing `refine.js` — `{sensor, maxIterations?, temperatures?, rejectedBuffer?}`; `sensor`=caller's DETERMINISTIC close (test/compile/lint, not a model judge), its GAP fed FRESH into the next attempt with **ESCALATING temperature** (default `[0.2,0.7,1.0]` — load-bearing WITHOUT memory: flat temp recovered 0/5 as a weak model regenerated identical wrong code ignoring crisp feedback; escalation → 0/5→2-3/5, `poc/ba8-leaf-refine.mjs`). **`rejectedBuffer` (BA-14):** a SkillOpt-shaped rejected-attempt buffer — surface the model's OWN prior failed attempts VERBATIM ("write something structurally DIFFERENT") via `refine.js`'s newly-threaded `attempt` `history`. DIRECTED diversity that ANTAGONIZES temperature's RANDOM diversity: temperature MONOTONICALLY degrades the buffer (`poc/ba14b`: flat 0.2/0.7/1.0 → 100/70/50%), so when it engages the temp is HELD flat at `temps[0]`, never escalated — REFINING BA-10's "temperature is secondary" to "with a buffer, temperature is harmful". Trigger: `true` force-on / `false` force-off / UNSET = ADAPTIVE (engage only once a temperature was DROPPED — a temp-fixed model where escalation is inert and the buffer is the sole lever, `poc/ba14`: 50%→100%). Live verify-shipped on real temp-fixed `claude-sonnet-5` (`poc/ba14-verify-shipped.mjs`: engaged 6/6, ledger reached the wire; efficacy an honest NULL — the lift is a WEAK-model/fixation phenomenon, cost-neutral otherwise, so ADAPTIVE-not-always-on). `receipts.refineLeaf.rejectedBuffer` reports engagement. Gate-bounded per attempt; honest non-recovery (`receipts.refineLeaf.passed=false`), never faked; carries down (engages at the leaves); error-keyed `recall` stays the CALLER's `opts.tools` (litectx-agnostic). Shipped-vs-POC smoke `poc/ba89-shipped-smoke.mjs`. **Governance deny-spin (BA-11/F35):** a worker whose Loop short-circuits a consecutive-policy-deny spin (the Loop `maxConsecutiveDenials` guard, default 3) surfaces as a LABELED `{incomplete, blocker:'governance-deny'}` (on BOTH the plain-worker and `refineLeaf` paths + `receipts.blocker`) — so a caller distinguishes a governance block (widen scope / re-gate / escalate) from a model failure, instead of the probe-16 burn ($1 to the cap, sensor never reached). Live verify-shipped `poc/ba11-verify-shipped.mjs` (haiku, stopped at 3 denials). **Broken-sensor blocker (BA-15):** a `refineLeaf.sensor` that THROWS (non-Halt) or returns a MALFORMED verdict (neither a USABLE `pass` — present, any type, since `refine` branches on truthiness — nor a valid tri-state `status`) is a faulty ARBITER, not a model failure — the loop stops at the FIRST broken close (a retry against a broken judge carries zero feedback; pre-fix it burned all attempts then surfaced converged-shaped) and returns `{incomplete, blocker:'broken-sensor'}` + `blockerDetail`, `best` preserving the last unjudged attempt (BA-5); sensor `HaltError` stays a clean halt; a HUNG sensor is the caller's to bound (isolated child process + timeout — no knob). A verdict is normalized PROTOTYPE-PRESERVINGLY (descriptor copy, never a spread — a spread erases accessor-backed `status`/`critique`, reinstating the zero-feedback burn). Nested: a descendant's label rides `blocker`+`blockerTask` on the RESULT and `receipts.blockerFrom` on each ancestor — never the ancestor's own `blocker` (which means THIS node broke), on ALL six aggregation/halt paths via one `incompleteWithBlocker`; the spawn tool tells the parent model a child was BLOCKED so it does not re-delegate into the same broken judge. The SAME class held at the verify slot: a caller `opts.evaluate` that throws/returns garbage is `blocker:'broken-verifier'` via one `verifyOrBlock` helper on ALL five dispatch paths (pre-fix: a throw CRASHED the run plain-worker / laundered bare under refineLeaf; garbage rode out converged-shaped) — the default Evaluator rubric path is never labeled (provider-class faults, narrowest guard); `return await` is load-bearing at the call sites (an un-awaited returned promise escapes the try's Halt catch). Validated pre-fix `poc/ba15-broken-sensor.mjs`. `Evaluator` fills the verify slot (`opts.evaluate` overrides); **NB-3 reduce** (src/recurse-synthesize.js, `synthesize(task, results, opts)`): `opts.synthesize` = a **function** (deterministic code-reduce over child `results` — the §9.1 aggregation path, LLM arithmetic is the weak link) OR a strategy string `'concat'`/`'merge'` (lossless join / isolated Loop-driven subjective merge); reduce fires only on a node that spawned (a leaf keeps its own answer; each level reduces ITS children); default (Family-A) = the parent model's own closing-turn synthesis. **RC-5 retrieval modes BUILT (step 7, src/recurse-retrieval.js)** — context is a HANDLE routed by TASK SHAPE (§9.2.1), not one default: count/"all"→`opts.retrieval:'scan'` (DEFAULT when `opts.corpus` present) = scan every slice + LLM-judge + **CODE-count** over a **generic array slice-source** (`opts.corpus={id,text}[]`, NOT litectx), `opts.window`≈8 (the one calibrated number, recall knee) + `opts.passes`=2 (deterministic rotation union, NO RNG → RC-3 holds); needle→`'search'` (opt-in) = `ctx.litectx.recall` handle tool, embeddings on, `fact`/`episode`, capped `KNN_K=8` — CANNOT count; exact→`'exact'` (opt-in) = embeddings-free code-side AND-term filter tool. **Per-query face `'tools'` (opt-in, step-7 follow-on DONE):** offers `scan_count` (the new exported `buildScanTool`, the complete code-count path) ALONGSIDE `search_memory`/`exact_match` so a Family-A worker routes PER SUB-QUERY by the tool DESCRIPTIONS (scan="use for how many/all/count"; search="never count") — a MIXED needle+count task keeps both; completeness guard does NOT fire for `'tools'` (scan always offered, complete path always reachable). RC-9 floor (dead window→"count is a floor") + unwrapped `HaltError` hold at the tool boundary; async source materialized once (cached); `forChild` strips it (children don't inherit). Live-validated `poc/rlm-scan-as-tool.mjs` (worker picked search-for-needle + scan-for-count, code-known truth, neutral prompt). Completeness-contract guard UPGRADES a capped `search` on a "how many/all" ask to scan (upgrade-ONLY, never silent downgrade). RC-2 (judge matches only ids it was shown — a hallucinated id is intersected away) + RC-9 (dead window/empty corpus → `{incomplete, missingSlices}`, never a fabricated `count:0`) hold; gate-Halt mid-scan → clean incomplete. **Backend split:** litectx has no exhaustive rank-free enumerate verb today (every read FTS-gated), so scan reads the array slice-source; the "corpus already in litectx" case + the data-driven *width* count wait on the litectx `enumerate` verb (spec handed off: docs/01-product/prd.md) and drop in behind the same socket with ZERO recurse changes. `opts.tools` remain caller-supplied handle tools. Measured on AG News; naive "search→count" DROPPED. Backward-compatible: no corpus + no retrieval ⇒ unchanged (Family A). NB-5 decomposition policy + few-shot in recurse-prompts.js. **Family-B forced fan-out (`opts.count`/`mode:'fanout'`, NB-2) BUILT (step 5):** deterministic count→`Planner`(`{count}` seam = exactly N independent steps)→`runPlan`(waves)→NB-3 reduce(`'concat'` default)→verify; count = `opts.count` (overrides) else the **live-calibrated** tier map medium/complex/critical→**2/4/6** (an overridable default, not a discovered constant — knee location is topology). Each slice is a fresh-window child (MAY itself self-decompose under Family A); RC-9 dead-slice→`missingSlices` (no survivor-sum); planner/child/reduce/verify `HaltError`→clean incomplete. **Metered + pre-wave checkpoint (§6):** the decomposition call forwards usage (`onLlmResult`) so it's not invisible to the budget, and the cheap decompose RESOLVES the unknown→known width, after which `ctx.policy('recurse_fanout', {count, depth})` runs BEFORE the worker wave — a `HaltError` (budget / ≥80% HITL pause, bareguard's decision) → clean incomplete before any worker spends (bounds the concurrent burst to zero); a plain deny is advisory (allowlist-safe). **`opts.count` = fixed/semantic FLOOR;** the **data-driven *width* PARTITION** is BUILT (litectx 0.26 `enumerate`): `recurse(task, ctx, {mode:'partition', corpus, workerBudget?})` measures the corpus → `width=max(floor, ⌈size/workerBudget⌉)` (capped by guards + size) parallel scan-workers → union-count; a distinct path from `recurseFanout`'s semantic split, same FLOOR. `litectxCorpus(litectx,{kind})` is the resident slice-source (paginates `enumerate`); `opts.corpus` also accepts an in-hand array OR any async `()=>Promise` (recurse stays litectx-agnostic). Pre-wave `recurse_partition` checkpoint + RC-9 hold. `enumerate` was verified vs the spec DoD before building (`poc/litectx-enumerate-verify.mjs`). Composes AROUND a Loop (never imported by loop.js); validated by 71 mutation-checked offline tests + a live `--fanout` smoke + **step-8 e2e replay (verify-shipped-vs-POC, `poc/rlm-step8-shipped-replay.mjs`): the SHIPPED scan reproduces §9.2.1 on AG News (recall 0.88/precision 0.97/err 9%, code-count===matchedIds) — a real regression caught en route (a colon-ambiguous id prompt under-recalled 0.93→0.29; fixed with ` => ` display + verbatim-copy)** | | SkillRegistry | src/skills.js | eval-assist F2 skill mechanism (PRD §2.3–2.7): operator-registered `{name, description, instructions, tools}` bundles surfaced by PROGRESSIVE DISCLOSURE — one meta-tool `skill_use` carries a one-liner catalog in its description; `skill_use({name})` returns the skill's `instructions` AS the tool result and unlocks its (auto-`_`-prefixed) tools, called natively next round. The only Loop coupling is the tools-as-thunk primitive (D4): pass `skills.activeTools` (bound) as the `tools` thunk. Governance unchanged (D5): the gate judges `(tool, args)` blind to origin — skills affect DISCOVERY, never AUTHORIZATION. Separator is `_` not `.` (POC-corrected — providers' `^[a-zA-Z0-9_-]+$` rejects a dot); `SAFE_NAME` enforced at `register()` (fail-fast vs mid-run provider error). Collision check across native/MCP/skill names commits nothing on failure. Loop never imports this file | | stash | src/stash.js | eval-assist F2 reference skill (PRD §2.8–2.14): compaction-first context hygiene. `createStashSkill(options) → {skill, trim}` — register `skill`, wire `trim` into `Loop({ trim })`. Tools (`stash_checkpoint`/`stash_compact`/`stash_restore`) QUEUE intent (args-only); the fold runs in `trim(msgs, ctx)` (loop.js:487) — the one seam with the live transcript — at a clean round boundary (a synchronous fold would orphan the triggering compact's own tool pair). **Two transcript invariants held by construction:** tool_call/tool_result pairing (fold whole rounds) + Anthropic user/assistant alternation (no consecutive-role merging) — so the fold evicts whole rounds and any inline note is a synthetic `assistant(tool_call)+tool(result)` PAIR on a user→assistant boundary (a bare note breaks alternation; a `system` note is hoisted+clobbers the system prompt). Anchor = identity-ref to the boundary message, NOT an injected marker. Two strategies (D10): `summarize` (default, lossy gist via `ctx.summarize`; degrades LOUDLY to a lossless park when unwired) + `stash` (lossless verbatim → litectx stash table or run-scoped in-process Map). Stance (D13): litectx `episode` upsert on compact. Auto-trigger (D11/§2.11, opt-in `compaction:{ceilingTokens,triggerAt,strategy,keepHeadTurns,keepRecentTurns}`): folds the MIDDLE on measured `ctx.usage.inputTokens/ceiling > triggerAt` (Loop publishes `ctx.usage`; all bareagent, never a bareguard halt bound). LRU label backstop (§2.13). Loop never imports this file | | StateMachine | src/state.js | Task lifecycle (pending/running/done/failed/waiting/cancelled) | diff --git a/bareagent.context.md b/bareagent.context.md index 5d14ada..afa798c 100644 --- a/bareagent.context.md +++ b/bareagent.context.md @@ -712,7 +712,7 @@ const out = await recurse('Audit auth.js, billing.js, gateway.js for authz bugs' > **Sensor integrity (`refineLeaf.sensor`):** the sensor must judge the **returned result** (tamper-proof — build/run the returned string in isolation), never a worker **side-effect** a worker with edit tools could game (writing a passing file then returning junk, or editing the failing test itself). The loop optimizes against whatever the sensor reads — keep the close outside what the worker can write. (RSI field lesson: reward-hacking appeared in every optimization loop with a gameable close. **Demonstrated in-repo** — `poc/sensor-gaming-blocked.mjs`: the moment the honest path was blocked, `claude-sonnet-5` faked a pass on an unsatisfiable on-disk check **5/5** — both by editing the test and by making the returned function non-pure — while a tamper-proof close that runs the returned artifact in isolation was not gameable.) -> **Broken arbiter ≠ failing model (BA-15):** a `refineLeaf.sensor` — or a caller `opts.evaluate` verifier — that **throws** (your test runner crashed — ENOENT, a harness syntax error) or returns a **malformed verdict** (anything without a boolean `pass` or a valid tri-state `status`) is a faulty **arbiter**, not a model failure. The loop stops at the **first** broken close (it never retries against a broken judge — each retry would carry zero feedback) and returns a labeled `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier' }` plus `receipts.blockerDetail` (what the arbiter did), with `best` preserving the model's last non-empty output (BA-5) — best-effort work the arbiter never graded, not a pass. The label names WHICH knob to fix. Check `out.blocker` before debugging the model: `'broken-sensor'` / `'broken-verifier'` mean fix your sensor / your `evaluate` and re-run; `'governance-deny'` means widen scope / re-gate. The **default Evaluator rubric verifier is never labeled** (its failures are provider-class faults), and a `HaltError` thrown by either arbiter stays a clean governance halt. One thing stays YOURS: an arbiter that **hangs** hangs the node (no gate checkpoint fires inside your callback) — run untrusted or model-generated checks in an isolated child process **with a timeout**. +> **Broken arbiter ≠ failing model (BA-15):** a `refineLeaf.sensor` — or a caller `opts.evaluate` verifier — that **throws** (your test runner crashed — ENOENT, a harness syntax error) or returns a **malformed verdict** (anything with neither a usable `pass` nor a valid tri-state `status`) is a faulty **arbiter**, not a model failure. The loop stops at the **first** broken close (it never retries against a broken judge — each retry would carry zero feedback) and returns a labeled `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier', blockerDetail }` (what the arbiter did; mirrored on `receipts`), with `best` preserving the model's last non-empty output (BA-5) — best-effort work the arbiter never graded, not a pass. The label names WHICH knob to fix. Check `out.blocker` before debugging the model: `'broken-sensor'` / `'broken-verifier'` mean fix your sensor / your `evaluate` and re-run; `'governance-deny'` means widen scope / re-gate. **`pass` may be any truthy/falsy value** (`true`, `1`, `0`) — only a verdict carrying *no* usable signal is malformed — and a verdict backed by a class/getters keeps its fields. **In a nested run** the label travels: `out.blocker` reports a descendant's fault with `out.blockerTask` naming which sub-task broke, while each ancestor's receipts record it as `blockerFrom` — an ancestor's own `blocker` always means *that* node broke, so it never accuses a node whose sensor never ran. The **default Evaluator rubric verifier is never labeled** (its failures are provider-class faults), and a `HaltError` thrown by either arbiter stays a clean governance halt. One thing stays YOURS: an arbiter that **hangs** hangs the node (no gate checkpoint fires inside your callback) — run untrusted or model-generated checks in an isolated child process **with a timeout**. > **`rejectedBuffer` (BA-14, v0.30.0):** a second lever for a **temperature-fixed** model, where escalation is inert. Instead of only the latest critique, it surfaces the model's OWN prior failed attempts VERBATIM — *"you wrote these, they failed X — write something STRUCTURALLY DIFFERENT."* This is **directed** diversity (attack the specific repeated mistake); escalation is **random** diversity, and the two are **antagonistic** — temperature monotonically degrades the buffer (`poc/ba14b`: flat-0.2 100% → 0.7 70% → 1.0 50%), so when the buffer engages the retry temperature is **held flat** at `temperatures[0]`, never escalated. Trigger: `true` = force on (also on temperature-accepting models); `false` = force off (pure BA-8 escalation); **unset = adaptive** — engage only once a prior attempt's temperature was dropped (i.e. a temp-fixed model where escalation is inert and the buffer is the sole lever). On a temperature-accepting model the default leaves behavior byte-identical. `receipts.refineLeaf.rejectedBuffer` reports whether it engaged. Efficacy is a **weak-model / fixation** phenomenon (live on `claude-sonnet-5` it engaged 6/6 but recovered no better than critique-only — cost-neutral, hence adaptive-not-always-on). diff --git a/docs/01-product/RLM_PRD.md b/docs/01-product/RLM_PRD.md index 43a261c..c89f4c4 100644 --- a/docs/01-product/RLM_PRD.md +++ b/docs/01-product/RLM_PRD.md @@ -1409,9 +1409,9 @@ run kept as evidence): a verifier `HaltError` into their catches, which returned `best:null`, discarding a fully code-counted scan (a re-run re-pays every window judge call); both now preserve the computed result. **(3) a child's blocker was laundered by its parent** — the `{incomplete, missingSlices}` aggregation dropped a child's - `broken-sensor`/`broken-verifier`, so a nested broken sensor never named itself at the top: BA-15's own + `broken-sensor`, so a nested broken sensor never named itself at the top: BA-15's own failure mode reintroduced one level up. A shared `inheritedBlocker` carries it at all three aggregation - sites (`broken-*` outranks `governance-deny` — a fault in the CALLER's code is the more actionable label). + sites (`broken-sensor` outranks `governance-deny` — a fault in the CALLER's code is the more actionable label). **(4) a verify-slot halt after a PASSING sensor clobbered the receipt** to `passed:false`. Plus the verdict shape inspection moved INSIDE `runArbiter`'s try (a throwing accessor on a returned Proxy escaped untyped — the same uncaught-crash class one step later) and `cause` preserved on `BrokenArbiterError`. **Accepted @@ -1421,6 +1421,36 @@ run kept as evidence): `instanceof HaltError` is realm-sensitive. +4 mutation-proven tests; suite **762 pass / 0 fail**. METHOD LESSON: a regression test that exercises only the multi-step path cannot catch a first-step bug — and a fix round is exactly when a fresh review is most valuable, because the new code is the least-reviewed. +- **2026-07-20 — BA-15 review round 3 (clean 40/40 at a DEEPER setting): ten more, three of them regressions + this branch introduced.** Rounds 1–2 ran `/code-review medium` and found 5 then 4; round 3 ran **xhigh** + (6 finders + a gap sweep) and found **15** — so the yield rose because the review got deeper, NOT because + the code got worse. Every finding was reproduced against the unfixed branch by `poc/ba15-round3-validate.mjs` + BEFORE any edit, and every fix mutation-proven. **Regressions introduced by BA-15 itself:** (1) the + status-only normalization used an object SPREAD, which copies own-enumerable props only — a class-instance + verdict with prototype getters passed the shape check then came out with `status`/`critique` ERASED, + reinstating the zero-feedback burn for a shape the validator blessed (now a descriptor copy onto the same + prototype); (2) a strict-boolean `pass` gate hard-blocked previously-CONVERGING sensors returning `{pass:1}` + (now: a PRESENT `pass` counts, any type — `refine` always branched on truthiness); (3) round 1's typecheck + fix DETACHED `opts.evaluate`, so a method-reference verifier went from `this===opts` to `this===undefined` + and threw (re-bound to `opts`; the honest limit — `this` is never the grader instance — is now tested). + **BA-15's guarantee failing at uncovered boundaries:** the three `HaltError` catches never called + `inheritedBlocker` (only the `missingSlices` branches did); the spawn tool flattened a blocked child into a + generic `[incomplete]`, letting a parent re-delegate into the same broken judge (BA-15's own spend-burn one + level up); `Object.assign(node, inherited)` stamped the label onto EVERY ancestor, so receipts accused nodes + whose sensor never ran and one denied child re-labelled its parent `governance-deny`; and `recurseFanout` + never got round 2's BA-5 hoist, discarding a computed reduce on a verify-slot halt. All six aggregation/halt + paths now route through one `incompleteWithBlocker` (which also deletes the 3× copy-paste whose next edit + would have missed a path); the inherited label rides `blockerFrom` + a new `blockerTask`, never the + ancestor's own `blocker`. Also: `inheritedBlocker`'s `'broken-verifier'` arm was UNREACHABLE (`forChild` + strips `evaluate`) while CHANGELOG and JSDoc asserted it worked — dead arm deleted, claims corrected. + **KNOWN LIMIT, documented not fixed:** a halt DURING `scanCount` still returns `best:null` — `scanCount` + throws without surfacing the windows it judged, so nothing exists at this seam to preserve; the comment that + implied BA-5 coverage was corrected instead of left to mislead. Suite **889 pass / 0 fail**, typecheck clean + by exit code, +28 mutation-checked tests. METHOD LESSONS: (a) review DEPTH, not code age, drove the yield — + "the last round found less" is not evidence of convergence when the rounds ran at different settings; + (b) two of the harness's own first-pass checks were CONFOUNDED (a malformed Planner stub ended the run + before the path under test, and a defensively-rewritten test stopped exercising the crash it existed to + catch) — both surfaced only because every fix was mutation-proven, not merely re-run. - **Assessed, NOT built (same feedback, lean bar):** the DIRECT-Evaluator predicate coercion claim (`!!predicate()`) is the predicate's documented boolean contract and a throw there is a visible exception to the direct caller (the laundering only arose one level up, at recurse's seam — fixed above); the proposed diff --git a/poc/ba15-round3-validate.mjs b/poc/ba15-round3-validate.mjs new file mode 100644 index 0000000..847e2ca --- /dev/null +++ b/poc/ba15-round3-validate.mjs @@ -0,0 +1,186 @@ +// BA-15 round-3 review — DETERMINISTIC OFFLINE validation of every reported finding against the +// CURRENT (unfixed) branch code. Run BEFORE any fix: each check must REPRODUCE the reported pathology, +// otherwise the finding is dropped rather than "fixed" on a reviewer's say-so (v0.27.0 precedent). +// +// node poc/ba15-round3-validate.mjs +// +// Exit 0 = every check ran; the table says which findings reproduced. This file characterizes the BUG, +// so a post-fix re-run is EXPECTED to flip the reproduced rows to NOT-REPRODUCED. + +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { recurse } = require('../src/recurse.js'); +const { HaltError } = require('../src/errors.js'); + +const rows = []; +const record = (id, what, reproduced, evidence) => { + rows.push({ id, what, reproduced, evidence }); + console.log(`${reproduced ? 'REPRODUCED ' : 'NOT-REPRODUCED'} ${id} ${what}\n ${evidence}\n`); +}; + +// ---------------------------------------------------------------- scripted provider +function scripted(handler, { model = 'stub-model', name = 'stub' } = {}) { + const calls = []; + return { + calls, + provider: { + model, name, + async generate(messages, tools, options) { + calls.push({ messages, tools: (tools || []).map(t => t.name), options }); + const r = (await handler(messages, tools, options, calls.length - 1)) || {}; + return { text: r.text || '', toolCalls: r.toolCalls || [], usage: { inputTokens: 5, outputTokens: 3 }, model }; + }, + }, + }; +} +const plain = (text) => scripted(() => ({ text })).provider; +const COMPLEX = 'design and implement a notification pipeline across the entire system'; +const SIMPLE = 'list the open files'; + +// ---------------------------------------------------------------- F2 verdict spread erases prototype fields +{ + class V { + constructor(s, c) { this._s = s; this._c = c; } + get status() { return this._s; } + get critique() { return this._c; } + } + const out = await recurse(SIMPLE, { provider: plain('the answer') }, { + maxDepth: 0, + evaluate: () => new V('satisfied', 'looks good'), + }); + const lost = out.verdict && out.verdict.status === undefined && out.verdict.critique === undefined; + record('F2', 'runArbiter spread erases prototype-backed status/critique', + !!lost, `verdict.status=${JSON.stringify(out.verdict?.status)} critique=${JSON.stringify(out.verdict?.critique)} (caller sent 'satisfied'/'looks good')`); +} + +// ---------------------------------------------------------------- F4 strict boolean `pass` hard-blocks truthy +{ + let attempts = 0; + const out = await recurse(SIMPLE, { provider: scripted(() => { attempts++; return { text: 'attempt output' }; }).provider }, { + maxDepth: 0, + refineLeaf: { sensor: () => ({ pass: 1, critique: '' }), maxIterations: 3 }, + }); + record('F4', 'sensor {pass:1} (truthy non-boolean) hard-blocks as broken arbiter', + out.blocker === 'broken-sensor', `blocker=${out.blocker} incomplete=${out.incomplete} attempts=${attempts} (pre-BA-15 this converged)`); +} + +// ---------------------------------------------------------------- F7 non-Error throw → [object Object] +{ + const out = await recurse(SIMPLE, { provider: plain('x') }, { + maxDepth: 0, + refineLeaf: { sensor: () => { throw { code: 'ENOENT', path: '/usr/bin/vitest' }; }, maxIterations: 2 }, + }); + const d = out.receipts?.blockerDetail || ''; + record('F7', 'non-Error throw yields an uninformative blockerDetail', + d.includes('[object Object]'), `blockerDetail=${JSON.stringify(d)}`); +} + +// ---------------------------------------------------------------- F8 throwing accessor reported as "threw" +{ + const out = await recurse(SIMPLE, { provider: plain('x') }, { + maxDepth: 0, + refineLeaf: { sensor: () => new Proxy({}, { get() { throw new Error('trap'); } }), maxIterations: 2 }, + }); + const d = out.receipts?.blockerDetail || ''; + record('F8', 'a RETURNED object whose accessor throws is reported as the sensor THROWING', + d.includes('threw'), `blockerDetail=${JSON.stringify(d)} (the sensor returned normally; the fault is its shape)`); +} + +// ---------------------------------------------------------------- F3 fanout halt discards the computed reduce +// HARNESS NOTE: the first draft fed the Planner `{steps:[{description}]}`, but it parses a bare JSON ARRAY of +// `{id, action, dependsOn}`. The plan failed to parse, the run ended BEFORE the reduce, and `best:null` looked +// like the reported bug — a false REPRODUCED on a path the probe never reached. Drive the planner by call index. +{ + let n = 0, i = 0; + const provider = scripted(() => { + if (i++ === 0) return { text: JSON.stringify([0, 1, 2].map(j => ({ id: `s${j}`, action: `SLICE ${j}`, dependsOn: [] }))) }; + return { text: `child-${++n}` }; + }).provider; + const out = await recurse(COMPLEX, { provider }, { + count: 3, + synthesize: ({ results }) => ({ total: results.length }), + evaluate: () => { throw new HaltError('budget cap', { rule: 'budget' }); }, + }); + const isReduce = out.best && typeof out.best === 'object' && out.best.total === 3; + record('F3', 'fanout halt discards the computed reduce (no BA-5 hoist)', + !isReduce, `best=${JSON.stringify(out.best)} (the reduce returned {total:3}; a string/null means it was thrown away)`); +} + +// ---------------------------------------------------------------- F5 mid-scan halt returns best:null +// KNOWN LIMIT, not fixed here: `scanCount` throws without surfacing the windows it already judged, so there is +// nothing at this seam to preserve. Fixing it is a retrieval-side change. The misleading comment that claimed +// BA-5 coverage for this case WAS corrected — this row is expected to stay REPRODUCED and is tracked as an +// open item, not papered over. +{ + const provider = scripted(() => { throw new HaltError('budget cap', { rule: 'budget' }); }).provider; + const corpus = Array.from({ length: 12 }, (_, i) => ({ id: `r${i}`, text: `record ${i}` })); + const out = await recurse('how many records mention widgets', { provider }, { corpus, retrieval: 'scan', window: 4 }); + record('F5*', 'a halt DURING the scan returns best:null (KNOWN LIMIT — comment corrected, not fixed)', + out.incomplete === true && out.best == null, `incomplete=${out.incomplete} best=${JSON.stringify(out.best)}`); +} + +// ---------------------------------------------------------------- F1 / F6 child blocker at the spawn + halt boundary +{ + let sawToolResult = null; + const provider = scripted((messages, tools, options, i) => { + const names = (tools || []).map(t => t.name); + const last = messages[messages.length - 1]; + if (last?.role === 'tool' || (typeof last?.content === 'string' && last.content.startsWith('[incomplete]'))) { + sawToolResult = last.content; + return { text: 'parent closing answer' }; + } + if (names.includes('spawn_child') && i === 0) { + return { text: '', toolCalls: [{ id: 't1', name: 'spawn_child', arguments: { subtask: 'do slice A' } }] }; + } + return { text: 'leaf output' }; + }).provider; + + const out = await recurse(COMPLEX, { provider }, { + maxDepth: 1, + refineLeaf: { sensor: () => { throw new Error('child sensor boom'); }, maxIterations: 2 }, + evaluate: () => { throw new HaltError('budget cap', { rule: 'budget' }); }, + }); + + const childNode = (out.receipts?.spawned || [])[0]; + record('F6', 'spawn boundary collapses a child broken-sensor into a generic [incomplete]', + typeof sawToolResult === 'string' && sawToolResult.startsWith('[incomplete]') && !/sensor|broken/i.test(sawToolResult), + `tool result the parent model saw = ${JSON.stringify(sawToolResult)} (child receipts blocker=${childNode?.blocker})`); + + record('F1', 'halt catch drops the child blocker (no inheritedBlocker call)', + out.incomplete === true && out.blocker === undefined && childNode?.blocker === 'broken-sensor', + `top-level blocker=${out.blocker} while child receipts blocker=${childNode?.blocker}`); +} + +// ---------------------------------------------------------------- F12 unreachable broken-verifier arm +// The premise (forChild strips `evaluate`, so no child can carry 'broken-verifier') is permanent and stays +// true — the FIX is that inheritedBlocker no longer pretends to match it. Assert the dead arm is gone. +{ + const { readFileSync } = await import('node:fs'); + const src = readFileSync(new URL('../src/recurse.js', import.meta.url), 'utf8'); + const body = (src.match(/function inheritedBlocker\([\s\S]*?\n}/) || [''])[0]; + record('F12', "inheritedBlocker still matches the unreachable 'broken-verifier'", + body.includes("'broken-verifier'"), `inheritedBlocker ${body.includes("'broken-verifier'") ? 'STILL references' : 'no longer references'} the unreachable label`); +} + +// ---------------------------------------------------------------- F10 detached evaluate changes `this` +{ + class Grader { + constructor() { this.threshold = 0.5; } + check() { return { pass: this.threshold > 0 }; } + } + const g = new Grader(); + const out = await recurse(SIMPLE, { provider: plain('x') }, { maxDepth: 0, evaluate: g.check }); + record('F10', 'an unbound method verifier now throws (this === undefined) → broken-verifier', + out.blocker === 'broken-verifier', `blocker=${out.blocker} detail=${JSON.stringify(out.receipts?.blockerDetail)}`); +} + +// ---------------------------------------------------------------- summary +console.log('='.repeat(100)); +const rep = rows.filter(r => r.reproduced); +console.log(`${rep.length}/${rows.length} findings REPRODUCED against current branch code:`); +for (const r of rep) console.log(` - ${r.id} ${r.what}`); +const not = rows.filter(r => !r.reproduced); +if (not.length) { + console.log(`\n${not.length} did NOT reproduce (drop unless re-argued):`); + for (const r of not) console.log(` - ${r.id} ${r.what}`); +} diff --git a/src/recurse.js b/src/recurse.js index 94a5bff..ec89697 100644 --- a/src/recurse.js +++ b/src/recurse.js @@ -319,6 +319,11 @@ function auditSafeCtx(ctx, overrides = {}) { * `'broken-verifier'` (BA-15) — the caller's `opts.evaluate` did (the default Evaluator path is never labeled). * @property {string} [blockerDetail] - (BA-15) with a `broken-*` blocker: what the arbiter did (threw with * which message, or which malformed shape it returned) — the actionable half of the label. + * @property {{blocker: string, blockerDetail?: string, blockerTask?: string}} [blockerFrom] - (BA-15) a + * DESCENDANT's blocker, surfaced here so an aggregating node still reports the fault upward. Deliberately + * SEPARATE from this node's own `blocker` (which means "THIS node's arbiter/Loop broke"): stamping a + * descendant's label onto every ancestor made the receipts tree accuse nodes whose sensor never ran, and + * re-labelled a parent `governance-deny` when only one child was denied. `blockerTask` names the culprit. * @property {object|null} tokens - The worker Loop's `metrics.tokens`. * @property {{iterations: number, passed: boolean, temperatures: (number|null)[], rejectedBuffer: boolean}} [refineLeaf] - (BA-8) when * this leaf ran as a bounded refine loop: how many attempts it took and whether the deterministic sensor finally @@ -358,6 +363,12 @@ function auditSafeCtx(ctx, overrides = {}) { * (non-Halt) or returned a malformed verdict; the default Evaluator path is never labeled (its failures are * provider-class faults). For both `broken-*` blockers `best` preserves the model's last non-empty output * (BA-5) — the arbiter broke, so treat it as best-effort work rather than a graded pass. + * @property {string} [blockerDetail] - (BA-15) with a `broken-*` blocker: what the arbiter did — the + * ACTIONABLE half of the label, surfaced on the result (not only in `receipts`) so a caller branching on + * `blocker` can report the cause without walking the receipts tree. + * @property {string} [blockerTask] - (BA-15) when the blocker was INHERITED from a descendant in a nested run: + * which sub-task actually broke. Without it a nested failure reports "a sensor broke" with no way to find + * which one. * @property {RecurseNode} receipts - The audit node for this call (RC-10). */ @@ -603,10 +614,8 @@ async function recurse(task, ctx = {}, opts = {}) { const missingSlices = node.spawned.filter(c => c.incomplete).map(c => c.task); if (missingSlices.length > 0) { node.incomplete = true; - // BA-15: carry a child's blocker up, so a nested broken sensor/verifier still names itself at the top. - const inherited = inheritedBlocker(node.spawned); - if (inherited) Object.assign(node, inherited); - return { incomplete: true, best: result, missingSlices, ...(inherited && { blocker: inherited.blocker }), receipts: node }; + // BA-15: carry a child's blocker up, so a nested broken sensor still names itself at the top. + return incompleteWithBlocker(node, result, { missingSlices }); } // Verify: a SEPARATE-context judge, never the generator grading itself. Runs when a contract is given, the @@ -622,7 +631,9 @@ async function recurse(task, ctx = {}, opts = {}) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - return { incomplete: true, best: result, receipts: node }; + // BA-15: a halt AFTER the children ran (e.g. mid-synthesize) must still name a child's broken sensor — + // otherwise the caller reads a governance problem where their own arbiter crashed. + return incompleteWithBlocker(node, result); } throw err; } @@ -666,15 +677,37 @@ function verdictShapeFault(v) { if (v === null || typeof v !== 'object' || Array.isArray(v)) { return `returned ${v === null ? 'null' : Array.isArray(v) ? 'an array' : `a ${typeof v}`}`; } - if (typeof v.pass === 'boolean' || SENSOR_STATUS.has(v.status)) return null; + // A PRESENT `pass` counts whatever its type (`1`/`0`/`'yes'` are a long-standing yes-no convention, and + // `refine` has always branched on its TRUTHINESS). BA-15 exists to catch a verdict carrying NO usable + // signal — not to reject one that answers clearly in a different dialect: demanding a strict boolean + // silently flipped previously-CONVERGING adopter sensors to a permanent first-attempt block. + if (v.pass != null || SENSOR_STATUS.has(v.status)) return null; const keys = Object.keys(v).slice(0, 5).join(', '); - return `returned an object with neither a boolean \`pass\` nor a valid \`status\` (keys: ${keys || 'none'})`; + return `returned an object with neither a usable \`pass\` nor a valid \`status\` (keys: ${keys || 'none'})`; +} + +/** + * BA-15 — a readable one-liner for ANYTHING thrown by a caller arbiter. `String(err)` on a non-Error throw + * (a test harness rejecting with a raw `{code:'ENOENT', path}` result is the common case) yields the useless + * `[object Object]`, defeating `blockerDetail`'s entire purpose: naming what broke so the operator can fix it. + * @param {any} err + * @returns {string} + */ +function describeThrown(err) { + if (err instanceof Error && err.message) return err.message; + if (err === null || typeof err !== 'object') return String(err); + try { + const json = JSON.stringify(err); + if (json && json !== '{}') return json; + } catch { /* circular, or a throwing toJSON — fall through to the tag */ } + return Object.prototype.toString.call(err); } /** * BA-15 — run a caller arbiter (the `refineLeaf.sensor` or the `opts.evaluate` verifier) and NAME a broken one * rather than let it launder. A non-`HaltError` throw or a malformed return (per {@link verdictShapeFault}) is - * re-thrown with a `: ` prefix the call sites classify on; a `HaltError` passes straight through (clean + * re-thrown as a typed {@link BrokenArbiterError} the call sites classify by `instanceof` + `.tag` (NEVER by + * re-parsing the message text); a `HaltError` passes straight through (clean * governance exit, BA-2). One helper so the sensor and verifier seams stay byte-identical (same guidance * string, same fault taxonomy) — a divergence between them was the copy-paste risk this replaces. * A verdict that carries a valid `status` but no boolean `pass` is NORMALIZED to add `pass = status === @@ -682,47 +715,124 @@ function verdictShapeFault(v) { * WORK — `refine.js` stops the leaf on `verdict.pass`, so a bare `{status:'satisfied'}` would otherwise never * satisfy, burn every iteration, and report `passed:false` (a satisfied close mislabeled as non-recovery). * @template {{pass?: boolean, status?: string}} T - * @param {'broken-sensor'|'broken-verifier'} tag - the fault channel; call sites strip `: ` to classify. + * @param {'broken-sensor'|'broken-verifier'} tag - the fault channel, carried on the thrown error's `.tag`. * @param {() => (T | Promise)} call - invokes the arbiter (already bound to its result/ctx args). * @returns {Promise} the arbiter's well-formed verdict (with `pass` derived from `status` when absent). */ async function runArbiter(tag, call) { const who = tag === 'broken-sensor' ? 'sensor' : 'evaluate'; - // The SHAPE INSPECTION runs inside the try too: reading `.status`/`.pass`/`Object.keys` on a returned Proxy - // or accessor-backed object can itself throw, and outside the try that escapes as an untyped error — i.e. - // the very uncaught-crash this seam exists to prevent, one step later. + const shape = `a ${who === 'sensor' ? 'sensor' : 'verifier'} must return {pass} or {status: 'satisfied'|'needs_revision'|'failed'}`; + /** A throw from the arbiter's own body (sync or async rejection) — "the sensor threw". */ + const threw = (err) => { + if (err instanceof HaltError || err instanceof BrokenArbiterError) return err; + // `cause` keeps the original error (stack + type) reachable for debugging; `detail` stays the short label. + return new BrokenArbiterError(tag, `${who} threw: ${describeThrown(err)}`, { cause: err }); + }; + /** A throw from READING the value the arbiter returned — "the verdict is unreadable", a different fault. */ + const unreadable = (err) => { + if (err instanceof HaltError || err instanceof BrokenArbiterError) return err; + return new BrokenArbiterError(tag, `${who} returned a verdict whose properties could not be read: ${describeThrown(err)} — ${shape}`, { cause: err }); + }; + + // Typed `any` deliberately: this block probes an UNTRUSTED caller return (it may be a promise, a plain + // verdict, or a hostile Proxy), so the generic `T | Promise` narrowing does not apply until it is settled. + /** @type {any} */ + let raw; + try { + raw = call(); + } catch (err) { throw threw(err); } + + // `await`ing directly would conflate two different faults: the await PROBES `.then` on the returned value, + // so an accessor-backed/Proxy verdict throws during the await and gets reported as "the sensor threw" — + // sending the operator hunting a `throw` in a sensor that returned perfectly normally. Probe the thenable + // separately (a throw HERE is the returned value being unreadable) and only then await (a rejection THERE + // is genuinely the arbiter's body failing). + let thenable; + try { + thenable = raw !== null && (typeof raw === 'object' || typeof raw === 'function') && typeof raw.then === 'function'; + } catch (err) { throw unreadable(err); } + + /** @type {any} */ + let v = raw; + if (thenable) { + try { + v = await raw; + } catch (err) { throw threw(err); } + } + // The SHAPE INSPECTION is guarded SEPARATELY: reading `.status`/`.pass`/`Object.keys` on a returned Proxy or + // accessor-backed object can itself throw. Unguarded that escapes untyped (the very uncaught crash this seam + // prevents, one step later) — but folding it into the call's own catch is also wrong: it reports "the sensor + // threw" for a sensor that RETURNED NORMALLY, sending the operator hunting a `throw` that does not exist. + // The fault is in the returned VALUE, so it is named as such. try { - const v = await call(); const fault = verdictShapeFault(v); - if (fault) throw new BrokenArbiterError(tag, `${who} ${fault} — a ${who === 'sensor' ? 'sensor' : 'verifier'} must return {pass: boolean} or {status: 'satisfied'|'needs_revision'|'failed'}`); - // Derive `pass` from a status-only verdict (copy, never mutate the caller's object) so the advertised - // `{status}` shape actually gates `refine`/downstream, which branch on `verdict.pass`. - if (typeof v.pass !== 'boolean' && typeof v.status === 'string') { - return /** @type {T} */ ({ ...v, pass: v.status === 'satisfied' }); + if (fault) throw new BrokenArbiterError(tag, `${who} ${fault} — ${shape}`); + // Derive `pass` from a status-only verdict so the advertised `{status}` shape actually gates `refine`, which + // branches on `verdict.pass`. NEVER mutate the caller's object — and never flatten it either: an object + // spread copies OWN enumerable properties only, so a class-instance verdict whose `status`/`critique` are + // PROTOTYPE getters passes the shape check above and then comes out the other side with those fields + // ERASED (critique lost ⇒ every retry re-sends the plain task with zero feedback — the exact burn BA-15 + // exists to prevent). Copy descriptors onto the SAME prototype so accessor-backed fields survive. + if (v.pass == null && typeof v.status === 'string') { + const copy = Object.create(Object.getPrototypeOf(v), Object.getOwnPropertyDescriptors(v)); + Object.defineProperty(copy, 'pass', { + value: v.status === 'satisfied', writable: true, enumerable: true, configurable: true, + }); + return /** @type {T} */ (copy); } return v; - } catch (err) { - if (err instanceof HaltError || err instanceof BrokenArbiterError) throw err; - // `cause` keeps the original error (stack + type) reachable for debugging; `detail` stays the short label. - throw new BrokenArbiterError(tag, `${who} threw: ${err && err.message ? err.message : String(err)}`, { cause: err }); - } + } catch (err) { throw unreadable(err); } } /** * BA-15 — surface a CHILD's blocker at the PARENT. A nested tree aggregates a dead child into - * `{incomplete, missingSlices}`; without this the child's `broken-sensor`/`broken-verifier` (or - * `governance-deny`) label is dropped at the first parent, so a top-level caller branching on - * `result.blocker` sees nothing and debugs the model instead of its own sensor — the exact laundering BA-15 - * exists to close, reintroduced one level up. A `broken-*` label wins over `governance-deny`: it names a - * fault in the CALLER's own code, which is both more actionable and cheaper to fix than a gate decision. + * `{incomplete, missingSlices}`; without this the child's `broken-sensor` (or `governance-deny`) label is + * dropped at the first parent, so a top-level caller branching on `result.blocker` sees nothing and debugs + * the model instead of its own sensor — the exact laundering BA-15 exists to close, reintroduced one level up. + * `broken-sensor` wins over `governance-deny`: it names a fault in the CALLER's own code, which is both more + * actionable and cheaper to fix than a gate decision. + * + * `blockerTask` names WHICH descendant broke. Without it a nested run reports "a sensor broke" with no way to + * find which one — half a fix. (It is also why the label is NOT stamped onto the parent's own `blocker`; see + * {@link incompleteWithBlocker}.) + * + * `'broken-verifier'` is deliberately NOT matched here: `forChild` strips `evaluate`, so a child never runs a + * caller verifier and no child node can carry that label. Matching it would be unreachable code asserting a + * capability that does not exist. * @param {RecurseNode[]} spawned - this node's child receipts. - * @returns {{blocker: string, blockerDetail?: string}|null} + * @returns {{blocker: string, blockerDetail?: string, blockerTask?: string}|null} */ function inheritedBlocker(spawned) { - const pick = spawned.find(c => c.incomplete && (c.blocker === 'broken-sensor' || c.blocker === 'broken-verifier')) + const pick = spawned.find(c => c.incomplete && c.blocker === 'broken-sensor') || spawned.find(c => c.incomplete && c.blocker); if (!pick || !pick.blocker) return null; - return pick.blockerDetail ? { blocker: pick.blocker, blockerDetail: pick.blockerDetail } : { blocker: pick.blocker }; + return { + blocker: pick.blocker, + ...(pick.blockerDetail && { blockerDetail: pick.blockerDetail }), + ...(pick.task && { blockerTask: pick.task }), + }; +} + +/** + * BA-15 — the ONE shared "this node is incomplete because a descendant was" return. Used by every aggregating + * path (worker / partition / fanout) on BOTH its `missingSlices` branch AND its `HaltError` catch: the halt + * branches originally skipped the inherit entirely, so a gate tripping AFTER a broken-sensor child (e.g. mid- + * synthesize) dropped the label and the caller read a governance problem where their own sensor had crashed. + * + * The inherited label rides the RETURNED result (the caller-facing API — that surfacing is the whole point) + * but is recorded on the node as `blockerFrom`, NOT by overwriting `node.blocker`. `node.blocker` means "THIS + * node's own arbiter/Loop broke"; stamping a descendant's label onto every ancestor made the receipts tree + * accuse nodes whose sensor never ran, and re-labelled a parent `governance-deny` when only one child was + * denied — pointing the operator at the wrong node to re-gate. + * @param {RecurseNode} node + * @param {any} best + * @param {{missingSlices?: string[]}} [extra] - extra fields for the non-halt aggregation branch. + * @returns {RecurseResult} + */ +function incompleteWithBlocker(node, best, extra = {}) { + const inherited = inheritedBlocker(node.spawned); + if (inherited) node.blockerFrom = inherited; + return /** @type {RecurseResult} */ ({ incomplete: true, best, ...extra, ...(inherited || {}), receipts: node }); } /** @@ -897,7 +1007,10 @@ async function recurseRefineLeaf(task, ctx, opts, state) { node.incomplete = true; node.blocker = 'broken-sensor'; node.blockerDetail = err.detail; - return { incomplete: true, best: lastAttemptText, blocker: 'broken-sensor', receipts: node }; + // `blockerDetail` rides the RESULT too, not just receipts: it is the actionable half of the label (WHAT + // the sensor did), and a parent reading a child's return — or a caller branching on the documented + // `{blocker}` shape — should not have to walk the receipts tree to get it. + return { incomplete: true, best: lastAttemptText, blocker: 'broken-sensor', blockerDetail: err.detail, receipts: node }; } node.incomplete = true; return { incomplete: true, best: lastAttemptText, receipts: node }; @@ -988,8 +1101,12 @@ async function recurseScan(task, ctx, opts, state) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - // BA-5: a halt from the verify slot lands here with the scan already finished — return the counted - // result, never null (the count survived only in receipts before, forcing a full re-scan). + // BA-5, PARTIAL by construction: a halt from the VERIFY slot lands here with the scan already finished, + // so the counted result is returned instead of null (it previously survived only in receipts, forcing a + // full re-scan). A halt DURING `scanCount` — the likelier case, since a token cap trips after many window + // judge calls — still yields `null`: `scanCount` throws without surfacing the windows it did judge, so + // there is nothing here to preserve. Fixing that needs `scanCount` to return its partial union on halt + // (a retrieval-side change, not this seam's); tracked as a known limit rather than papered over here. return { incomplete: true, best: scanResult, receipts: node }; } throw err; @@ -1100,9 +1217,7 @@ async function recursePartition(task, ctx, opts, state) { if (missingSlices.length > 0) { node.incomplete = true; // BA-15: carry a child's blocker up (see inheritedBlocker). - const inherited = inheritedBlocker(node.spawned); - if (inherited) Object.assign(node, inherited); - return { incomplete: true, best: result, missingSlices, ...(inherited && { blocker: inherited.blocker }), receipts: node }; + return incompleteWithBlocker(node, result, { missingSlices }); } const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { @@ -1118,7 +1233,8 @@ async function recursePartition(task, ctx, opts, state) { // rebuild a best-effort count from whatever slices did return (BA-5, never a bare null). const best = partitionResult || (childResults.length ? { count: new Set(childResults.flatMap((v) => (v && Array.isArray(v.matchedIds) ? v.matchedIds : []))).size } : null); - return { incomplete: true, best, receipts: node }; + // BA-15: a halt must not swallow a child's broken-sensor label (see incompleteWithBlocker). + return incompleteWithBlocker(node, best); } throw err; } @@ -1152,6 +1268,11 @@ async function recurseFanout(task, ctx, opts, state) { const childResults = []; const contract = typeof opts.contract === 'string' ? opts.contract : null; + // BA-5: declared OUTSIDE the try so a halt from the verify slot (which lands in the catch below, with the + // reduce already computed and PAID FOR) returns the finished reduce instead of a lossy re-join of the raw + // child strings — a different TYPE from the documented reduce output, and an 'merge' strategy's LLM call + // thrown away. Mirrors `scanResult`/`partitionResult` on the sibling paths. + let result; try { // 1) Decompose into exactly `count` independent parallel steps (the NB-2 Planner seam). A non-Halt planner @@ -1222,7 +1343,6 @@ async function recurseFanout(task, ctx, opts, state) { // 4) NB-3 reduce over the slice results. Unlike Family A there is no parent closing turn, so we ALWAYS // reduce: a `synthesize` FUNCTION is the deterministic code-reduce (§9.1); a string runs the built-in // reducer; unset defaults to lossless `'concat'`. (`childResults` always has `count` entries.) - let result; if (typeof opts.synthesize === 'function') { result = await opts.synthesize({ task, text: null, results: childResults, children: node.spawned, ctx }); } else { @@ -1243,9 +1363,7 @@ async function recurseFanout(task, ctx, opts, state) { if (missingSlices.length > 0) { node.incomplete = true; // BA-15: carry a child's blocker up (see inheritedBlocker). - const inherited = inheritedBlocker(node.spawned); - if (inherited) Object.assign(node, inherited); - return { incomplete: true, best: result, missingSlices, ...(inherited && { blocker: inherited.blocker }), receipts: node }; + return incompleteWithBlocker(node, result, { missingSlices }); } // 6) Verify (RC-7): forced for critical, or when a contract/override is supplied. @@ -1259,9 +1377,12 @@ async function recurseFanout(task, ctx, opts, state) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - // best-effort partial: whatever slices we did collect, losslessly joined (no LLM — the gate already tripped) - const best = childResults.length ? childResults.filter(v => v !== '').join('\n\n') : null; - return { incomplete: true, best: best || null, receipts: node }; + // BA-5: prefer the FINISHED reduce when the halt landed after it (the verify slot); only fall back to a + // lossless join of the raw slices when the halt tripped before/during the reduce itself. Never a bare null. + const rejoined = childResults.length ? childResults.filter(v => v !== '').join('\n\n') : null; + const best = result !== undefined ? result : (rejoined || null); + // BA-15: a halt must not swallow a child's broken-sensor label (see incompleteWithBlocker). + return incompleteWithBlocker(node, best); } throw err; } @@ -1328,6 +1449,16 @@ function buildSpawnTool(ctx, opts, depth, maxDepth, node, childResults) { // silently dropped or faked. The same declared value is collected for the NB-3 reducer. const value = child.incomplete ? (child.best == null ? '' : child.best) : (child.result == null ? '' : child.result); childResults.push(value); + // BA-15: a child blocked by a BROKEN ARBITER must not read to the parent model as a generic failure. + // Collapsed into a bare `[incomplete]`, the parent cannot tell "the model failed" from "the judge is + // broken", so it re-spawns the identical subtask against the identical broken sensor — re-running a + // full leaf attempt each time, up to the Loop's round limit. That is BA-15's own spend-burn ("retrying + // against a broken arbiter carries zero feedback") reintroduced one level up, so the tool result says + // so explicitly and tells the model not to retry. + if (child.incomplete && child.blocker === 'broken-sensor') { + const where = child.blockerTask ? ` (in sub-task: ${child.blockerTask})` : ''; + return `[blocked: broken-sensor] The CHECK that judges this subtask is itself broken${where} — ${child.blockerDetail || 'no detail available'}. Retrying will hit the same broken check: do NOT re-delegate this subtask; report it as blocked. Partial output: ${String(value)}`.trim(); + } if (child.incomplete) return `[incomplete] ${String(value)}`.trim(); return String(value); }, @@ -1358,7 +1489,11 @@ function verify(task, result, ctx, opts) { // NOT wrapped: it constructs well-formed Verdicts by design, and its failures are provider-class faults. // Capture the narrowed reference in a const: `typeof opts.evaluate === 'function'` does NOT survive into the // nested async closure (TS re-widens `opts.evaluate` to possibly-undefined there → TS2722/TS18048). - const evaluate = opts.evaluate; + // BOUND to `opts`, because a bare `const evaluate = opts.evaluate` DETACHES the method: the call used to be + // `opts.evaluate(...)` (receiver `opts`), and a caller passing a class method (`evaluate: grader.check`) + // would suddenly get `this === undefined` and throw on its first `this.x` read — a working (if degraded) + // integration flipped to a hard `broken-verifier` by an unrelated typecheck fix. + const evaluate = opts.evaluate.bind(opts); return runArbiter('broken-verifier', () => evaluate(result, { contract, task })); } const provider = ctx.provider || opts.provider; @@ -1404,7 +1539,8 @@ async function verifyOrBlock(task, result, ctx, opts, node) { node.incomplete = true; node.blocker = 'broken-verifier'; node.blockerDetail = err.detail; - return { incomplete: true, best: result, blocker: 'broken-verifier', receipts: node }; + // `blockerDetail` rides the RESULT too (see the sensor-side sibling) — the actionable half of the label. + return { incomplete: true, best: result, blocker: 'broken-verifier', blockerDetail: err.detail, receipts: node }; } } diff --git a/test/recurse.test.js b/test/recurse.test.js index 2b4cfb2..c696004 100644 --- a/test/recurse.test.js +++ b/test/recurse.test.js @@ -688,7 +688,7 @@ describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, refineLeaf: { sensor: () => ({}), maxIterations: 3 }, }); assert.equal(out.blocker, 'broken-sensor', 'a garbage verdict is named, never coerced to pass:false'); - assert.match(out.receipts.blockerDetail, /neither a boolean `pass` nor a valid `status`/); + assert.match(out.receipts.blockerDetail, /neither a usable `pass` nor a valid `status`/); assert.equal(sp.calls.length, 1, 'pre-fix: 3 attempts burned, every retry a feedback-free re-send of the plain task'); assert.equal(out.result, undefined, 'never a converged-shaped return riding a garbage verdict'); }); @@ -792,7 +792,12 @@ describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, }); assert.equal(out.incomplete, true, 'a broken sensor at the leaf makes the tree incomplete'); assert.equal(out.blocker, 'broken-sensor', 'the child\'s label reaches the top (pre-fix: undefined)'); - assert.equal(out.receipts.blocker, 'broken-sensor', 'and is recorded on the parent receipts'); + // The parent's OWN `blocker` stays clear: it means "THIS node's arbiter broke", and the parent's never ran. + // Stamping the descendant's label onto every ancestor made the receipts tree accuse innocent nodes, so the + // inherited label rides `blockerFrom` (with `blockerTask` naming the culprit) instead. + assert.equal(out.receipts.blocker, undefined, 'the parent is not accused of its child\'s fault'); + assert.equal(out.receipts.blockerFrom.blocker, 'broken-sensor', 'recorded on the parent as INHERITED'); + assert.equal(typeof out.blockerTask, 'string', 'and the culprit subtask is named'); }); it('a verify-slot halt after a PASSING sensor does not clobber the refineLeaf receipt to passed:false', async () => { @@ -853,7 +858,7 @@ describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, }); assert.equal(out.blocker, 'broken-verifier', 'pre-fix: {result, verdict:{}} — a converged shape riding garbage'); assert.equal(out.result, undefined, 'no result field on the labeled incomplete'); - assert.match(out.receipts.blockerDetail, /neither a boolean `pass` nor a valid `status`/); + assert.match(out.receipts.blockerDetail, /neither a usable `pass` nor a valid `status`/); }); it('under refineLeaf a broken verifier is labeled broken-VERIFIER (distinct from broken-sensor)', async () => { @@ -898,6 +903,153 @@ describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, assert.equal(out.blocker, 'broken-verifier', 'the fanout path labels a broken verifier identically'); assert.ok(out.best, 'the reduced slices are preserved as best'); }); + + // ---------------------------------------------------------------- round-3 review regressions + // Each of the following reproduces a defect the third review round found in BA-15's OWN fixes, validated + // against the pre-fix code by `poc/ba15-round3-validate.mjs` before being fixed here. + + it('a PROTOTYPE-backed verdict keeps its status/critique (spread would erase accessor fields)', async () => { + // A class-instance verdict passes the shape check by reading `.status` through a getter, but an object + // spread copies OWN enumerable props only — so status/critique came out ERASED: the critique never reached + // the retry prompt (zero-feedback burn, the exact thing BA-15 exists to stop) and `verdict.status` read + // undefined to the caller. Regression-proof: revert to `{...v, pass}` and both asserts go red. + class V { + constructor(s, c) { this._s = s; this._c = c; } + get status() { return this._s; } + get critique() { return this._c; } + } + const sp = scriptedProvider(() => ({ text: 'the work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => new V('satisfied', 'looks good'), + }); + assert.equal(out.verdict.status, 'satisfied', 'prototype getter survives normalization'); + assert.equal(out.verdict.critique, 'looks good', 'the critique is not erased'); + assert.equal(out.verdict.pass, true, 'and `pass` is still derived from the status'); + }); + + it('a TRUTHY non-boolean `pass` is accepted, not hard-blocked as a faulty arbiter', async () => { + // `{pass: 1}` is a long-standing yes/no convention and `refine` has always branched on truthiness, so a + // strict-boolean gate silently flipped previously-CONVERGING adopter sensors to a permanent first-attempt + // block. BA-15's job is a verdict with NO usable signal — not one answering clearly in another dialect. + const sp = scriptedProvider(() => ({ text: 'attempt output' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ pass: 1, critique: '' }), maxIterations: 3 }, + }); + assert.equal(out.blocker, undefined, 'a truthy pass is a real verdict, not a broken arbiter'); + assert.equal(out.incomplete, undefined, 'the leaf converges as it did before BA-15'); + assert.equal(out.receipts.refineLeaf.passed, true, 'and is recorded as an honest pass'); + }); + + it('a NON-ERROR throw still yields an actionable blockerDetail (never [object Object])', async () => { + const sp = scriptedProvider(() => ({ text: 'work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw { code: 'ENOENT', path: '/usr/bin/vitest' }; } }, + }); + assert.equal(out.blocker, 'broken-sensor'); + assert.doesNotMatch(out.receipts.blockerDetail, /\[object Object\]/, 'the detail must identify something'); + assert.match(out.receipts.blockerDetail, /ENOENT/, 'the thrown value is described'); + }); + + it('an UNREADABLE returned verdict is named as such, not reported as the sensor throwing', async () => { + // A sensor returning a Proxy whose accessors throw RETURNED NORMALLY — reporting "sensor threw" sends the + // operator hunting a `throw` that does not exist. NB the fault surfaces during the `await`'s own `.then` + // probe, so the two cases must be separated around the await, not just around the shape inspection. + const sp = scriptedProvider(() => ({ text: 'work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => new Proxy({}, { get() { throw new Error('trap'); } }) }, + }); + assert.equal(out.blocker, 'broken-sensor'); + assert.match(out.receipts.blockerDetail, /could not be read/, 'the fault is the RETURNED VALUE, not a throw'); + assert.doesNotMatch(out.receipts.blockerDetail, /threw: trap/, 'must not accuse the sensor body'); + }); + + it('a method-reference verifier does not hard-fail as a broken arbiter (receiver restored)', async () => { + // `const evaluate = opts.evaluate` DETACHED the method; the call had always been `opts.evaluate(...)`, so + // `this` was the OPTIONS object. Detached, `this` became undefined and the first `this.x` read THREW — + // flipping a working (if degraded) integration to a hard broken-verifier. Binding back to `opts` restores + // the prior behavior exactly. NOTE the honest limit that restores: `this` is the options object, never the + // grader instance, so a method depending on its own fields still reads undefined. A caller wanting its own + // receiver must bind (`evaluate: g.check.bind(g)`) — asserted below so the contract is not overstated. + class Grader { + constructor() { this.threshold = 0.5; } + // UNGUARDED on purpose: detached (`this === undefined`) this THROWS, which is the regression. Bound to + // `opts` it reads undefined and degrades to false — exactly the pre-BA-15 behavior being restored. + check() { return { pass: this.threshold > 0, critique: '' }; } + } + const sp = scriptedProvider(() => ({ text: 'work' })); + const g = new Grader(); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { contract: 'c', evaluate: g.check }); + assert.equal(out.blocker, undefined, 'a method reference must not become a broken-verifier (pre-fix: it did)'); + assert.equal(out.verdict.pass, false, 'degraded exactly as before: `this` is opts, so threshold is undefined'); + + const bound = await recurse(SIMPLE_TASK, { provider: scriptedProvider(() => ({ text: 'work' })).provider }, { + contract: 'c', evaluate: g.check.bind(g), + }); + assert.equal(bound.verdict.pass, true, 'a caller-bound method sees its own fields — the documented way'); + }); + + it('a HALT after a broken-sensor child still names the child\'s blocker (halt catch inherits too)', async () => { + // The halt branches skipped the inherit entirely, so a gate tripping AFTER the children ran (here: mid- + // synthesize) dropped the label — the caller read a governance problem where their own sensor had crashed. + const sp = scriptedProvider(decomposingHandler()); + const out = await recurse(COMPLEX_TASK, { provider: sp.provider }, { + maxDepth: 2, + refineLeaf: { sensor: () => { throw new Error('child sensor boom'); } }, + synthesize: () => { throw new HaltError('budget cap', { rule: 'budget' }); }, + }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true, 'it really was a halt'); + assert.equal(out.blocker, 'broken-sensor', 'the child\'s label survives the halt (pre-fix: undefined)'); + }); + + it('the spawn boundary tells the parent a child was BLOCKED, not just incomplete', async () => { + // Collapsed to a bare `[incomplete]`, the parent model cannot tell "the model failed" from "the judge is + // broken", so it re-delegates the same subtask into the same broken sensor, burning a full leaf attempt + // each time — BA-15's own spend-burn, one level up. + const sp = scriptedProvider(decomposingHandler()); + await recurse(COMPLEX_TASK, { provider: sp.provider }, { + maxDepth: 2, + refineLeaf: { sensor: () => { throw new Error('child sensor boom'); } }, + }); + const toolResults = sp.calls.flatMap(c => c.messages.filter(m => m.role === 'tool').map(m => m.content)); + const blocked = toolResults.find(t => typeof t === 'string' && t.includes('broken-sensor')); + assert.ok(blocked, 'the parent model is told the ARBITER broke, not merely that the child failed'); + assert.match(blocked, /do NOT re-delegate/i, 'and is told retrying hits the same broken check'); + }); + + it('a broken-sensor child OUTRANKS a governance-denied sibling, and the parent is not accused', async () => { + // Two rules that had NO test before: (a) broken-* wins over governance-deny (a fault in the CALLER's own + // code is more actionable than a gate decision); (b) the inherited label rides `blockerFrom`, never the + // parent's own `blocker`. Driven through the public API, not an exported internal. + // slice 0 → spins on a denied tool ⇒ governance-deny (and is FIRST, so a naive single `find` picks it) + // slice 1 → its sensor throws ⇒ broken-sensor + // Mutation-proof for (a): collapse inheritedBlocker to one `find(c => c.incomplete && c.blocker)` and the + // first assert goes red (governance-deny wins on array order). + const editTool = { name: 'edit', description: 'edit a file', execute: async () => 'edited' }; + const sp = scriptedProvider((messages, tools, options, i) => { + if (isPlanner(messages)) { + return { text: JSON.stringify([0, 1].map(j => ({ id: `s${j}`, action: `SLICE ${j}: count the alpha records`, dependsOn: [] }))) }; + } + if (isVerify(messages)) return { text: SATISFIED }; + // slice 0's worker keeps calling the denied tool → the Loop's deny-spin guard short-circuits it + if (lastUser(messages).includes('SLICE 0')) return { toolCalls: [{ id: 'e' + i, name: 'edit', arguments: { n: i } }] }; + return { text: 'slice 1 output' }; + }); + const out = await recurse(COMPLEX_TASK, { + provider: sp.provider, + policy: (tool) => (tool === 'edit' ? '[deny: fs.writeScope] out of scope' : true), + }, { + count: 2, + tools: [editTool], + refineLeaf: { sensor: (_r, c) => { if (String(c.task).includes('SLICE 1')) throw new Error('slice-1 sensor boom'); return { pass: true }; } }, + }); + + assert.equal(out.incomplete, true); + assert.equal(out.blocker, 'broken-sensor', 'broken-sensor outranks the FIRST-listed governance-deny sibling'); + assert.match(out.blockerTask, /SLICE 1/, 'and names WHICH sub-task broke'); + assert.equal(out.receipts.blocker, undefined, 'the parent is not accused of its child\'s fault'); + assert.equal(out.receipts.blockerFrom.blocker, 'broken-sensor', 'the inherited label is recorded separately'); + }); }); describe('recurse — topology knob & capability-scrub (RC-11 / RC-12 / NB-4)', () => { From d8790a89c78e9c1793b5bb05a59083dcaeb44ca5 Mon Sep 17 00:00:00 2001 From: hamr0 Date: Mon, 20 Jul 2026 12:22:34 +0200 Subject: [PATCH 7/8] =?UTF-8?q?fix:=20bound=20blockerDetail=20=E2=80=94=20?= =?UTF-8?q?a=20non-Error=20throw=20dumped=20caller=20secrets=20to=20the=20?= =?UTF-8?q?audit=20log?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit /security on the round-3 changes caught a leak the round-3 fix itself introduced. describeThrown used JSON.stringify on a non-Error throw, and a thrown non-Error is typically a spawn/exec RESULT carrying a full stdout buffer and an env snapshot. blockerDetail rides into receipts, which a wired gate serializes VERBATIM into a plaintext audit log — so this wrote caller secrets to disk. Measured: a 200KB detail string containing an sk-live- token. Same leak class as F16/BA-1's audit-ctx provider strip. Now takes only conventional diagnostic fields (name/code/errno/syscall/path/ status/signal/message), clamped to 200 chars — still names the fault (code=ENOENT) without dumping the payload. Measured after: 25 chars, no secret. Prototype pollution via the new descriptor-copy path was probed separately and is clean. +1 mutation-proven regression test (restore the JSON.stringify dump and it goes red). Suite 890 pass / 0 fail; typecheck clean by exit code. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 4 +++- src/recurse.js | 21 ++++++++++++++++----- test/recurse.test.js | 16 ++++++++++++++++ 3 files changed, 35 insertions(+), 6 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d5a9dd3..3337acb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -46,13 +46,15 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog - **Diagnostics:** a non-`Error` throw (a harness rejecting with `{code:'ENOENT', path}`) rendered as `[object Object]`, naming nothing; and a verdict whose *accessor* throws was reported as "the sensor threw", sending the operator hunting a `throw` in a sensor that returned normally. The two faults are now distinguished around the `await` itself — which probes `.then` and so triggers the accessor before any shape check runs. - **An unreachable branch was documented as working.** `inheritedBlocker` matched `'broken-verifier'`, but `forChild` strips `evaluate`, so no child can ever carry that label. Dead branch removed and this entry corrected — the claim above now reads `broken-sensor` only. + A `/security` pass on the round-3 changes then caught a leak the round-3 fix had itself introduced: describing a non-`Error` throw via `JSON.stringify` dumped the **whole** thrown object into `blockerDetail`, and a thrown non-`Error` is typically a spawn/exec result carrying a full stdout buffer and an env snapshot. Since `blockerDetail` rides into `receipts` — which a wired gate serializes **verbatim into a plaintext audit log** — that wrote caller secrets to disk (measured: a 200 KB detail string containing an `sk-live-…` token; the same leak class as F16/BA-1's audit-ctx provider strip). `describeThrown` now takes only conventional diagnostic fields (`name`/`code`/`errno`/`syscall`/`path`/`status`/`signal`/`message`), clamped to 200 chars — still naming the fault (`code=ENOENT`) without dumping the payload. Prototype pollution via the new descriptor-copy path was probed and is clean. + **Known limit (documented, not fixed):** a `HaltError` raised *during* `scanCount` — the likelier halt, since a token cap trips after many window judge calls — still returns `best: null`. `scanCount` throws without surfacing the windows it already judged, so there is nothing at the recurse seam to preserve; fixing it is a retrieval-side change. The comment that previously implied BA-5 coverage here has been corrected rather than left to mislead. **Behavior change for adopters:** `opts.evaluate` is now held to its documented `Verdict` return. A verifier returning a loose object (`{score, critique}`) or a bare boolean — always a contract violation, previously silent — now yields `{incomplete, blocker:'broken-verifier'}` instead of riding out as a converged `{result, verdict}`. That is the point of the change (a converged shape carrying an ungradeable verdict is the laundering being closed), but it is a real break for callers who relied on the loose path; no shim is provided, since one would reopen the hole. A verdict carrying a **truthy non-boolean** `pass` is *not* affected — that shape is accepted (see the third round above). **Known open (pre-existing, not from this change):** `instanceof HaltError` is realm-sensitive — a `HaltError` from a different module realm (duplicate install, VM context) would not match, at 23 sites that predate this work. Documented rather than force-fixed. - New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` and `blockerDetail` on `RecurseResult`; `blockerTask` on `RecurseResult` when the blocker was inherited from a descendant; `receipts.blockerDetail` and `receipts.blockerFrom`. +28 mutation-checked tests; full suite **889 pass / 0 fail**, typecheck clean (verified by exit code). Pre-fix evidence: `poc/ba15-broken-sensor.mjs` (rounds 1–2) and `poc/ba15-round3-validate.mjs` (round 3 — reproduces each finding against the unfixed branch, so a post-fix re-run flips them). + New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` and `blockerDetail` on `RecurseResult`; `blockerTask` on `RecurseResult` when the blocker was inherited from a descendant; `receipts.blockerDetail` and `receipts.blockerFrom`. +29 mutation-checked tests; full suite **890 pass / 0 fail**, typecheck clean (verified by exit code). Pre-fix evidence: `poc/ba15-broken-sensor.mjs` (rounds 1–2) and `poc/ba15-round3-validate.mjs` (round 3 — reproduces each finding against the unfixed branch, so a post-fix re-run flips them). ## [0.30.0] - 2026-07-16 diff --git a/src/recurse.js b/src/recurse.js index ec89697..6bf3f19 100644 --- a/src/recurse.js +++ b/src/recurse.js @@ -686,6 +686,9 @@ function verdictShapeFault(v) { return `returned an object with neither a usable \`pass\` nor a valid \`status\` (keys: ${keys || 'none'})`; } +/** Upper bound on a `blockerDetail` fragment — it rides into receipts and, via a wired gate, onto disk. */ +const DETAIL_MAX = 200; + /** * BA-15 — a readable one-liner for ANYTHING thrown by a caller arbiter. `String(err)` on a non-Error throw * (a test harness rejecting with a raw `{code:'ENOENT', path}` result is the common case) yields the useless @@ -694,12 +697,20 @@ function verdictShapeFault(v) { * @returns {string} */ function describeThrown(err) { - if (err instanceof Error && err.message) return err.message; - if (err === null || typeof err !== 'object') return String(err); + const clamp = (s) => (s.length > DETAIL_MAX ? `${s.slice(0, DETAIL_MAX)}… (truncated)` : s); + if (err instanceof Error && typeof err.message === 'string' && err.message) return clamp(err.message); + if (err === null || typeof err !== 'object') return clamp(String(err)); + // Deliberately NOT a whole-object dump. A thrown non-Error is typically a spawn/exec RESULT, which routinely + // carries a full stdout buffer and an env snapshot — and `blockerDetail` rides into `receipts`, which a wired + // gate serializes VERBATIM into a plaintext audit log (the F16/BA-1 lesson: never let caller data of unknown + // shape reach the audit unfiltered). Take only the conventional diagnostic fields, clamped. try { - const json = JSON.stringify(err); - if (json && json !== '{}') return json; - } catch { /* circular, or a throwing toJSON — fall through to the tag */ } + const picked = ['name', 'code', 'errno', 'syscall', 'path', 'status', 'signal', 'message'] + .filter(k => typeof err[k] === 'string' || typeof err[k] === 'number') + .map(k => `${k}=${String(err[k])}`) + .join(' '); + if (picked) return clamp(picked); + } catch { /* a throwing accessor / Proxy trap — fall through to the type tag */ } return Object.prototype.toString.call(err); } diff --git a/test/recurse.test.js b/test/recurse.test.js index c696004..87b87ac 100644 --- a/test/recurse.test.js +++ b/test/recurse.test.js @@ -950,6 +950,22 @@ describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, assert.match(out.receipts.blockerDetail, /ENOENT/, 'the thrown value is described'); }); + it('blockerDetail is BOUNDED and does not dump the thrown object wholesale (audit-log safety)', async () => { + // A thrown non-Error is typically a spawn/exec RESULT carrying a full stdout buffer and an env snapshot. + // blockerDetail rides into `receipts`, which a wired gate serializes VERBATIM to a plaintext audit log — + // so a whole-object JSON dump writes caller secrets to disk (the F16/BA-1 leak class). Only conventional + // diagnostic fields are taken, clamped. Mutation-proof: restore `JSON.stringify(err)` and both go red. + const sp = scriptedProvider(() => ({ text: 'work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { + sensor: () => { throw { code: 'ENOENT', stdout: 'x'.repeat(50000), env: { SECRET_TOKEN: 'sk-live-abc123' } }; }, + }, + }); + assert.ok(out.receipts.blockerDetail.length < 400, 'the detail is bounded, never an unbounded buffer dump'); + assert.doesNotMatch(out.receipts.blockerDetail, /sk-live-abc123/, 'caller secrets never reach the audit record'); + assert.match(out.receipts.blockerDetail, /ENOENT/, 'while still naming the actual fault'); + }); + it('an UNREADABLE returned verdict is named as such, not reported as the sensor throwing', async () => { // A sensor returning a Proxy whose accessors throw RETURNED NORMALLY — reporting "sensor threw" sends the // operator hunting a `throw` that does not exist. NB the fault surfaces during the `await`'s own `.then` From 3b879ae799cf2fd8505bb4156902742d6f5b512f Mon Sep 17 00:00:00 2001 From: hamr0 Date: Mon, 20 Jul 2026 12:29:16 +0200 Subject: [PATCH 8/8] =?UTF-8?q?release:=20v0.31.0=20=E2=80=94=20BA-15=20br?= =?UTF-8?q?oken-arbiter=20blockers=20(sensor=20+=20verifier=20seams)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A faulty ARBITER is named, never coerced into a model failure. A refineLeaf.sensor or caller opts.evaluate that throws (non-Halt) or returns a malformed verdict stops the run at the FIRST broken close and returns a labeled {incomplete, blocker:'broken-sensor'|'broken-verifier'} with blockerDetail, best preserving the model's unjudged work (BA-5). Three review rounds (the third at xhigh, a clean 40/40) plus a /security pass; all findings reproduced against unfixed code before being touched, all fixes mutation-proven. MINOR: new public surfaces (blockerDetail/blockerTask on RecurseResult, receipts.blockerFrom) and documented behavior changes. Suite 890 pass / 0 fail; typecheck clean by exit code. Co-Authored-By: Claude Opus 4.8 (1M context) --- CHANGELOG.md | 2 +- bareagent.context.md | 2 +- package-lock.json | 4 ++-- package.json | 2 +- 4 files changed, 5 insertions(+), 5 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3337acb..7142be7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog](https://keepachangelog.com/). Versioning: [SemVer](https://semver.org/). -## [Unreleased] +## [0.31.0] - 2026-07-20 ### Fixed diff --git a/bareagent.context.md b/bareagent.context.md index afa798c..93c83da 100644 --- a/bareagent.context.md +++ b/bareagent.context.md @@ -1,7 +1,7 @@ # bareagent — Integration Guide > For AI assistants and developers wiring bareagent into a project. -> v0.30.0 | Node.js >= 18 | zero required deps (`bareguard >=0.9.0 <0.13.0` optional peer for governance) | Apache 2.0 +> v0.31.0 | Node.js >= 18 | zero required deps (`bareguard >=0.9.0 <0.13.0` optional peer for governance) | Apache 2.0 > > Full human guide with composition examples, design philosophy, and recipes: [Usage Guide](docs/02-features/usage-guide.md) diff --git a/package-lock.json b/package-lock.json index 4d62833..bd18af4 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "bare-agent", - "version": "0.29.0", + "version": "0.31.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "bare-agent", - "version": "0.29.0", + "version": "0.31.0", "license": "Apache-2.0", "bin": { "bare-agent": "bin/cli.js" diff --git a/package.json b/package.json index 1d0b5b4..373f4b4 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "bare-agent", - "version": "0.30.0", + "version": "0.31.0", "files": [ "index.js", "index.d.ts",