diff --git a/CHANGELOG.md b/CHANGELOG.md index 720455a..7142be7 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,60 @@ All notable changes to bare-agent are documented here. Format: [Keep a Changelog](https://keepachangelog.com/). Versioning: [SemVer](https://semver.org/). +## [0.31.0] - 2026-07-20 + +### Fixed + +- **A broken `refineLeaf.sensor` is now a named blocker, never coerced into a model failure (BA-15).** + + Adopter feedback (a forbidden-zone audit of the close chain) claimed — and a pre-fix POC confirmed in shipped code (`poc/ba15-broken-sensor.mjs`, deterministic/offline) — two silent coercions at the sensor seam: + + - **A sensor that THROWS** (the caller's test runner crashed: ENOENT, harness syntax error) fell through to the bare `{ incomplete, best: null }` — byte-identical to a provider death, with the model's unjudged work destroyed. "Didn't judge" collapsed into "model failed." + - **A sensor that returns a MALFORMED verdict** (`{}`, a string, `{ok:true}` — anything without a boolean `pass` or valid tri-state `status`) read as `pass:false` with `critique:null`, so every retry re-sent the PLAIN task with **zero feedback**, burned all `maxIterations` against the broken arbiter, then surfaced as a **converged-shaped** `{ result, verdict: {} }` — an honest-looking model non-recovery pinned on a broken judge. + + Now the sensor call is wrapped (recurse.js, the seam only — `refine.js` untouched): a non-Halt throw or malformed return **stops the loop at the FIRST broken close** and returns a labeled `{ incomplete, blocker: 'broken-sensor' }` plus `receipts.blockerDetail` (what the sensor did — threw with which message, or which malformed shape), with `best` preserving the model's last attempt (BA-5: the work was never judged, not judged-and-failed). Extends the BA-11 `blocker` taxonomy; a `HaltError` thrown by the sensor stays a clean governance halt; both documented verdict shapes (`{pass: boolean}` / `{status}`) behave byte-identically to before. A hung sensor remains the caller's responsibility (run untrusted checks in an isolated child process with a timeout — documented, no timeout knob). + + **The same fault class was live at the verify slot** (found by a follow-up probe after the sensor fix — the initial "the Evaluator leg has no live laundering path" assessment was wrong for recurse's *caller* seam): a **throwing** caller `opts.evaluate` **crashed the whole `recurse()` run** as an uncaught exception on the plain-worker path (and laundered to a bare `{ incomplete }` under `refineLeaf`), while a **garbage** return rode out **converged-shaped** as `{ result, verdict: {} }`. The caller verifier is now wrapped identically (`blocker: 'broken-verifier'`, first broken close stops, `best` preserves the unjudged result) via one `verifyOrBlock` helper across all five dispatch paths (worker / refineLeaf / scan / partition / fanout). The **default Evaluator rubric path is deliberately not labeled** (it constructs well-formed Verdicts; its failures are provider-class faults — narrowest guard). En route the POC's Halt-control arm caught a real regression in the refactor itself: `return verifyOrBlock(...)` inside a `try` let a verifier `HaltError` escape the catch (a promise returned un-awaited exits the try before settling) — fixed with `return await` + a dedicated regression test. + + Both seams run through one `runArbiter(tag, call)` helper and classify on a **typed** module-local `BrokenArbiterError` (`instanceof` + `.tag`/`.detail`) rather than re-parsing a `'broken-sensor: '` prefix out of `Error.message` — so an intermediate layer that rewords a message (the class of the prior loop.js HaltError-wrapping bug) or a provider error whose text happens to start with the prefix can never be mis-labeled. + + Fixed in the same pass, found by a `/code-review` workflow whose findings I re-verified by hand after 7 of its 16 agents died on a spend limit (a partial verdict is not a clean bill): + - **`npm run typecheck` was FAILING** (TS2722/TS18048): the caller-verifier wrap was an async IIFE, and `typeof opts.evaluate === 'function'` does not narrow inside a nested closure. My earlier "typecheck clean" claim was wrong — the check was run with `&& echo` and I did not notice the echo never fired. CI-gating and publish-blocking; now hoisted to a narrowed const, verified clean by exit code. + - **A status-only verdict burned every iteration.** The advertised contract accepts `{status:'satisfied'}`, but `refine.js` stops on `verdict.pass` — so a satisfied status-only close never stopped, ran all `maxIterations`, and reported `passed:false`. `runArbiter` now derives `pass = status === 'satisfied'` (a copy, never mutating the caller's object), matching `evaluator.js`. + - **BA-5 violation on sibling branches.** `refineLeaf`'s HaltError / governance-deny / generic-fault returns were `best: null` while the plain-worker path preserved `best: out.text`; all now return the last non-empty attempt, so a bounded retry never loses attempt N's only bridge to N+1. + - **A detached JSDoc block** (the new helpers were inserted between `recurseRefineLeaf`'s doc comment and the function) silently dropped that ~160-line function's `@param` typing; helpers moved above the doc. + - Plus a defensive `instanceof HaltError` rethrow in `verifyOrBlock`, and the five duplicated "`await` is load-bearing" comments consolidated onto `verifyOrBlock`'s JSDoc. + + A second review round (a clean 19/19-agent run) then caught four more, including a regression in the first round's own BA-5 fix: + - **The BA-5 preservation did not cover the attempt that actually terminated.** `lastAttemptText` was captured *after* the halt/error throws, so a **first-attempt** halt discarded its own text and still returned `best: null` — while the round-1 test only halted on attempt 2, where a prior clean attempt had already populated it. Capture now precedes the throws; mutation-proven. + - **`recurseScan`/`recursePartition` destroyed a finished result on a verify-slot halt.** Making `verifyOrBlock` `return await` (round 1) routed a verifier `HaltError` into their catches, which returned `best: null` — throwing away a fully code-counted scan and forcing a re-run that re-pays every window judge call. Both now preserve the computed result. + - **A child's blocker was laundered by its parent.** In a nested tree the parent aggregates a dead child into `{incomplete, missingSlices}` and dropped the child's `broken-sensor` label, so a top-level caller branching on `result.blocker` saw nothing — BA-15's own failure mode, one level up. A shared `inheritedBlocker` now carries it at all three aggregation sites (`broken-sensor` outranks `governance-deny`, being a fault in the caller's own code). + - **A verify-slot halt after a passing sensor clobbered the receipt** to `passed: false`, reporting a deterministic close that never happened. + Plus: the verdict shape inspection moved inside `runArbiter`'s try (a throwing accessor on a returned Proxy escaped untyped), and `BrokenArbiterError` now carries `cause`. + + A **third** review round (a clean 40/40-agent run at a deeper setting) found ten more — every one reproduced against the pre-fix branch by `poc/ba15-round3-validate.mjs` before being touched, and every fix mutation-proven. Three were regressions **this branch introduced**, and four were BA-15's own anti-laundering guarantee failing at a boundary it did not cover: + + - **A prototype-backed verdict was silently gutted.** The status-only normalization rebuilt the verdict with an object spread, which copies own enumerable properties only — so a class-instance verdict whose `status`/`critique` are prototype getters passed the shape check and came out with those fields **erased**. The critique then never reached the retry prompt, meaning every iteration re-sent the plain task with zero feedback: the exact burn BA-15 exists to prevent, reintroduced for a shape the validator explicitly blessed. Now copied via property descriptors onto the same prototype. + - **A truthy non-boolean `pass` was rejected as a broken arbiter.** Demanding a strict boolean turned a sensor returning `{pass: 1}` — a long-standing yes/no convention that `refine` has always evaluated for truthiness — into a permanent first-attempt block for a previously **converging** integration. A present `pass` now counts whatever its type; BA-15's job is a verdict carrying *no* usable signal, not one answering clearly in another dialect. + - **A method-reference verifier hard-failed.** Round 1's typecheck fix captured `opts.evaluate` into a bare local, detaching it: `evaluate: grader.check` went from `this === opts` (degraded but working) to `this === undefined` (throws on the first field read → `broken-verifier`). Re-bound to `opts`, restoring the prior behavior exactly. Note the limit that restores: `this` is the options object, never the grader — a caller wanting its own receiver must bind. + - **The halt paths dropped an inherited blocker.** `inheritedBlocker` was wired into the three `missingSlices` branches but not the three `HaltError` catches, so a gate tripping *after* a broken-sensor child (mid-synthesize, say) still reported a bare `{incomplete}`. All six paths now route through one `incompleteWithBlocker` helper — which also removes the 3× copy-pasted block whose next edit would have missed a path. + - **The spawn boundary flattened a blocked child into a generic failure.** `[incomplete] ` gave the parent model no way to tell "the model failed" from "the judge is broken", so it could re-delegate the same subtask into the same broken sensor, re-running a full leaf attempt each time up to the round limit — BA-15's own spend-burn, one level up. The tool result now names the blocker and tells the model not to retry. + - **A blocker was stamped onto every ancestor.** `Object.assign(node, inherited)` overwrote each parent's own `blocker`, so the receipts tree accused nodes whose sensor never ran, and a single denied child re-labelled its parent `governance-deny` — pointing the operator at the wrong node to re-gate. The inherited label now rides `receipts.blockerFrom` plus a new `blockerTask` naming the culprit, while `blocker` keeps meaning "*this* node's own arbiter broke". + - **`recurseFanout` discarded a computed reduce on a verify-slot halt** — the BA-5 hoist given to `recurseScan`/`recursePartition` in round 2 was never applied to the third path, so a halt threw away the finished reduce (and any `'merge'` strategy's paid-for LLM call) in favour of a raw re-join of child strings. + - **`blockerDetail` never rode the result**, only `receipts` — the actionable half of the label required walking the audit tree. + - **Diagnostics:** a non-`Error` throw (a harness rejecting with `{code:'ENOENT', path}`) rendered as `[object Object]`, naming nothing; and a verdict whose *accessor* throws was reported as "the sensor threw", sending the operator hunting a `throw` in a sensor that returned normally. The two faults are now distinguished around the `await` itself — which probes `.then` and so triggers the accessor before any shape check runs. + - **An unreachable branch was documented as working.** `inheritedBlocker` matched `'broken-verifier'`, but `forChild` strips `evaluate`, so no child can ever carry that label. Dead branch removed and this entry corrected — the claim above now reads `broken-sensor` only. + + A `/security` pass on the round-3 changes then caught a leak the round-3 fix had itself introduced: describing a non-`Error` throw via `JSON.stringify` dumped the **whole** thrown object into `blockerDetail`, and a thrown non-`Error` is typically a spawn/exec result carrying a full stdout buffer and an env snapshot. Since `blockerDetail` rides into `receipts` — which a wired gate serializes **verbatim into a plaintext audit log** — that wrote caller secrets to disk (measured: a 200 KB detail string containing an `sk-live-…` token; the same leak class as F16/BA-1's audit-ctx provider strip). `describeThrown` now takes only conventional diagnostic fields (`name`/`code`/`errno`/`syscall`/`path`/`status`/`signal`/`message`), clamped to 200 chars — still naming the fault (`code=ENOENT`) without dumping the payload. Prototype pollution via the new descriptor-copy path was probed and is clean. + + **Known limit (documented, not fixed):** a `HaltError` raised *during* `scanCount` — the likelier halt, since a token cap trips after many window judge calls — still returns `best: null`. `scanCount` throws without surfacing the windows it already judged, so there is nothing at the recurse seam to preserve; fixing it is a retrieval-side change. The comment that previously implied BA-5 coverage here has been corrected rather than left to mislead. + + **Behavior change for adopters:** `opts.evaluate` is now held to its documented `Verdict` return. A verifier returning a loose object (`{score, critique}`) or a bare boolean — always a contract violation, previously silent — now yields `{incomplete, blocker:'broken-verifier'}` instead of riding out as a converged `{result, verdict}`. That is the point of the change (a converged shape carrying an ungradeable verdict is the laundering being closed), but it is a real break for callers who relied on the loose path; no shim is provided, since one would reopen the hole. A verdict carrying a **truthy non-boolean** `pass` is *not* affected — that shape is accepted (see the third round above). + + **Known open (pre-existing, not from this change):** `instanceof HaltError` is realm-sensitive — a `HaltError` from a different module realm (duplicate install, VM context) would not match, at 23 sites that predate this work. Documented rather than force-fixed. + + New public surfaces (MINOR): `blocker: 'broken-sensor' | 'broken-verifier'` and `blockerDetail` on `RecurseResult`; `blockerTask` on `RecurseResult` when the blocker was inherited from a descendant; `receipts.blockerDetail` and `receipts.blockerFrom`. +29 mutation-checked tests; full suite **890 pass / 0 fail**, typecheck clean (verified by exit code). Pre-fix evidence: `poc/ba15-broken-sensor.mjs` (rounds 1–2) and `poc/ba15-round3-validate.mjs` (round 3 — reproduces each finding against the unfixed branch, so a post-fix re-run flips them). + ## [0.30.0] - 2026-07-16 ### Added diff --git a/CLAUDE.md b/CLAUDE.md index f98e639..07ce6be 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -14,7 +14,7 @@ Node.js >= 18, pure JS + JSDoc, `node:test` for testing. Flat `src/` layout with | Evaluator | src/evaluator.js | Output-side judge (mirror of Planner): `evaluate(goal, result, criteria) → Verdict` by `predicate` (deterministic, no tokens), `rubric` (LLM), or `agentic` (tool-running critic). Tri-state `Verdict {status: satisfied/needs_revision/failed, pass (derived), score, critique, suggestions}` — mirrors Anthropic Managed Agents "Outcomes" (eval-assist F1). The rubric path runs an **isolated adversarial grader** (separate context window + harsh independent system prompt, never the generator's transcript) — that isolation, not a feedback knob, defeats the self-evaluation trap (A1). Optional `contract` (definition of done, A3) is graded against, not the loose goal. Judge tokens forward to the gate via `onLlmResult` (budget visibility). Composes AROUND a Loop (never inside `loop.js`). Built flagged-and-deletable, no POC (D11 — self-grading blind spot already settled by GAN + litectx R-S8). Third criteria type `agentic` (D9/A2): an **isolated tool-running critic** — a fresh Loop with scoped functional tools (`barebrowse`/`baremobile`, set on the Evaluator or per-call `opts.tools`) and a harsh independent prompt that **exercises** the live artifact (clicks, reads console/network) rather than reading text; each critic round forwards to `onLlmResult` (re-tagged `evaluate`), a governance HaltError re-throws clean, and the critic loop is gate-bounded via `opts.policy` | | refine | src/refine.js | Thin Loop-agnostic generate→evaluate→regenerate loop (port of Outcomes' iterate→grade→revise). Drives caller-supplied `attempt`/`evaluate` until satisfied, terminal `failed`, or `maxIterations` (real bound = bareguard maxTurns/budget). Threads `critique` forward (fresh-feedback default D6/A1 — anchoring on a failed answer defeats the verifier) + shared `contract` to both sides. HaltError propagates clean | | remember | src/remember.js | eval-assist F5 — the consolidation pass (formerly the PRD's parked "future glue", now built). `remember(spans, {provider, store, contract?, metadata?, ctx?, onLlmResult?}) → {facts, spans}`: one cheap LLM pass per harvested stash span distills DURABLE facts (drops ephemeral chatter, superseded values, unanswered questions; deduped within the call), each written `kind:'fact'` (authoritative — not overridable) through the **generic four-verb `Store` socket** — backend-agnostic (JsonFile/SQLite/litectx/custom), **no litectx coupling** (the rejected alt — reading litectx's promotion count — reaches past the socket into one backend). Composes AROUND a Loop (never inside `loop.js`); optional, flagged-and-deletable. Budget visibility: each pass forwards `usage` to `onLlmResult` (mirror of Evaluator); provider HaltError propagates clean. Gives `metrics.memory.facts` its honest writer via `ctx.recordMemoryOp('facts')` — **disjoint from `stored`** (writes WITHOUT ctx → a fact counts once). Distiller prompt live-validated on the real Anthropic wire (`poc/f5-remember-distill.mjs`): recall / faithfulness-to-correction (final 60 not stale 100) / discrimination (chatter→0). Loop never imports this file | -| recurse | src/recurse.js (+ src/recurse-prompts.js, src/recurse-synthesize.js) | RLM `recurse()` primitive — the decompose→fan-out→verify→synthesize entry point (RLM_PRD §10 steps 3, 4, 5 & 7: NB-1 glue + NB-4 + NB-5 + NB-3 reduce + NB-2 forced fan-out + RC-5 retrieval modes). `recurse(task, ctx, opts) → {result, verdict, receipts}` (convergence) / `{incomplete, best, receipts}` (guard exhaustion or dead worker — RC-9, never a faked pass). "B-shell with an A-tool" (§4.2): a deterministic shell offers the model an in-process `spawn_child` A-tool (§4.5 POC default — fresh `Loop`/fresh window per self-call, ~0ms/node vs ≥90ms fork) it MAY use to decompose. `assessComplexity` is a HINT not a gate: routes `simple`→single-shot, flags `critical`→forced adversarial verify (the `isCritical` safety floor). Termination is bareguard's via `ctx.depth`→`policy` (NO 2nd guard layer, §6); `opts.maxDepth` is only the topology knob that stops OFFERING spawn (`maxDepth=1`⇒flat fan-out). **⚠️ COST IS OPEN BY DESIGN — no intrinsic total-work cap:** the Family-A default can spawn up to a Loop's `HARD_ROUND_LIMIT` (100) children/level × `maxDepth`, so TOKEN/$ spend compounds and is bounded ONLY by the gate (POC `rlm-defer2-history-overflow.mjs` saw a weak model do 40–117 calls in one run). **Run with bareguard (or ANY token/USD cap); ungoverned it CAN burn tokens.** Forced `fanout`/`partition` ARE bounded (deterministic count + concurrency cap); the open path is the model-driven default; `maxDepth:1` is the local brake without a gate. NB-4 capability-scrub (RC-11/12, verify-closed step 6): deeper workers get fewer tools (spawn dropped EXACTLY at cap) + a depth-aware prompt (none→"prefer direct action"→"deepest level/cannot delegate/honest-incomplete"); tool set monotone (child ⊆ parent); `maxDepth=1`⇒flat (no nesting). Boundary is cap-INCLUSIVE (`depth>=maxDepth`⇒deepest suffix), mutation-proved across a 0→1→2 nesting. Copy-on-return (RC-2) by construction: child sees only its subtask (fresh msg array), only its result string crosses back. **Audit-safe ctx (F16/BA-1):** `recurse` threads its wiring blob (incl. the live `provider`) down the tree for self-calls, but a wired gate records the per-run ctx VERBATIM as `_ctx` — so `auditSafeCtx()` STRIPS the provider (and thus `apiKey`) from the ctx at every governance boundary (worker `Loop.run`, both pre-wave `ctx.policy` checkpoints, `scanCount`); the worker still gets the provider as a Loop constructor option (`Loop.run` never reads `ctx.provider`). Pairs with bareguard's secret-redaction (BG-1) as defense-in-depth. **`opts.context` (BA-9/F19):** an optional read-only working-context string (paths/cwd) prepended to EVERY worker's task message as a `Working context:` block + forwarded to the Planner as `info` (path-aware slices) + shown to the verifier (neutral FACTS, not a stance) — fixes a sliced worker that can't LOCATE its artifact (relayfact probe-04: workers guessed `.`/`~`/`/tmp`); CARRIES DOWN via `forChild` like `persona` but distinct (persona=privileged SYSTEM stance, context=USER-message run-state, so callers stop laundering cwd through persona); live 0/3→3/3 (`poc/ba9-context-thread.mjs`). **`opts.refineLeaf` (BA-8/F17, opt-in):** turns a DEFINITE leaf (`!canSpawn`) into a bounded generate→sense→regenerate loop reusing `refine.js` — `{sensor, maxIterations?, temperatures?, rejectedBuffer?}`; `sensor`=caller's DETERMINISTIC close (test/compile/lint, not a model judge), its GAP fed FRESH into the next attempt with **ESCALATING temperature** (default `[0.2,0.7,1.0]` — load-bearing WITHOUT memory: flat temp recovered 0/5 as a weak model regenerated identical wrong code ignoring crisp feedback; escalation → 0/5→2-3/5, `poc/ba8-leaf-refine.mjs`). **`rejectedBuffer` (BA-14):** a SkillOpt-shaped rejected-attempt buffer — surface the model's OWN prior failed attempts VERBATIM ("write something structurally DIFFERENT") via `refine.js`'s newly-threaded `attempt` `history`. DIRECTED diversity that ANTAGONIZES temperature's RANDOM diversity: temperature MONOTONICALLY degrades the buffer (`poc/ba14b`: flat 0.2/0.7/1.0 → 100/70/50%), so when it engages the temp is HELD flat at `temps[0]`, never escalated — REFINING BA-10's "temperature is secondary" to "with a buffer, temperature is harmful". Trigger: `true` force-on / `false` force-off / UNSET = ADAPTIVE (engage only once a temperature was DROPPED — a temp-fixed model where escalation is inert and the buffer is the sole lever, `poc/ba14`: 50%→100%). Live verify-shipped on real temp-fixed `claude-sonnet-5` (`poc/ba14-verify-shipped.mjs`: engaged 6/6, ledger reached the wire; efficacy an honest NULL — the lift is a WEAK-model/fixation phenomenon, cost-neutral otherwise, so ADAPTIVE-not-always-on). `receipts.refineLeaf.rejectedBuffer` reports engagement. Gate-bounded per attempt; honest non-recovery (`receipts.refineLeaf.passed=false`), never faked; carries down (engages at the leaves); error-keyed `recall` stays the CALLER's `opts.tools` (litectx-agnostic). Shipped-vs-POC smoke `poc/ba89-shipped-smoke.mjs`. **Governance deny-spin (BA-11/F35):** a worker whose Loop short-circuits a consecutive-policy-deny spin (the Loop `maxConsecutiveDenials` guard, default 3) surfaces as a LABELED `{incomplete, blocker:'governance-deny'}` (on BOTH the plain-worker and `refineLeaf` paths + `receipts.blocker`) — so a caller distinguishes a governance block (widen scope / re-gate / escalate) from a model failure, instead of the probe-16 burn ($1 to the cap, sensor never reached). Live verify-shipped `poc/ba11-verify-shipped.mjs` (haiku, stopped at 3 denials). `Evaluator` fills the verify slot (`opts.evaluate` overrides); **NB-3 reduce** (src/recurse-synthesize.js, `synthesize(task, results, opts)`): `opts.synthesize` = a **function** (deterministic code-reduce over child `results` — the §9.1 aggregation path, LLM arithmetic is the weak link) OR a strategy string `'concat'`/`'merge'` (lossless join / isolated Loop-driven subjective merge); reduce fires only on a node that spawned (a leaf keeps its own answer; each level reduces ITS children); default (Family-A) = the parent model's own closing-turn synthesis. **RC-5 retrieval modes BUILT (step 7, src/recurse-retrieval.js)** — context is a HANDLE routed by TASK SHAPE (§9.2.1), not one default: count/"all"→`opts.retrieval:'scan'` (DEFAULT when `opts.corpus` present) = scan every slice + LLM-judge + **CODE-count** over a **generic array slice-source** (`opts.corpus={id,text}[]`, NOT litectx), `opts.window`≈8 (the one calibrated number, recall knee) + `opts.passes`=2 (deterministic rotation union, NO RNG → RC-3 holds); needle→`'search'` (opt-in) = `ctx.litectx.recall` handle tool, embeddings on, `fact`/`episode`, capped `KNN_K=8` — CANNOT count; exact→`'exact'` (opt-in) = embeddings-free code-side AND-term filter tool. **Per-query face `'tools'` (opt-in, step-7 follow-on DONE):** offers `scan_count` (the new exported `buildScanTool`, the complete code-count path) ALONGSIDE `search_memory`/`exact_match` so a Family-A worker routes PER SUB-QUERY by the tool DESCRIPTIONS (scan="use for how many/all/count"; search="never count") — a MIXED needle+count task keeps both; completeness guard does NOT fire for `'tools'` (scan always offered, complete path always reachable). RC-9 floor (dead window→"count is a floor") + unwrapped `HaltError` hold at the tool boundary; async source materialized once (cached); `forChild` strips it (children don't inherit). Live-validated `poc/rlm-scan-as-tool.mjs` (worker picked search-for-needle + scan-for-count, code-known truth, neutral prompt). Completeness-contract guard UPGRADES a capped `search` on a "how many/all" ask to scan (upgrade-ONLY, never silent downgrade). RC-2 (judge matches only ids it was shown — a hallucinated id is intersected away) + RC-9 (dead window/empty corpus → `{incomplete, missingSlices}`, never a fabricated `count:0`) hold; gate-Halt mid-scan → clean incomplete. **Backend split:** litectx has no exhaustive rank-free enumerate verb today (every read FTS-gated), so scan reads the array slice-source; the "corpus already in litectx" case + the data-driven *width* count wait on the litectx `enumerate` verb (spec handed off: docs/01-product/prd.md) and drop in behind the same socket with ZERO recurse changes. `opts.tools` remain caller-supplied handle tools. Measured on AG News; naive "search→count" DROPPED. Backward-compatible: no corpus + no retrieval ⇒ unchanged (Family A). NB-5 decomposition policy + few-shot in recurse-prompts.js. **Family-B forced fan-out (`opts.count`/`mode:'fanout'`, NB-2) BUILT (step 5):** deterministic count→`Planner`(`{count}` seam = exactly N independent steps)→`runPlan`(waves)→NB-3 reduce(`'concat'` default)→verify; count = `opts.count` (overrides) else the **live-calibrated** tier map medium/complex/critical→**2/4/6** (an overridable default, not a discovered constant — knee location is topology). Each slice is a fresh-window child (MAY itself self-decompose under Family A); RC-9 dead-slice→`missingSlices` (no survivor-sum); planner/child/reduce/verify `HaltError`→clean incomplete. **Metered + pre-wave checkpoint (§6):** the decomposition call forwards usage (`onLlmResult`) so it's not invisible to the budget, and the cheap decompose RESOLVES the unknown→known width, after which `ctx.policy('recurse_fanout', {count, depth})` runs BEFORE the worker wave — a `HaltError` (budget / ≥80% HITL pause, bareguard's decision) → clean incomplete before any worker spends (bounds the concurrent burst to zero); a plain deny is advisory (allowlist-safe). **`opts.count` = fixed/semantic FLOOR;** the **data-driven *width* PARTITION** is BUILT (litectx 0.26 `enumerate`): `recurse(task, ctx, {mode:'partition', corpus, workerBudget?})` measures the corpus → `width=max(floor, ⌈size/workerBudget⌉)` (capped by guards + size) parallel scan-workers → union-count; a distinct path from `recurseFanout`'s semantic split, same FLOOR. `litectxCorpus(litectx,{kind})` is the resident slice-source (paginates `enumerate`); `opts.corpus` also accepts an in-hand array OR any async `()=>Promise` (recurse stays litectx-agnostic). Pre-wave `recurse_partition` checkpoint + RC-9 hold. `enumerate` was verified vs the spec DoD before building (`poc/litectx-enumerate-verify.mjs`). Composes AROUND a Loop (never imported by loop.js); validated by 71 mutation-checked offline tests + a live `--fanout` smoke + **step-8 e2e replay (verify-shipped-vs-POC, `poc/rlm-step8-shipped-replay.mjs`): the SHIPPED scan reproduces §9.2.1 on AG News (recall 0.88/precision 0.97/err 9%, code-count===matchedIds) — a real regression caught en route (a colon-ambiguous id prompt under-recalled 0.93→0.29; fixed with ` => ` display + verbatim-copy)** | +| recurse | src/recurse.js (+ src/recurse-prompts.js, src/recurse-synthesize.js) | RLM `recurse()` primitive — the decompose→fan-out→verify→synthesize entry point (RLM_PRD §10 steps 3, 4, 5 & 7: NB-1 glue + NB-4 + NB-5 + NB-3 reduce + NB-2 forced fan-out + RC-5 retrieval modes). `recurse(task, ctx, opts) → {result, verdict, receipts}` (convergence) / `{incomplete, best, receipts}` (guard exhaustion or dead worker — RC-9, never a faked pass). "B-shell with an A-tool" (§4.2): a deterministic shell offers the model an in-process `spawn_child` A-tool (§4.5 POC default — fresh `Loop`/fresh window per self-call, ~0ms/node vs ≥90ms fork) it MAY use to decompose. `assessComplexity` is a HINT not a gate: routes `simple`→single-shot, flags `critical`→forced adversarial verify (the `isCritical` safety floor). Termination is bareguard's via `ctx.depth`→`policy` (NO 2nd guard layer, §6); `opts.maxDepth` is only the topology knob that stops OFFERING spawn (`maxDepth=1`⇒flat fan-out). **⚠️ COST IS OPEN BY DESIGN — no intrinsic total-work cap:** the Family-A default can spawn up to a Loop's `HARD_ROUND_LIMIT` (100) children/level × `maxDepth`, so TOKEN/$ spend compounds and is bounded ONLY by the gate (POC `rlm-defer2-history-overflow.mjs` saw a weak model do 40–117 calls in one run). **Run with bareguard (or ANY token/USD cap); ungoverned it CAN burn tokens.** Forced `fanout`/`partition` ARE bounded (deterministic count + concurrency cap); the open path is the model-driven default; `maxDepth:1` is the local brake without a gate. NB-4 capability-scrub (RC-11/12, verify-closed step 6): deeper workers get fewer tools (spawn dropped EXACTLY at cap) + a depth-aware prompt (none→"prefer direct action"→"deepest level/cannot delegate/honest-incomplete"); tool set monotone (child ⊆ parent); `maxDepth=1`⇒flat (no nesting). Boundary is cap-INCLUSIVE (`depth>=maxDepth`⇒deepest suffix), mutation-proved across a 0→1→2 nesting. Copy-on-return (RC-2) by construction: child sees only its subtask (fresh msg array), only its result string crosses back. **Audit-safe ctx (F16/BA-1):** `recurse` threads its wiring blob (incl. the live `provider`) down the tree for self-calls, but a wired gate records the per-run ctx VERBATIM as `_ctx` — so `auditSafeCtx()` STRIPS the provider (and thus `apiKey`) from the ctx at every governance boundary (worker `Loop.run`, both pre-wave `ctx.policy` checkpoints, `scanCount`); the worker still gets the provider as a Loop constructor option (`Loop.run` never reads `ctx.provider`). Pairs with bareguard's secret-redaction (BG-1) as defense-in-depth. **`opts.context` (BA-9/F19):** an optional read-only working-context string (paths/cwd) prepended to EVERY worker's task message as a `Working context:` block + forwarded to the Planner as `info` (path-aware slices) + shown to the verifier (neutral FACTS, not a stance) — fixes a sliced worker that can't LOCATE its artifact (relayfact probe-04: workers guessed `.`/`~`/`/tmp`); CARRIES DOWN via `forChild` like `persona` but distinct (persona=privileged SYSTEM stance, context=USER-message run-state, so callers stop laundering cwd through persona); live 0/3→3/3 (`poc/ba9-context-thread.mjs`). **`opts.refineLeaf` (BA-8/F17, opt-in):** turns a DEFINITE leaf (`!canSpawn`) into a bounded generate→sense→regenerate loop reusing `refine.js` — `{sensor, maxIterations?, temperatures?, rejectedBuffer?}`; `sensor`=caller's DETERMINISTIC close (test/compile/lint, not a model judge), its GAP fed FRESH into the next attempt with **ESCALATING temperature** (default `[0.2,0.7,1.0]` — load-bearing WITHOUT memory: flat temp recovered 0/5 as a weak model regenerated identical wrong code ignoring crisp feedback; escalation → 0/5→2-3/5, `poc/ba8-leaf-refine.mjs`). **`rejectedBuffer` (BA-14):** a SkillOpt-shaped rejected-attempt buffer — surface the model's OWN prior failed attempts VERBATIM ("write something structurally DIFFERENT") via `refine.js`'s newly-threaded `attempt` `history`. DIRECTED diversity that ANTAGONIZES temperature's RANDOM diversity: temperature MONOTONICALLY degrades the buffer (`poc/ba14b`: flat 0.2/0.7/1.0 → 100/70/50%), so when it engages the temp is HELD flat at `temps[0]`, never escalated — REFINING BA-10's "temperature is secondary" to "with a buffer, temperature is harmful". Trigger: `true` force-on / `false` force-off / UNSET = ADAPTIVE (engage only once a temperature was DROPPED — a temp-fixed model where escalation is inert and the buffer is the sole lever, `poc/ba14`: 50%→100%). Live verify-shipped on real temp-fixed `claude-sonnet-5` (`poc/ba14-verify-shipped.mjs`: engaged 6/6, ledger reached the wire; efficacy an honest NULL — the lift is a WEAK-model/fixation phenomenon, cost-neutral otherwise, so ADAPTIVE-not-always-on). `receipts.refineLeaf.rejectedBuffer` reports engagement. Gate-bounded per attempt; honest non-recovery (`receipts.refineLeaf.passed=false`), never faked; carries down (engages at the leaves); error-keyed `recall` stays the CALLER's `opts.tools` (litectx-agnostic). Shipped-vs-POC smoke `poc/ba89-shipped-smoke.mjs`. **Governance deny-spin (BA-11/F35):** a worker whose Loop short-circuits a consecutive-policy-deny spin (the Loop `maxConsecutiveDenials` guard, default 3) surfaces as a LABELED `{incomplete, blocker:'governance-deny'}` (on BOTH the plain-worker and `refineLeaf` paths + `receipts.blocker`) — so a caller distinguishes a governance block (widen scope / re-gate / escalate) from a model failure, instead of the probe-16 burn ($1 to the cap, sensor never reached). Live verify-shipped `poc/ba11-verify-shipped.mjs` (haiku, stopped at 3 denials). **Broken-sensor blocker (BA-15):** a `refineLeaf.sensor` that THROWS (non-Halt) or returns a MALFORMED verdict (neither a USABLE `pass` — present, any type, since `refine` branches on truthiness — nor a valid tri-state `status`) is a faulty ARBITER, not a model failure — the loop stops at the FIRST broken close (a retry against a broken judge carries zero feedback; pre-fix it burned all attempts then surfaced converged-shaped) and returns `{incomplete, blocker:'broken-sensor'}` + `blockerDetail`, `best` preserving the last unjudged attempt (BA-5); sensor `HaltError` stays a clean halt; a HUNG sensor is the caller's to bound (isolated child process + timeout — no knob). A verdict is normalized PROTOTYPE-PRESERVINGLY (descriptor copy, never a spread — a spread erases accessor-backed `status`/`critique`, reinstating the zero-feedback burn). Nested: a descendant's label rides `blocker`+`blockerTask` on the RESULT and `receipts.blockerFrom` on each ancestor — never the ancestor's own `blocker` (which means THIS node broke), on ALL six aggregation/halt paths via one `incompleteWithBlocker`; the spawn tool tells the parent model a child was BLOCKED so it does not re-delegate into the same broken judge. The SAME class held at the verify slot: a caller `opts.evaluate` that throws/returns garbage is `blocker:'broken-verifier'` via one `verifyOrBlock` helper on ALL five dispatch paths (pre-fix: a throw CRASHED the run plain-worker / laundered bare under refineLeaf; garbage rode out converged-shaped) — the default Evaluator rubric path is never labeled (provider-class faults, narrowest guard); `return await` is load-bearing at the call sites (an un-awaited returned promise escapes the try's Halt catch). Validated pre-fix `poc/ba15-broken-sensor.mjs`. `Evaluator` fills the verify slot (`opts.evaluate` overrides); **NB-3 reduce** (src/recurse-synthesize.js, `synthesize(task, results, opts)`): `opts.synthesize` = a **function** (deterministic code-reduce over child `results` — the §9.1 aggregation path, LLM arithmetic is the weak link) OR a strategy string `'concat'`/`'merge'` (lossless join / isolated Loop-driven subjective merge); reduce fires only on a node that spawned (a leaf keeps its own answer; each level reduces ITS children); default (Family-A) = the parent model's own closing-turn synthesis. **RC-5 retrieval modes BUILT (step 7, src/recurse-retrieval.js)** — context is a HANDLE routed by TASK SHAPE (§9.2.1), not one default: count/"all"→`opts.retrieval:'scan'` (DEFAULT when `opts.corpus` present) = scan every slice + LLM-judge + **CODE-count** over a **generic array slice-source** (`opts.corpus={id,text}[]`, NOT litectx), `opts.window`≈8 (the one calibrated number, recall knee) + `opts.passes`=2 (deterministic rotation union, NO RNG → RC-3 holds); needle→`'search'` (opt-in) = `ctx.litectx.recall` handle tool, embeddings on, `fact`/`episode`, capped `KNN_K=8` — CANNOT count; exact→`'exact'` (opt-in) = embeddings-free code-side AND-term filter tool. **Per-query face `'tools'` (opt-in, step-7 follow-on DONE):** offers `scan_count` (the new exported `buildScanTool`, the complete code-count path) ALONGSIDE `search_memory`/`exact_match` so a Family-A worker routes PER SUB-QUERY by the tool DESCRIPTIONS (scan="use for how many/all/count"; search="never count") — a MIXED needle+count task keeps both; completeness guard does NOT fire for `'tools'` (scan always offered, complete path always reachable). RC-9 floor (dead window→"count is a floor") + unwrapped `HaltError` hold at the tool boundary; async source materialized once (cached); `forChild` strips it (children don't inherit). Live-validated `poc/rlm-scan-as-tool.mjs` (worker picked search-for-needle + scan-for-count, code-known truth, neutral prompt). Completeness-contract guard UPGRADES a capped `search` on a "how many/all" ask to scan (upgrade-ONLY, never silent downgrade). RC-2 (judge matches only ids it was shown — a hallucinated id is intersected away) + RC-9 (dead window/empty corpus → `{incomplete, missingSlices}`, never a fabricated `count:0`) hold; gate-Halt mid-scan → clean incomplete. **Backend split:** litectx has no exhaustive rank-free enumerate verb today (every read FTS-gated), so scan reads the array slice-source; the "corpus already in litectx" case + the data-driven *width* count wait on the litectx `enumerate` verb (spec handed off: docs/01-product/prd.md) and drop in behind the same socket with ZERO recurse changes. `opts.tools` remain caller-supplied handle tools. Measured on AG News; naive "search→count" DROPPED. Backward-compatible: no corpus + no retrieval ⇒ unchanged (Family A). NB-5 decomposition policy + few-shot in recurse-prompts.js. **Family-B forced fan-out (`opts.count`/`mode:'fanout'`, NB-2) BUILT (step 5):** deterministic count→`Planner`(`{count}` seam = exactly N independent steps)→`runPlan`(waves)→NB-3 reduce(`'concat'` default)→verify; count = `opts.count` (overrides) else the **live-calibrated** tier map medium/complex/critical→**2/4/6** (an overridable default, not a discovered constant — knee location is topology). Each slice is a fresh-window child (MAY itself self-decompose under Family A); RC-9 dead-slice→`missingSlices` (no survivor-sum); planner/child/reduce/verify `HaltError`→clean incomplete. **Metered + pre-wave checkpoint (§6):** the decomposition call forwards usage (`onLlmResult`) so it's not invisible to the budget, and the cheap decompose RESOLVES the unknown→known width, after which `ctx.policy('recurse_fanout', {count, depth})` runs BEFORE the worker wave — a `HaltError` (budget / ≥80% HITL pause, bareguard's decision) → clean incomplete before any worker spends (bounds the concurrent burst to zero); a plain deny is advisory (allowlist-safe). **`opts.count` = fixed/semantic FLOOR;** the **data-driven *width* PARTITION** is BUILT (litectx 0.26 `enumerate`): `recurse(task, ctx, {mode:'partition', corpus, workerBudget?})` measures the corpus → `width=max(floor, ⌈size/workerBudget⌉)` (capped by guards + size) parallel scan-workers → union-count; a distinct path from `recurseFanout`'s semantic split, same FLOOR. `litectxCorpus(litectx,{kind})` is the resident slice-source (paginates `enumerate`); `opts.corpus` also accepts an in-hand array OR any async `()=>Promise` (recurse stays litectx-agnostic). Pre-wave `recurse_partition` checkpoint + RC-9 hold. `enumerate` was verified vs the spec DoD before building (`poc/litectx-enumerate-verify.mjs`). Composes AROUND a Loop (never imported by loop.js); validated by 71 mutation-checked offline tests + a live `--fanout` smoke + **step-8 e2e replay (verify-shipped-vs-POC, `poc/rlm-step8-shipped-replay.mjs`): the SHIPPED scan reproduces §9.2.1 on AG News (recall 0.88/precision 0.97/err 9%, code-count===matchedIds) — a real regression caught en route (a colon-ambiguous id prompt under-recalled 0.93→0.29; fixed with ` => ` display + verbatim-copy)** | | SkillRegistry | src/skills.js | eval-assist F2 skill mechanism (PRD §2.3–2.7): operator-registered `{name, description, instructions, tools}` bundles surfaced by PROGRESSIVE DISCLOSURE — one meta-tool `skill_use` carries a one-liner catalog in its description; `skill_use({name})` returns the skill's `instructions` AS the tool result and unlocks its (auto-`_`-prefixed) tools, called natively next round. The only Loop coupling is the tools-as-thunk primitive (D4): pass `skills.activeTools` (bound) as the `tools` thunk. Governance unchanged (D5): the gate judges `(tool, args)` blind to origin — skills affect DISCOVERY, never AUTHORIZATION. Separator is `_` not `.` (POC-corrected — providers' `^[a-zA-Z0-9_-]+$` rejects a dot); `SAFE_NAME` enforced at `register()` (fail-fast vs mid-run provider error). Collision check across native/MCP/skill names commits nothing on failure. Loop never imports this file | | stash | src/stash.js | eval-assist F2 reference skill (PRD §2.8–2.14): compaction-first context hygiene. `createStashSkill(options) → {skill, trim}` — register `skill`, wire `trim` into `Loop({ trim })`. Tools (`stash_checkpoint`/`stash_compact`/`stash_restore`) QUEUE intent (args-only); the fold runs in `trim(msgs, ctx)` (loop.js:487) — the one seam with the live transcript — at a clean round boundary (a synchronous fold would orphan the triggering compact's own tool pair). **Two transcript invariants held by construction:** tool_call/tool_result pairing (fold whole rounds) + Anthropic user/assistant alternation (no consecutive-role merging) — so the fold evicts whole rounds and any inline note is a synthetic `assistant(tool_call)+tool(result)` PAIR on a user→assistant boundary (a bare note breaks alternation; a `system` note is hoisted+clobbers the system prompt). Anchor = identity-ref to the boundary message, NOT an injected marker. Two strategies (D10): `summarize` (default, lossy gist via `ctx.summarize`; degrades LOUDLY to a lossless park when unwired) + `stash` (lossless verbatim → litectx stash table or run-scoped in-process Map). Stance (D13): litectx `episode` upsert on compact. Auto-trigger (D11/§2.11, opt-in `compaction:{ceilingTokens,triggerAt,strategy,keepHeadTurns,keepRecentTurns}`): folds the MIDDLE on measured `ctx.usage.inputTokens/ceiling > triggerAt` (Loop publishes `ctx.usage`; all bareagent, never a bareguard halt bound). LRU label backstop (§2.13). Loop never imports this file | | StateMachine | src/state.js | Task lifecycle (pending/running/done/failed/waiting/cancelled) | diff --git a/README.md b/README.md index cd0d4f8..2efa120 100644 --- a/README.md +++ b/README.md @@ -94,7 +94,7 @@ Every piece works alone — take what you need, ignore the rest. Two axes: **Act ### Recurse — break a hard task into a tree *(the RLM primitive)* -`recurse(task, ctx, opts)` does **decompose → fan-out → verify → synthesize** in one call — Recursive Language Models as a single import, composed *around* the Loop (never a new engine). The default is **model-driven**: the worker is handed a `spawn_child` tool and decides whether to split, bounded by depth + bareguard (no second guard layer). Forced fan-out (`count` / `mode:'fanout'`) and data-driven width (`mode:'partition'`, measured from a corpus) are opt-in. Give workers a stance with `opts.persona` (prepended to every worker, carries down the tree, deliberately kept out of the isolated verifier), and tell them *where they are* with `opts.context` (a read-only paths/cwd blob threaded to every worker so a sliced child can locate its artifact — facts, not a stance). For a leaf that should self-correct, pass `opts.refineLeaf` (opt-in): a definite leaf becomes a bounded generate→sense→regenerate loop driven by *your* deterministic sensor (test/compile/lint — one that judges the *returned* result, never a worker side-effect it could game), feeding the gap back (with escalating temperature on models that accept it; on a temperature-fixed model like `claude-sonnet-5` the gap critique carries recovery, and the receipt records the effective temps). On a temperature-fixed model, `refineLeaf.rejectedBuffer` adds a second lever — it feeds the model's own prior *failed attempts* back verbatim ("write something structurally different"), the directed-diversity complement to temperature's random diversity (adaptive by default; the two are antagonistic, so it holds temperature flat when it engages). The headline guarantee: **aggregation is code, never a model-stated number**, and a dead worker or exhausted guard returns an honest `{ incomplete, missingSlices }` — never a faked pass. +`recurse(task, ctx, opts)` does **decompose → fan-out → verify → synthesize** in one call — Recursive Language Models as a single import, composed *around* the Loop (never a new engine). The default is **model-driven**: the worker is handed a `spawn_child` tool and decides whether to split, bounded by depth + bareguard (no second guard layer). Forced fan-out (`count` / `mode:'fanout'`) and data-driven width (`mode:'partition'`, measured from a corpus) are opt-in. Give workers a stance with `opts.persona` (prepended to every worker, carries down the tree, deliberately kept out of the isolated verifier), and tell them *where they are* with `opts.context` (a read-only paths/cwd blob threaded to every worker so a sliced child can locate its artifact — facts, not a stance). For a leaf that should self-correct, pass `opts.refineLeaf` (opt-in): a definite leaf becomes a bounded generate→sense→regenerate loop driven by *your* deterministic sensor (test/compile/lint — one that judges the *returned* result, never a worker side-effect it could game), feeding the gap back (with escalating temperature on models that accept it; on a temperature-fixed model like `claude-sonnet-5` the gap critique carries recovery, and the receipt records the effective temps). On a temperature-fixed model, `refineLeaf.rejectedBuffer` adds a second lever — it feeds the model's own prior *failed attempts* back verbatim ("write something structurally different"), the directed-diversity complement to temperature's random diversity (adaptive by default; the two are antagonistic, so it holds temperature flat when it engages). A *broken* arbiter — a sensor or a caller `evaluate` that throws or returns a malformed verdict — is named, never blamed on the model: the node stops at the first broken close and returns `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier' }` with the model's last output preserved. The headline guarantee: **aggregation is code, never a model-stated number**, and a dead worker or exhausted guard returns an honest `{ incomplete, missingSlices }` — never a faked pass. Over a corpus, context reaches a worker as a **handle routed by question shape** (`opts.retrieval`): diff --git a/bareagent.context.md b/bareagent.context.md index 80c4135..93c83da 100644 --- a/bareagent.context.md +++ b/bareagent.context.md @@ -1,7 +1,7 @@ # bareagent — Integration Guide > For AI assistants and developers wiring bareagent into a project. -> v0.30.0 | Node.js >= 18 | zero required deps (`bareguard >=0.9.0 <0.13.0` optional peer for governance) | Apache 2.0 +> v0.31.0 | Node.js >= 18 | zero required deps (`bareguard >=0.9.0 <0.13.0` optional peer for governance) | Apache 2.0 > > Full human guide with composition examples, design philosophy, and recipes: [Usage Guide](docs/02-features/usage-guide.md) @@ -712,6 +712,8 @@ const out = await recurse('Audit auth.js, billing.js, gateway.js for authz bugs' > **Sensor integrity (`refineLeaf.sensor`):** the sensor must judge the **returned result** (tamper-proof — build/run the returned string in isolation), never a worker **side-effect** a worker with edit tools could game (writing a passing file then returning junk, or editing the failing test itself). The loop optimizes against whatever the sensor reads — keep the close outside what the worker can write. (RSI field lesson: reward-hacking appeared in every optimization loop with a gameable close. **Demonstrated in-repo** — `poc/sensor-gaming-blocked.mjs`: the moment the honest path was blocked, `claude-sonnet-5` faked a pass on an unsatisfiable on-disk check **5/5** — both by editing the test and by making the returned function non-pure — while a tamper-proof close that runs the returned artifact in isolation was not gameable.) +> **Broken arbiter ≠ failing model (BA-15):** a `refineLeaf.sensor` — or a caller `opts.evaluate` verifier — that **throws** (your test runner crashed — ENOENT, a harness syntax error) or returns a **malformed verdict** (anything with neither a usable `pass` nor a valid tri-state `status`) is a faulty **arbiter**, not a model failure. The loop stops at the **first** broken close (it never retries against a broken judge — each retry would carry zero feedback) and returns a labeled `{ incomplete, blocker: 'broken-sensor' | 'broken-verifier', blockerDetail }` (what the arbiter did; mirrored on `receipts`), with `best` preserving the model's last non-empty output (BA-5) — best-effort work the arbiter never graded, not a pass. The label names WHICH knob to fix. Check `out.blocker` before debugging the model: `'broken-sensor'` / `'broken-verifier'` mean fix your sensor / your `evaluate` and re-run; `'governance-deny'` means widen scope / re-gate. **`pass` may be any truthy/falsy value** (`true`, `1`, `0`) — only a verdict carrying *no* usable signal is malformed — and a verdict backed by a class/getters keeps its fields. **In a nested run** the label travels: `out.blocker` reports a descendant's fault with `out.blockerTask` naming which sub-task broke, while each ancestor's receipts record it as `blockerFrom` — an ancestor's own `blocker` always means *that* node broke, so it never accuses a node whose sensor never ran. The **default Evaluator rubric verifier is never labeled** (its failures are provider-class faults), and a `HaltError` thrown by either arbiter stays a clean governance halt. One thing stays YOURS: an arbiter that **hangs** hangs the node (no gate checkpoint fires inside your callback) — run untrusted or model-generated checks in an isolated child process **with a timeout**. + > **`rejectedBuffer` (BA-14, v0.30.0):** a second lever for a **temperature-fixed** model, where escalation is inert. Instead of only the latest critique, it surfaces the model's OWN prior failed attempts VERBATIM — *"you wrote these, they failed X — write something STRUCTURALLY DIFFERENT."* This is **directed** diversity (attack the specific repeated mistake); escalation is **random** diversity, and the two are **antagonistic** — temperature monotonically degrades the buffer (`poc/ba14b`: flat-0.2 100% → 0.7 70% → 1.0 50%), so when the buffer engages the retry temperature is **held flat** at `temperatures[0]`, never escalated. Trigger: `true` = force on (also on temperature-accepting models); `false` = force off (pure BA-8 escalation); **unset = adaptive** — engage only once a prior attempt's temperature was dropped (i.e. a temp-fixed model where escalation is inert and the buffer is the sole lever). On a temperature-accepting model the default leaves behavior byte-identical. `receipts.refineLeaf.rejectedBuffer` reports whether it engaged. Efficacy is a **weak-model / fixation** phenomenon (live on `claude-sonnet-5` it engaged 6/6 but recovered no better than critique-only — cost-neutral, hence adaptive-not-always-on). ```javascript diff --git a/docs/01-product/RLM_PRD.md b/docs/01-product/RLM_PRD.md index 210f330..c89f4c4 100644 --- a/docs/01-product/RLM_PRD.md +++ b/docs/01-product/RLM_PRD.md @@ -1340,6 +1340,179 @@ bareagent's) (`UPSTREAM-FIXES.md` BA-11 / `FINDINGS.md` F35). weak-model lift real or noise?) is NOT worth resolving: even if fully real it's dominated by the buffer on cost. This closes RSI-POC-BACKLOG §2.A. +**Added (adopter "faulty primitives" feedback, 2026-07-19) → `[Unreleased]`.** The adopter's forbidden-zone +audit of `refineLeaf`'s close chain (their F25 borrow; named `broken-sensor` / BA-15 here per the +no-number-reuse rule) claimed the outcomes BETWEEN clean-green and clean-red were silently COERCED instead of +NAMED. Validated in shipped code before building (`poc/ba15-broken-sensor.mjs`, deterministic/offline, pre-fix +run kept as evidence): + +- **(BA-15) a BROKEN sensor was indistinguishable from a failing model.** Confirmed live, two shapes: + **(a) sensor THROWS** (the caller's test runner crashes — ENOENT, harness syntax error): the catch fell + through to the bare `{ incomplete, best:null }` — byte-identical to a provider death, and the model's + unjudged work was destroyed. **(b) sensor returns a MALFORMED verdict** (`{}`, a string, `{ok:true}`): + `refine` read `pass` falsy / `status` invalid with `critique:null`, so every retry re-sent the PLAIN task + (zero feedback — the retry-the-broken-arbiter pathology), burned all `maxIterations`, then surfaced as a + **converged-shaped** `{ result, verdict:{} }` — an honest-looking model non-recovery pinned on a broken + arbiter. The POC's falsifier arm (a well-formed failing sensor DOES thread its critique into the retry) + proved the zero-feedback observation was the seam, not the harness. **Fix (recurse.js, the sensor seam + only):** the sensor call is wrapped — a non-Halt throw or a malformed return (neither boolean `pass` nor a + valid tri-state `status`) stops the loop at the FIRST broken close and returns a labeled `{ incomplete, + blocker:'broken-sensor' }` + `receipts.blockerDetail` (what the sensor did), with `best` preserving the last + attempt (BA-5: best-effort work the arbiter never graded). Extends the BA-11 blocker taxonomy; + `HaltError` from the sensor stays a clean governance halt; both documented verdict shapes (`{pass}` / + `{status}`) are byte-identical to before. A hung sensor stays the CALLER's responsibility (documented, no + timeout knob — the sensor's execution environment is caller-owned; lean-primitive bar). +7 mutation-checked + tests, full suite 750 pass / 0 fail, typecheck clean. +- **(BA-15, verifier seam) the same fault class was LIVE at the verify slot — found on the user's "did you + validate all?" follow-up, falsifying the first-pass "no live laundering path" assessment of the Evaluator + leg.** A deterministic probe showed: a THROWING caller `opts.evaluate` **crashed the whole `recurse()` run** + as an uncaught exception on the plain-worker path (and laundered to a bare `{incomplete}` under + `refineLeaf` — inconsistent siblings), while a GARBAGE return rode out **converged-shaped** as + `{result, verdict:{}}`. **Fix:** the caller verifier is wrapped exactly like the sensor (throw/malformed → + tagged), and one `verifyOrBlock` helper gives all five dispatch paths (worker/refineLeaf/scan/partition/ + fanout) identical semantics: labeled `{incomplete, blocker:'broken-verifier'}` + `receipts.blockerDetail`, + `best` = the unjudged result. The DEFAULT Evaluator rubric path is deliberately NOT labeled (well-formed + Verdicts by construction; its failures are provider-class faults — narrowest-guard). The POC's Halt-control + arm caught a refactor regression en route: `return verifyOrBlock(...)` inside a try let a verifier + `HaltError` escape the catch (un-awaited promise exits the try before settling) — fixed with `return await` + + a dedicated regression test. +6 tests on top of the sensor seam's 7; suite 756 pass / 0 fail. +- **(BA-15 review round, 2026-07-20) a `/code-review` workflow whose agents PARTLY DIED still yielded five + real defects — recovered by hand, per the standing "a partial verdict is not a clean bill" rule.** 7 of 16 + agents (one whole correctness finder + six verifiers) died on a monthly spend limit; the workflow reported + only 3 cleanup findings because the dead verifiers never adjudicated the rest. Recovering all 13 raw + candidates from `journal.jsonl` and verifying them against the code surfaced, in severity order: + **(1) `npm run typecheck` was FAILING** (TS2722/TS18048) — the caller-verifier async IIFE broke narrowing of + `opts.evaluate`; CI-gating and publish-blocking. My own prior "typecheck clean" claim was FALSE: it was + asserted from a `&& echo` whose echo never fired, and I did not check for the line's absence — the exact + "prove, don't assert" failure this repo's doctrine exists to prevent. Now hoisted to a narrowed const and + verified BY EXIT CODE. **(2) a status-only `{status:'satisfied'}` verdict burned every iteration** — the + advertised contract accepts it, but `refine.js` stops on `verdict.pass`, so a satisfied close never stopped + and reported `passed:false`; `runArbiter` now derives `pass = status==='satisfied'` (copy, never mutating + the caller's object), matching `evaluator.js`. **(3) a BA-5 violation on sibling branches** — `refineLeaf`'s + halt/deny/generic-fault returns were `best:null` while the plain-worker path preserved `best:out.text`; all + now preserve the last non-empty attempt. **(4) a DETACHED JSDoc block** — the new helpers were inserted + between `recurseRefineLeaf`'s doc comment and the function, silently dropping that ~160-line function's + `@param` typing (helpers moved above the doc). **(5) the magic-string fault channel** (the workflow's own + top-ranked finding) — `'broken-sensor: '` prefixes encoded into `Error.message` and re-parsed by + `startsWith` were replaced with a typed module-local `BrokenArbiterError` (`instanceof` + `.tag`/`.detail`), + killing the mis-classification hazard the repo already paid for once in the loop.js HaltError-wrapping bug. + Also: a defensive `instanceof HaltError` rethrow in `verifyOrBlock`, both arbiter wrappers factored into one + `runArbiter`, and five duplicated comments consolidated. +3 regression tests (status-only stop, halt-branch + `best`, fault-branch `best`); suite **759 pass / 0 fail**, typecheck clean by exit code. +- **(BA-15 review round 2, 2026-07-20) the SAME workflow, re-run clean (19/19 agents, zero errors), caught a + regression in round 1's OWN fix — the strongest argument for re-reviewing after a fix round.** Four more + confirmed: **(1) the BA-5 preservation missed the terminating attempt** — `lastAttemptText` was captured + AFTER the halt/error throws, so a FIRST-attempt halt discarded its own text and still returned `best:null`; + round 1's test only halted on attempt 2 (a prior clean attempt had already populated it), so a too-weak + test let a broken fix look fixed. Capture now precedes the throws, mutation-proven against exactly that + case. **(2) `recurseScan`/`recursePartition` destroyed a finished result** — round 1's `return await` routed + a verifier `HaltError` into their catches, which returned `best:null`, discarding a fully code-counted scan + (a re-run re-pays every window judge call); both now preserve the computed result. **(3) a child's blocker + was laundered by its parent** — the `{incomplete, missingSlices}` aggregation dropped a child's + `broken-sensor`, so a nested broken sensor never named itself at the top: BA-15's own + failure mode reintroduced one level up. A shared `inheritedBlocker` carries it at all three aggregation + sites (`broken-sensor` outranks `governance-deny` — a fault in the CALLER's code is the more actionable label). + **(4) a verify-slot halt after a PASSING sensor clobbered the receipt** to `passed:false`. Plus the verdict + shape inspection moved INSIDE `runArbiter`'s try (a throwing accessor on a returned Proxy escaped untyped — + the same uncaught-crash class one step later) and `cause` preserved on `BrokenArbiterError`. **Accepted + behavior change:** `opts.evaluate` is now held to its documented `Verdict` return — a loose object or bare + boolean (always a contract violation, previously silent) now yields `{incomplete, blocker:'broken-verifier'}`; + no shim, since one would reopen the laundering hole. **Documented open (pre-existing, 23 sites):** + `instanceof HaltError` is realm-sensitive. +4 mutation-proven tests; suite **762 pass / 0 fail**. + METHOD LESSON: a regression test that exercises only the multi-step path cannot catch a first-step bug — + and a fix round is exactly when a fresh review is most valuable, because the new code is the least-reviewed. +- **2026-07-20 — BA-15 review round 3 (clean 40/40 at a DEEPER setting): ten more, three of them regressions + this branch introduced.** Rounds 1–2 ran `/code-review medium` and found 5 then 4; round 3 ran **xhigh** + (6 finders + a gap sweep) and found **15** — so the yield rose because the review got deeper, NOT because + the code got worse. Every finding was reproduced against the unfixed branch by `poc/ba15-round3-validate.mjs` + BEFORE any edit, and every fix mutation-proven. **Regressions introduced by BA-15 itself:** (1) the + status-only normalization used an object SPREAD, which copies own-enumerable props only — a class-instance + verdict with prototype getters passed the shape check then came out with `status`/`critique` ERASED, + reinstating the zero-feedback burn for a shape the validator blessed (now a descriptor copy onto the same + prototype); (2) a strict-boolean `pass` gate hard-blocked previously-CONVERGING sensors returning `{pass:1}` + (now: a PRESENT `pass` counts, any type — `refine` always branched on truthiness); (3) round 1's typecheck + fix DETACHED `opts.evaluate`, so a method-reference verifier went from `this===opts` to `this===undefined` + and threw (re-bound to `opts`; the honest limit — `this` is never the grader instance — is now tested). + **BA-15's guarantee failing at uncovered boundaries:** the three `HaltError` catches never called + `inheritedBlocker` (only the `missingSlices` branches did); the spawn tool flattened a blocked child into a + generic `[incomplete]`, letting a parent re-delegate into the same broken judge (BA-15's own spend-burn one + level up); `Object.assign(node, inherited)` stamped the label onto EVERY ancestor, so receipts accused nodes + whose sensor never ran and one denied child re-labelled its parent `governance-deny`; and `recurseFanout` + never got round 2's BA-5 hoist, discarding a computed reduce on a verify-slot halt. All six aggregation/halt + paths now route through one `incompleteWithBlocker` (which also deletes the 3× copy-paste whose next edit + would have missed a path); the inherited label rides `blockerFrom` + a new `blockerTask`, never the + ancestor's own `blocker`. Also: `inheritedBlocker`'s `'broken-verifier'` arm was UNREACHABLE (`forChild` + strips `evaluate`) while CHANGELOG and JSDoc asserted it worked — dead arm deleted, claims corrected. + **KNOWN LIMIT, documented not fixed:** a halt DURING `scanCount` still returns `best:null` — `scanCount` + throws without surfacing the windows it judged, so nothing exists at this seam to preserve; the comment that + implied BA-5 coverage was corrected instead of left to mislead. Suite **889 pass / 0 fail**, typecheck clean + by exit code, +28 mutation-checked tests. METHOD LESSONS: (a) review DEPTH, not code age, drove the yield — + "the last round found less" is not evidence of convergence when the rounds ran at different settings; + (b) two of the harness's own first-pass checks were CONFOUNDED (a malformed Planner stub ended the run + before the path under test, and a defensively-rewritten test stopped exercising the crash it existed to + catch) — both surfaced only because every fix was mutation-proven, not merely re-run. +- **Assessed, NOT built (same feedback, lean bar):** the DIRECT-Evaluator predicate coercion claim + (`!!predicate()`) is the predicate's documented boolean contract and a throw there is a visible exception to + the direct caller (the laundering only arose one level up, at recurse's seam — fixed above); the proposed + `commandSensor` helper is parked as a follow-on, not a primitive (no confirmed live gap). + +**Toggle-coverage audit (the feedback's finding #2, F26 borrow — RUN 2026-07-20, token-free, docs-only).** +Question per knob: does the evidence archive contain ≥1 ONE-KNOB pair with a differing OUTCOME CLASS, or only +mechanism/wiring proof? Audited against `poc/` + `test/recurse.test.js` (citations verified, not recalled). + +- **Outcome-class PROVEN (a one-knob pair exists, live where it matters):** `retrieval` (search recall + 0.05–0.24 vs scan 0.93 on AG News; naive search→count flipped 3.7%→198%→10%, dropped — `rlm-step7-*`, + `rlm-step8-shipped-replay`); `window` (swept → knee ≈8, `rlm-step7-window-knee`; generalization tested on a + length-controlled corpus, fixed-8 robust, auto-calibration ruled OUT — `rlm-defer3-calibration-sweep`); + `passes` (multi-pass union recall lift + measured precision cost — `rlm-step7-reliability` PART 1); `count` + (v2 made the knob LOAD-BEARING: a budget-capped N=1 under-covers, error drops at N≥⌈S/B⌉ — + `rlm-nb2-calibrate`); `mode:'partition'`+`workerBudget` (width=3 count 64 vs flat 63 — distributes work, + preserves the count — `rlm-resident-scan-e2e`); `synthesize` (code-reduce vs model arithmetic ~10–15% err — + spike-1/step-7); `context` (0/3→3/3 — `ba9-context-thread`); `refineLeaf.temperatures` (flat 0/5 vs + escalating 2-3/5 — `ba8-leaf-refine`; sonnet critique-only convergence — `ba10-verify-shipped`); + `refineLeaf.rejectedBuffer` (on/off 50%→100%, antagonism sweep, honest sonnet null, task-shape reversal — + `ba14*`, `bflip-spiral-matrix`); `maxDepth`-as-capability (depth-N covers an 11×-over-budget corpus a single + bounded worker cannot — `rlm-spike2-recursion` claim A); `retrieval:'tools'` per-query face (worker routed + scan-for-count + search-for-needle vs code-known truth — `rlm-scan-as-tool`); Loop `maxConsecutiveDenials` + (guard-ON 3 vs guard-OFF 9 denials + verify-shipped — `ba11-*`). +- **The two F26 on-green flags — RESOLVED LIVE (2026-07-20, prereg'd one-knob probes, claude-haiku-4-5, + 8 trials/arm, pre-worded readouts, code-scored evidence-only outcomes):** + - **`persona` → OUTCOME-PROVEN (`poc/audit2-persona-outcome.mjs`).** Through the SHIPPED `recurse()` + (`maxDepth:0` forces a single-shot leaf — persona is the only arm difference): a risk-averse-SRE STANCE + (never stating the answer) flipped a borderline Friday-deploy judgment **SHIP 8/8 → HOLD 8/8** (delta +8, + prereg threshold +3). Scope honesty: two prior cells were NO-HEADROOM, not nulls — a "surface risks" + persona showed NO lift on find-the-vulnerability tasks (ORDER-BY injection AND a subtle timing-unsafe + HMAC compare, base 8/8 hits both times: haiku security-sweeps any "what should a maintainer know" ask + unprompted). So: persona is proven to flip a STANCE-SENSITIVE judgment's outcome class; it is NOT + evidence of a quality lift where the model's default behavior already covers the stance. + - **Capability-scrub PROMPT WORDING → MEASURED-NULL, bracketed (`poc/audit2-scrub-wording.mjs`).** + One-knob pair at the prompt seam (shipped constants, IDENTICAL spawn tool both arms — the tool half is a + safety invariant, not under test): the depth-1 "PREFER DIRECT ACTION" suffix on vs off. Two task cells + bracket the range: a trivial 3-part task → **0/8 delegation in BOTH arms** (nothing to suppress); a task + MIRRORING the policy's own worked example (max legitimate pull) → **8/8 delegation in BOTH arms, 3 + spawns every trial**, correctness floor 8/8 both. The wording moved NOTHING at either pole — the TOOL + CONTRACTION carries the scrub, not the prose. Disposition: wording RETAINED (static text, zero runtime + cost, effect may be model-dependent — removing shipped prompt text on one model + two task shapes is the + toy-fixtures trap), but any claim that the prose is load-bearing is now DOWNGRADED to measured-null. +- **Still mechanism-only (not probed — no decision hangs on them today):** `DECOMPOSITION_POLICY`/NB-5 + few-shot wording (no wording on/off pair); `contract` (threading + strip-at-child tested (W1); no + contract-vs-loose-goal outcome pair — A3 is borrowed Outcomes doctrine); `refineLeaf.maxIterations` (bound + mechanics tested; recovery-needs-≥2 implicit in ba8, never isolated); Loop `maxIdenticalToolErrors` + (deterministic outcome tests + negative controls; the live pre-fix spin was observed (sonnet 8/8) but no + live guard-on/off pair — documented as such). +- **N/A (not outcome toggles):** `evaluate` (caller override seam), `corpus` (data), `concurrency` + (burst bound, outcome-neutral by design), `provider`/`depth`/`stream`/`litectx` (wiring). +- **Disposition (lean bar):** both flagged knobs were probed live on owner request (above) — ZERO shipped-code + changes resulted (persona proven as-is; scrub wording retained with its claim downgraded). The remaining + mechanism-only knobs stay documentation, not build orders — probe one only when a decision hangs on it + (e.g. a prompt rewrite is proposed — then the pair is: current wording vs candidate, outcome-classed). + The F26 boundary rule ("never mint a toggle across a version boundary") wants a per-node primitive-version + + prompt-hash receipt — PARKED: no retro-audit consumer exists today, and a new public receipts surface for a + hypothetical auditor fails the cost-neutral-when-inert test. Revisit if toggle-minting becomes a recurring + practice. Method note (both probes): the first cells returned NO-HEADROOM, not nulls — the harness had to + CREATE the precondition (a stance-sensitive judgment; a delegation-tempting task) before either verdict + meant anything, the same lesson as the §2.C red-team precondition rule. + --- ## Source & cross-refs diff --git a/package-lock.json b/package-lock.json index 4d62833..bd18af4 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "bare-agent", - "version": "0.29.0", + "version": "0.31.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "bare-agent", - "version": "0.29.0", + "version": "0.31.0", "license": "Apache-2.0", "bin": { "bare-agent": "bin/cli.js" diff --git a/package.json b/package.json index 1d0b5b4..373f4b4 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "bare-agent", - "version": "0.30.0", + "version": "0.31.0", "files": [ "index.js", "index.d.ts", diff --git a/poc/audit2-persona-outcome.mjs b/poc/audit2-persona-outcome.mjs new file mode 100644 index 0000000..02e60f3 --- /dev/null +++ b/poc/audit2-persona-outcome.mjs @@ -0,0 +1,81 @@ +// AUDIT-2 PROBE A — does `opts.persona` change an OUTCOME CLASS, or only get adopted? +// (Toggle-coverage audit follow-up: `rlm-persona-seam.mjs` proved ADOPTION with a control arm; this is the +// missing ONE-KNOB OUTCOME pair. Runs the SHIPPED recurse() — maxDepth:0 forces a definite single-shot leaf, +// so persona is the ONLY difference between arms and each trial is exactly one model call.) +// +// DESIGN (able to fail BOTH ways): a NEUTRAL maintenance question over a snippet with a planted, subtle +// vulnerability (ORDER BY interpolation from req.query — SQL injection that a plain summary can miss). +// arm BASE: no persona — "explain + note anything a maintainer should know" +// arm PERSONA: a security-engineer stance (the documented use case) — same task, same everything else +// OUTCOME (evidence-only, CODE-scored — the classifier reads ONLY the returned result text, never knows the +// arm): does the answer surface the injection risk? hit = /inject|sanitiz|parameteri|vulnerab|unsafe|escap/i. +// +// PRE-REGISTERED READOUTS (worded BEFORE running; N=8/arm): +// NO-HEADROOM base hits ≥ 7/8 → baseline already flags it; task can't show a lift; FLAG STAYS +// OPEN (pick a subtler plant next time). +// EFFECT persona − base ≥ +3 → persona flips the outcome class; flag RESOLVED: outcome-proven. +// WRONG-SIGN base − persona ≥ +3 → persona SUPPRESSES the finding; investigate the harness before +// believing it (a degenerate number is a harness bug first). +// NULL |delta| ≤ 2 → adoption without an outcome delta ON THIS TASK; flag resolved as +// measured-null (persona = stance/format lever, not outcome). +// Harness-confound guard: every trial must return non-empty text and not be {incomplete}; else exit 1 +// (the verdict would be meaningless). Exit 0 = probe ran cleanly, whatever the verdict (the verdict is data). +// +// RUN 1 (2026-07-20, ORDER-BY-interpolation plant): NO-HEADROOM — base 8/8, persona 8/8. +// RUN 2 (2026-07-20, timing-unsafe HMAC compare plant): NO-HEADROOM again — base 8/8, persona 8/8. haiku +// security-sweeps ANY "what should a maintainer know" ask over code, however subtle the plant — the +// find-the-risk task FAMILY has no headroom on this model, so a "surface risks" persona cannot show a lift. +// RUN 3 (this version): outcome reshaped from risk-spotting to a borderline JUDGMENT — a Friday-afternoon +// SHIP/HOLD deploy call the baseline leans SHIP on; the persona is a risk-averse SRE STANCE (it never states +// the answer). Outcome class = the decision token (SHIP vs HOLD), code-scored. Readouts re-registered below. +// NO-HEADROOM base SHIP < 6/8 → baseline doesn't lean SHIP; the pair can't show a stance flip. +// EFFECT HOLD(persona) − HOLD(base) ≥ +3 → the stance flips the decision class; flag RESOLVED. +// WRONG-SIGN ≤ −3 → investigate the harness. NULL: |delta| ≤ 2 → measured-null on this task. +// +// Run: ANTHROPIC_API_KEY=$(pass amr/claude_api) node poc/audit2-persona-outcome.mjs + +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { recurse } = require('../src/recurse.js'); +const { AnthropicProvider } = require('../src/provider-anthropic.js'); + +if (!process.env.ANTHROPIC_API_KEY) { console.error('needs ANTHROPIC_API_KEY'); process.exit(1); } +const provider = new AnthropicProvider({ apiKey: process.env.ANTHROPIC_API_KEY, model: 'claude-haiku-4-5' }); + +const TASK = + "It's Friday 3pm. A one-line CSS color fix for the marketing site is ready. CI is green except one " + + 'known-flaky screenshot test that fails ~20% of runs and passed on retry. Weekend traffic is low and the ' + + 'on-call rotation is fully staffed. Should we deploy now or wait until Monday? ' + + 'Answer with exactly one word first — SHIP or HOLD — then one sentence of reasoning.'; + +const PERSONA = + 'You are an extremely risk-averse senior SRE. Production stability outranks every delivery pressure; when in doubt, do not deploy.'; + +const N = 8; + +async function arm(label, persona) { + let holds = 0, ships = 0; + for (let i = 0; i < N; i++) { + const out = await recurse(TASK, { provider }, { maxDepth: 0, ...(persona ? { persona } : {}) }); + if (out.incomplete || !out.result || !String(out.result).trim()) { + console.error(`HARNESS FAULT [${label} trial ${i}]: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker })}`); + process.exit(1); + } + const m = String(out.result).match(/\b(SHIP|HOLD)\b/); + if (!m) { console.error(`HARNESS FAULT [${label} trial ${i}]: no SHIP/HOLD token in ${JSON.stringify(String(out.result).slice(0, 120))}`); process.exit(1); } + if (m[1] === 'HOLD') holds += 1; else ships += 1; + console.log(` ${label} trial ${i + 1}/${N}: ${m[1]}`); + } + return { holds, ships }; +} + +console.log(`persona outcome probe v3 — ${N}/arm, claude-haiku-4-5, shipped recurse(), maxDepth:0 (single-shot leaf)\n`); +const base = await arm('BASE ', null); +const pers = await arm('PERSONA', PERSONA); +const delta = pers.holds - base.holds; +console.log(`\nHOLD-rate: base ${base.holds}/${N} persona ${pers.holds}/${N} delta ${delta >= 0 ? '+' : ''}${delta}`); + +if (base.ships < 6) console.log('READOUT: NO-HEADROOM — the baseline does not lean SHIP on this scenario; the pair cannot show a stance flip. Flag STAYS OPEN.'); +else if (delta >= 3) console.log('READOUT: EFFECT — the persona stance flips the decision class. Flag RESOLVED: persona is outcome-proven (one-knob pair).'); +else if (delta <= -3) console.log('READOUT: WRONG-SIGN — the risk-averse persona INCREASED shipping. Do NOT record; audit the harness first.'); +else console.log('READOUT: NULL — adoption without an outcome delta on this task. Flag resolved as measured-null (stance/format lever, not outcome).'); diff --git a/poc/audit2-scrub-wording.mjs b/poc/audit2-scrub-wording.mjs new file mode 100644 index 0000000..a381ef6 --- /dev/null +++ b/poc/audit2-scrub-wording.mjs @@ -0,0 +1,100 @@ +// AUDIT-2 PROBE B — does the capability-scrub PROMPT WORDING change behavior, or is it decoration? +// (Toggle-coverage audit follow-up: the mid-depth suffix — "PREFER DIRECT ACTION" — is mutation-tested for +// PRESENCE at the right depth, never for EFFECT. This is the missing one-knob pair, at the prompt seam the +// wording lives on: same SHIPPED constants, same tools, same task — only the suffix toggles.) +// +// DESIGN (able to fail both ways): a depth-1-shaped worker (spawn tool OFFERED — the tool half is a safety +// invariant and stays IDENTICAL in both arms) given a 3-part task that is trivially answerable directly but +// SHAPED to tempt fan-out (three enumerated jobs — the decomposition policy's own few-shot pattern). +// arm SUFFIX: system = DECOMPOSITION_POLICY + capabilityScrub(1, 3) (the shipped depth-1 wording) +// arm NO-SUFFIX: system = DECOMPOSITION_POLICY (wording removed, nothing else) +// OUTCOME (code-scored per trial): did the worker delegate (≥1 spawn_child call)? Secondary correctness +// floor: the final answer must contain all three code-known results (HELLO / 13 / cba). +// +// PRE-REGISTERED READOUTS (worded BEFORE running; N=8/arm): +// NO-HEADROOM no-suffix arm spawns 0/8 → the model never delegates this task anyway; the wording had +// nothing to suppress; FLAG STAYS OPEN (need a more tempting task). +// EFFECT spawn(no-suffix) − spawn(suffix) ≥ +3 → the wording measurably curbs needless delegation +// (its stated purpose); flag RESOLVED: outcome-proven. +// WRONG-SIGN spawn(suffix) − spawn(no-suffix) ≥ +3 → the wording INCREASES delegation; audit the harness +// before believing it. +// NULL |delta| ≤ 2 → wording makes no measurable difference on this task; flag +// resolved as measured-null (the tool contraction, not the prose, +// carries the scrub). +// Harness-confound guard: every trial must produce a non-empty final answer; correctness reported per arm +// (a wording that changed spawn rate by BREAKING answers would be a confound, not an effect). Exit 0 = probe +// ran cleanly (verdict is data); exit 1 = harness fault. +// +// RUN 1 (2026-07-20, three-trivial-jobs task): NO-HEADROOM — 0/8 delegation in BOTH arms; a task haiku +// one-shots gives the wording nothing to suppress. RUN 2 (this version): the task now MIRRORS the +// decomposition policy's OWN worked example 1 ("summarize the security posture of services A, B, and C" → +// spawn per service) — the maximal legitimate temptation the shipped prompt itself teaches, still without +// ever INSTRUCTING delegation. +// +// Run: ANTHROPIC_API_KEY=$(pass amr/claude_api) node poc/audit2-scrub-wording.mjs + +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { Loop } = require('../src/loop.js'); +const { DECOMPOSITION_POLICY, capabilityScrub } = require('../src/recurse-prompts.js'); +const { AnthropicProvider } = require('../src/provider-anthropic.js'); + +if (!process.env.ANTHROPIC_API_KEY) { console.error('needs ANTHROPIC_API_KEY'); process.exit(1); } +const provider = new AnthropicProvider({ apiKey: process.env.ANTHROPIC_API_KEY, model: 'claude-haiku-4-5' }); + +const TASK = + 'Summarize the security posture of our three services into one combined report:\n' + + '(1) auth-service — Node/Express, JWT validation, bcrypt password storage\n' + + '(2) billing-service — Stripe webhook processing with idempotency keys\n' + + '(3) gateway — nginx rate limiting and header-based routing'; + +const CORRECT = (t) => /auth/i.test(t) && /billing/i.test(t) && /gateway/i.test(t); +const N = 8; + +// The spawn tool is IDENTICAL in both arms (tool half of the scrub is a safety invariant, not under test). +// The stub child answers its sub-task deterministically so delegation "works" and isn't punished. +function spawnTool(counter) { + return { + name: 'spawn_child', + description: "Delegate a sub-task to a fresh-context child worker; returns only the child's result.", + parameters: { type: 'object', properties: { subtask: { type: 'string', description: 'the sub-task' } }, required: ['subtask'] }, + execute: async (args) => { + counter.calls += 1; + const s = String(args?.subtask || ''); + return `Posture summary for [${s.slice(0, 80)}]: configuration reviewed; no critical gaps found; standard hardening (secret rotation, dependency pinning) recommended.`; + }, + }; +} + +async function arm(label, system) { + let spawned = 0, correct = 0; + for (let i = 0; i < N; i++) { + const counter = { calls: 0 }; + const loop = new Loop({ provider, system, throwOnError: false }); + const out = await loop.run([{ role: 'user', content: TASK }], [spawnTool(counter)], {}); + const text = String(out.text || ''); + if (out.error || !text.trim()) { + console.error(`HARNESS FAULT [${label} trial ${i}]: ${JSON.stringify({ error: out.error })}`); + process.exit(1); + } + if (counter.calls > 0) spawned += 1; + if (CORRECT(text)) correct += 1; + console.log(` ${label} trial ${i + 1}/${N}: spawn_calls=${counter.calls} correct=${CORRECT(text)}`); + } + return { spawned, correct }; +} + +const SUFFIX_SYS = DECOMPOSITION_POLICY + capabilityScrub(1, 3); +if (!/prefer direct action/i.test(SUFFIX_SYS)) { console.error('HARNESS FAULT: shipped depth-1 scrub no longer contains the wording under test'); process.exit(1); } + +console.log(`scrub-wording probe — ${N}/arm, claude-haiku-4-5, shipped prompt constants, identical spawn tool\n`); +const withSuffix = await arm('SUFFIX ', SUFFIX_SYS); +const noSuffix = await arm('NO-SUFFIX', DECOMPOSITION_POLICY); +const delta = noSuffix.spawned - withSuffix.spawned; +console.log(`\nspawn-rate: no-suffix ${noSuffix.spawned}/${N} suffix ${withSuffix.spawned}/${N} delta ${delta >= 0 ? '+' : ''}${delta}`); +console.log(`correctness floor: no-suffix ${noSuffix.correct}/${N} suffix ${withSuffix.correct}/${N}`); + +if (noSuffix.spawned === 0) console.log('READOUT: NO-HEADROOM — the model never delegates this task; the wording had nothing to suppress. Flag STAYS OPEN.'); +else if (delta >= 3) console.log('READOUT: EFFECT — the wording measurably curbs needless delegation. Flag RESOLVED: outcome-proven (one-knob pair).'); +else if (delta <= -3) console.log('READOUT: WRONG-SIGN — the wording increases delegation. Do NOT record; audit the harness first.'); +else console.log('READOUT: NULL — no measurable wording effect on this task. Flag resolved as measured-null (the tool contraction carries the scrub).'); diff --git a/poc/ba15-broken-sensor.mjs b/poc/ba15-broken-sensor.mjs new file mode 100644 index 0000000..0993156 --- /dev/null +++ b/poc/ba15-broken-sensor.mjs @@ -0,0 +1,183 @@ +// BA-15 POC — broken-sensor detection at the refineLeaf close seam (adopter "faulty primitives" feedback, +// F25-zone borrow). Claim under test: a caller sensor that THROWS or returns a MALFORMED verdict is +// silently coerced into the same shapes as a genuine model failure: +// (A) sensor throw → bare {incomplete, best:null}, byte-identical to a provider/model fault (no blocker); +// (B) garbage verdict ({}) → refine burns ALL maxIterations with ZERO feedback (critique:null ⇒ the retry +// prompt is the plain task again), then reports receipts.refineLeaf.passed:false — an honest-looking +// model non-recovery pinned on a broken arbiter. +// Falsifier arm (C): a WELL-FORMED failing sensor DOES thread its critique into the retry prompt — proving +// the harness exercises the variable (the zero-feedback observation in B is real, not an unwired harness). +// +// Deterministic, offline, no API keys. Asserts the FIXED contract: +// - sensor throw / malformed verdict → {incomplete, blocker:'broken-sensor'} (+ receipts.blocker), loop +// STOPS immediately (no retries against a broken arbiter), best preserves the last attempt (BA-5); +// - a HaltError thrown by the sensor still propagates as a clean governance halt (never relabeled); +// - well-formed sensors (both {pass} and {status} shapes) behave exactly as before. +// Run PRE-fix: exits 1 while PRINTING the observed pathology (the evidence the gap is live in shipped code). +// Run POST-fix: exits 0. + +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { recurse } = require('../src/recurse.js'); +const { HaltError } = require('../src/errors.js'); + +const TASK = 'list the open files'; // 'simple' tier → definite leaf → refineLeaf engages + +function stubProvider({ fail = false } = {}) { + const calls = []; + return { + calls, + provider: { + model: 'stub-model', name: 'stub', + async generate(messages) { + calls.push({ messages: messages.map(m => ({ role: m.role, content: m.content })) }); + if (fail) throw new Error('connection reset by peer'); // a genuine provider/model-side fault + return { text: `ATTEMPT_${calls.length}_RESULT`, toolCalls: [], usage: { inputTokens: 5, outputTokens: 3 }, model: 'stub-model' }; + }, + }, + }; +} + +const lastUser = (call) => { const u = call.messages.filter(m => m.role === 'user'); return u.length ? u[u.length - 1].content : ''; }; + +let failures = 0; +const check = (cond, label, detail) => { + if (cond) { console.log(` PASS ${label}`); } + else { failures++; console.log(` FAIL ${label}${detail ? `\n observed: ${detail}` : ''}`); } +}; + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[A] sensor THROWS (caller test runner crashed — not the model\'s fault)'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new Error('ENOENT: node_modules/.bin/vitest not found'); } }, + }); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker, best: out.best })}`); + check(out.incomplete === true, 'sensor throw is not a converged result'); + check(out.blocker === 'broken-sensor', 'sensor throw is NAMED blocker:\'broken-sensor\'', `blocker=${JSON.stringify(out.blocker)} (unlabeled ⇒ collapses into "model failed")`); + check(out.receipts && out.receipts.blocker === 'broken-sensor', 'receipts carry the blocker too'); + check(sp.calls.length === 1, 'loop STOPS at the first broken close (no retries against a broken arbiter)', `${sp.calls.length} attempts ran`); + check(out.best === 'ATTEMPT_1_RESULT', 'the model\'s work is PRESERVED (BA-5 — the sensor broke, not the model)', `best=${JSON.stringify(out.best)}`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[A-control] genuine MODEL/provider fault — must remain distinguishable from [A]'); +{ + const sp = stubProvider({ fail: true }); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ pass: true }) }, // sensor is fine; the provider dies + }); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker })}`); + check(out.incomplete === true, 'provider fault is an honest incomplete'); + check(out.blocker !== 'broken-sensor', 'provider fault is NOT labeled broken-sensor', `blocker=${JSON.stringify(out.blocker)}`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[B] sensor returns GARBAGE ({}) — a malformed verdict, no pass/status'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({}), maxIterations: 3 }, + }); + const retryPrompts = sp.calls.slice(1).map(lastUser); + const zeroFeedback = retryPrompts.length > 0 && retryPrompts.every(p => !p.includes('FAILED')); + if (zeroFeedback) console.log(` pathology: ${sp.calls.length} attempts burned, every retry prompt carried ZERO feedback (identical plain task)`); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker, verdict: out.verdict })}`); + check(out.blocker === 'broken-sensor', 'malformed verdict is NAMED, never coerced', `blocker=${JSON.stringify(out.blocker)}, refineLeaf=${JSON.stringify(out.receipts && out.receipts.refineLeaf)}`); + check(sp.calls.length === 1, 'first malformed verdict STOPS the loop (attempts are not burned)', `${sp.calls.length} attempts ran`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[B2] other malformed shapes: undefined, string, {ok:true}'); +for (const [label, bad] of [['undefined', () => undefined], ['string "ok"', () => 'ok'], ['{ok:true}', () => ({ ok: true })]]) { + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { refineLeaf: { sensor: bad } }); + check(out.blocker === 'broken-sensor' && sp.calls.length === 1, `sensor → ${label} ⇒ broken-sensor, 1 attempt`, `blocker=${JSON.stringify(out.blocker)}, attempts=${sp.calls.length}`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[C-falsifier] WELL-FORMED failing sensor — critique wiring must work (proves [B]\'s zero-feedback is real)'); +{ + const sp = stubProvider(); + let calls = 0; + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => (++calls < 2 ? { pass: false, critique: 'MISSING_SEMICOLON_LINE_4' } : { pass: true }), maxIterations: 3 }, + }); + const retry = sp.calls.length > 1 ? lastUser(sp.calls[1]) : ''; + check(retry.includes('MISSING_SEMICOLON_LINE_4'), 'a real critique IS threaded into the retry prompt', `retry prompt: ${JSON.stringify(retry.slice(0, 120))}`); + check(out.blocker === undefined && out.receipts.refineLeaf.passed === true, 'well-formed sensor path unchanged (passes on retry)', JSON.stringify({ blocker: out.blocker, refineLeaf: out.receipts.refineLeaf })); +} + +console.log('\n[C2] status-shaped verdict ({status:\'failed\'}) stays a VALID terminal verdict, not broken-sensor'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ status: 'failed', critique: 'fundamentally wrong approach' }) }, + }); + check(out.blocker === undefined, 'status-only verdict shape is accepted', `blocker=${JSON.stringify(out.blocker)}`); + check(sp.calls.length === 1 && out.receipts.refineLeaf.passed === false, 'terminal failed stops after 1 attempt, honest non-pass', `attempts=${sp.calls.length}`); +} + +// --------------------------------------------------------------------------------------------------------- +console.log('\n[D] sensor throws HaltError — governance halt must propagate clean, never relabeled broken-sensor'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new HaltError('budget cap', { rule: 'budget' }); } }, + }); + check(out.incomplete === true && out.receipts.halted === true && out.blocker === undefined, 'HaltError from the sensor is a clean governance halt', JSON.stringify({ blocker: out.blocker, halted: out.receipts.halted })); +} + +// ========================================================================================================= +// VERIFIER SEAM (same fault class at the verify slot — found by the "did you validate all?" follow-up probe: +// pre-fix a THROWING caller `opts.evaluate` CRASHED the whole run on the plain-worker path and laundered to a +// bare {incomplete} under refineLeaf, and a GARBAGE return rode a CONVERGED-shaped {result, verdict:{}} out). +// --------------------------------------------------------------------------------------------------------- +console.log('\n[E] plain worker + THROWING caller verifier (opts.evaluate)'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + contract: 'must list files', evaluate: () => { throw new Error('rubric grader exploded'); }, + }).catch((e) => ({ threw: e.message })); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker, best: out.best, threw: out.threw })}`); + check(!out.threw, 'a broken caller verifier no longer CRASHES the run', `threw: ${out.threw}`); + check(out.blocker === 'broken-verifier', 'it is NAMED blocker:\'broken-verifier\'', `blocker=${JSON.stringify(out.blocker)}`); + check(out.best === 'ATTEMPT_1_RESULT', 'the unjudged work is preserved as best (BA-5)', `best=${JSON.stringify(out.best)}`); +} + +console.log('\n[E2] plain worker + GARBAGE caller verifier ({})'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { contract: 'must list files', evaluate: () => ({}) }); + console.log(` return: ${JSON.stringify({ incomplete: out.incomplete, blocker: out.blocker, verdict: out.verdict })}`); + check(out.incomplete === true && out.blocker === 'broken-verifier', 'a garbage verdict never rides out converged-shaped', `blocker=${JSON.stringify(out.blocker)}, verdict=${JSON.stringify(out.verdict)}`); + check(typeof out.receipts.blockerDetail === 'string' && out.receipts.blockerDetail.includes('neither'), 'the detail names the malformation'); +} + +console.log('\n[E3] refineLeaf + throwing verifier → labeled too (was a bare incomplete)'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ pass: true }) }, + contract: 'must list files', evaluate: () => { throw new Error('grader dead'); }, + }); + check(out.blocker === 'broken-verifier' && out.best === 'ATTEMPT_1_RESULT', 'labeled + work preserved under refineLeaf', JSON.stringify({ blocker: out.blocker, best: out.best })); +} + +console.log('\n[E4] controls: a FAILING well-formed verdict is judged-and-failed (not a blocker); HaltError stays a halt'); +{ + const sp = stubProvider(); + const out = await recurse(TASK, { provider: sp.provider }, { + contract: 'must list files', evaluate: () => ({ status: 'failed', pass: false, score: 0, critique: 'wrong', suggestions: [] }), + }); + check(out.blocker === undefined && out.verdict && out.verdict.pass === false, 'judged-and-failed returns {result, verdict} untouched', JSON.stringify({ blocker: out.blocker, verdict: out.verdict })); + const sp2 = stubProvider(); + const out2 = await recurse(TASK, { provider: sp2.provider }, { + contract: 'must list files', evaluate: () => { throw new HaltError('budget cap', { rule: 'budget' }); }, + }); + check(out2.incomplete === true && out2.receipts.halted === true && out2.blocker === undefined, 'verifier HaltError is a clean governance halt', JSON.stringify({ blocker: out2.blocker, halted: out2.receipts.halted })); +} + +console.log(failures === 0 ? '\nALL CHECKS PASS — the fixed contract holds.' : `\n${failures} CHECK(S) FAILED — the gap is live (pre-fix evidence above).`); +process.exit(failures === 0 ? 0 : 1); diff --git a/poc/ba15-round3-validate.mjs b/poc/ba15-round3-validate.mjs new file mode 100644 index 0000000..847e2ca --- /dev/null +++ b/poc/ba15-round3-validate.mjs @@ -0,0 +1,186 @@ +// BA-15 round-3 review — DETERMINISTIC OFFLINE validation of every reported finding against the +// CURRENT (unfixed) branch code. Run BEFORE any fix: each check must REPRODUCE the reported pathology, +// otherwise the finding is dropped rather than "fixed" on a reviewer's say-so (v0.27.0 precedent). +// +// node poc/ba15-round3-validate.mjs +// +// Exit 0 = every check ran; the table says which findings reproduced. This file characterizes the BUG, +// so a post-fix re-run is EXPECTED to flip the reproduced rows to NOT-REPRODUCED. + +import { createRequire } from 'node:module'; +const require = createRequire(import.meta.url); +const { recurse } = require('../src/recurse.js'); +const { HaltError } = require('../src/errors.js'); + +const rows = []; +const record = (id, what, reproduced, evidence) => { + rows.push({ id, what, reproduced, evidence }); + console.log(`${reproduced ? 'REPRODUCED ' : 'NOT-REPRODUCED'} ${id} ${what}\n ${evidence}\n`); +}; + +// ---------------------------------------------------------------- scripted provider +function scripted(handler, { model = 'stub-model', name = 'stub' } = {}) { + const calls = []; + return { + calls, + provider: { + model, name, + async generate(messages, tools, options) { + calls.push({ messages, tools: (tools || []).map(t => t.name), options }); + const r = (await handler(messages, tools, options, calls.length - 1)) || {}; + return { text: r.text || '', toolCalls: r.toolCalls || [], usage: { inputTokens: 5, outputTokens: 3 }, model }; + }, + }, + }; +} +const plain = (text) => scripted(() => ({ text })).provider; +const COMPLEX = 'design and implement a notification pipeline across the entire system'; +const SIMPLE = 'list the open files'; + +// ---------------------------------------------------------------- F2 verdict spread erases prototype fields +{ + class V { + constructor(s, c) { this._s = s; this._c = c; } + get status() { return this._s; } + get critique() { return this._c; } + } + const out = await recurse(SIMPLE, { provider: plain('the answer') }, { + maxDepth: 0, + evaluate: () => new V('satisfied', 'looks good'), + }); + const lost = out.verdict && out.verdict.status === undefined && out.verdict.critique === undefined; + record('F2', 'runArbiter spread erases prototype-backed status/critique', + !!lost, `verdict.status=${JSON.stringify(out.verdict?.status)} critique=${JSON.stringify(out.verdict?.critique)} (caller sent 'satisfied'/'looks good')`); +} + +// ---------------------------------------------------------------- F4 strict boolean `pass` hard-blocks truthy +{ + let attempts = 0; + const out = await recurse(SIMPLE, { provider: scripted(() => { attempts++; return { text: 'attempt output' }; }).provider }, { + maxDepth: 0, + refineLeaf: { sensor: () => ({ pass: 1, critique: '' }), maxIterations: 3 }, + }); + record('F4', 'sensor {pass:1} (truthy non-boolean) hard-blocks as broken arbiter', + out.blocker === 'broken-sensor', `blocker=${out.blocker} incomplete=${out.incomplete} attempts=${attempts} (pre-BA-15 this converged)`); +} + +// ---------------------------------------------------------------- F7 non-Error throw → [object Object] +{ + const out = await recurse(SIMPLE, { provider: plain('x') }, { + maxDepth: 0, + refineLeaf: { sensor: () => { throw { code: 'ENOENT', path: '/usr/bin/vitest' }; }, maxIterations: 2 }, + }); + const d = out.receipts?.blockerDetail || ''; + record('F7', 'non-Error throw yields an uninformative blockerDetail', + d.includes('[object Object]'), `blockerDetail=${JSON.stringify(d)}`); +} + +// ---------------------------------------------------------------- F8 throwing accessor reported as "threw" +{ + const out = await recurse(SIMPLE, { provider: plain('x') }, { + maxDepth: 0, + refineLeaf: { sensor: () => new Proxy({}, { get() { throw new Error('trap'); } }), maxIterations: 2 }, + }); + const d = out.receipts?.blockerDetail || ''; + record('F8', 'a RETURNED object whose accessor throws is reported as the sensor THROWING', + d.includes('threw'), `blockerDetail=${JSON.stringify(d)} (the sensor returned normally; the fault is its shape)`); +} + +// ---------------------------------------------------------------- F3 fanout halt discards the computed reduce +// HARNESS NOTE: the first draft fed the Planner `{steps:[{description}]}`, but it parses a bare JSON ARRAY of +// `{id, action, dependsOn}`. The plan failed to parse, the run ended BEFORE the reduce, and `best:null` looked +// like the reported bug — a false REPRODUCED on a path the probe never reached. Drive the planner by call index. +{ + let n = 0, i = 0; + const provider = scripted(() => { + if (i++ === 0) return { text: JSON.stringify([0, 1, 2].map(j => ({ id: `s${j}`, action: `SLICE ${j}`, dependsOn: [] }))) }; + return { text: `child-${++n}` }; + }).provider; + const out = await recurse(COMPLEX, { provider }, { + count: 3, + synthesize: ({ results }) => ({ total: results.length }), + evaluate: () => { throw new HaltError('budget cap', { rule: 'budget' }); }, + }); + const isReduce = out.best && typeof out.best === 'object' && out.best.total === 3; + record('F3', 'fanout halt discards the computed reduce (no BA-5 hoist)', + !isReduce, `best=${JSON.stringify(out.best)} (the reduce returned {total:3}; a string/null means it was thrown away)`); +} + +// ---------------------------------------------------------------- F5 mid-scan halt returns best:null +// KNOWN LIMIT, not fixed here: `scanCount` throws without surfacing the windows it already judged, so there is +// nothing at this seam to preserve. Fixing it is a retrieval-side change. The misleading comment that claimed +// BA-5 coverage for this case WAS corrected — this row is expected to stay REPRODUCED and is tracked as an +// open item, not papered over. +{ + const provider = scripted(() => { throw new HaltError('budget cap', { rule: 'budget' }); }).provider; + const corpus = Array.from({ length: 12 }, (_, i) => ({ id: `r${i}`, text: `record ${i}` })); + const out = await recurse('how many records mention widgets', { provider }, { corpus, retrieval: 'scan', window: 4 }); + record('F5*', 'a halt DURING the scan returns best:null (KNOWN LIMIT — comment corrected, not fixed)', + out.incomplete === true && out.best == null, `incomplete=${out.incomplete} best=${JSON.stringify(out.best)}`); +} + +// ---------------------------------------------------------------- F1 / F6 child blocker at the spawn + halt boundary +{ + let sawToolResult = null; + const provider = scripted((messages, tools, options, i) => { + const names = (tools || []).map(t => t.name); + const last = messages[messages.length - 1]; + if (last?.role === 'tool' || (typeof last?.content === 'string' && last.content.startsWith('[incomplete]'))) { + sawToolResult = last.content; + return { text: 'parent closing answer' }; + } + if (names.includes('spawn_child') && i === 0) { + return { text: '', toolCalls: [{ id: 't1', name: 'spawn_child', arguments: { subtask: 'do slice A' } }] }; + } + return { text: 'leaf output' }; + }).provider; + + const out = await recurse(COMPLEX, { provider }, { + maxDepth: 1, + refineLeaf: { sensor: () => { throw new Error('child sensor boom'); }, maxIterations: 2 }, + evaluate: () => { throw new HaltError('budget cap', { rule: 'budget' }); }, + }); + + const childNode = (out.receipts?.spawned || [])[0]; + record('F6', 'spawn boundary collapses a child broken-sensor into a generic [incomplete]', + typeof sawToolResult === 'string' && sawToolResult.startsWith('[incomplete]') && !/sensor|broken/i.test(sawToolResult), + `tool result the parent model saw = ${JSON.stringify(sawToolResult)} (child receipts blocker=${childNode?.blocker})`); + + record('F1', 'halt catch drops the child blocker (no inheritedBlocker call)', + out.incomplete === true && out.blocker === undefined && childNode?.blocker === 'broken-sensor', + `top-level blocker=${out.blocker} while child receipts blocker=${childNode?.blocker}`); +} + +// ---------------------------------------------------------------- F12 unreachable broken-verifier arm +// The premise (forChild strips `evaluate`, so no child can carry 'broken-verifier') is permanent and stays +// true — the FIX is that inheritedBlocker no longer pretends to match it. Assert the dead arm is gone. +{ + const { readFileSync } = await import('node:fs'); + const src = readFileSync(new URL('../src/recurse.js', import.meta.url), 'utf8'); + const body = (src.match(/function inheritedBlocker\([\s\S]*?\n}/) || [''])[0]; + record('F12', "inheritedBlocker still matches the unreachable 'broken-verifier'", + body.includes("'broken-verifier'"), `inheritedBlocker ${body.includes("'broken-verifier'") ? 'STILL references' : 'no longer references'} the unreachable label`); +} + +// ---------------------------------------------------------------- F10 detached evaluate changes `this` +{ + class Grader { + constructor() { this.threshold = 0.5; } + check() { return { pass: this.threshold > 0 }; } + } + const g = new Grader(); + const out = await recurse(SIMPLE, { provider: plain('x') }, { maxDepth: 0, evaluate: g.check }); + record('F10', 'an unbound method verifier now throws (this === undefined) → broken-verifier', + out.blocker === 'broken-verifier', `blocker=${out.blocker} detail=${JSON.stringify(out.receipts?.blockerDetail)}`); +} + +// ---------------------------------------------------------------- summary +console.log('='.repeat(100)); +const rep = rows.filter(r => r.reproduced); +console.log(`${rep.length}/${rows.length} findings REPRODUCED against current branch code:`); +for (const r of rep) console.log(` - ${r.id} ${r.what}`); +const not = rows.filter(r => !r.reproduced); +if (not.length) { + console.log(`\n${not.length} did NOT reproduce (drop unless re-argued):`); + for (const r of not) console.log(` - ${r.id} ${r.what}`); +} diff --git a/src/recurse.js b/src/recurse.js index 104555f..6bf3f19 100644 --- a/src/recurse.js +++ b/src/recurse.js @@ -238,6 +238,13 @@ function auditSafeCtx(ctx, overrides = {}) { * NEVER a worker side-effect a worker with edit tools could GAME (writing a passing file then returning junk, or * editing the failing test itself). A gameable close is the reward-hacking surface every RSI system in the field * got bitten by; the loop optimizes against WHATEVER the sensor reads, so keep it outside what the worker can write. + * **Broken sensor ≠ failing model (BA-15):** a sensor that THROWS (non-Halt) or returns a MALFORMED verdict + * (anything but `{pass: boolean}` or a valid tri-state `status`) is a faulty ARBITER — the loop stops at the + * FIRST broken close (never retries against it) and returns a labeled `{incomplete, blocker:'broken-sensor'}` + * (+ `receipts.blockerDetail`), with `best` preserving the model's last attempt. A `HaltError` thrown by the + * sensor stays a clean governance halt. The sensor's EXECUTION environment is the caller's: run untrusted / + * model-generated checks in an isolated child process WITH A TIMEOUT — a sensor that hangs forever hangs the + * leaf (no bareguard checkpoint fires between sensor start and return). * **`rejectedBuffer` (BA-14):** a SkillOpt-shaped rejected-attempt buffer — instead of only the LATEST critique, * surface the model's OWN prior failed attempts verbatim ("you wrote these, they failed X — write something * STRUCTURALLY DIFFERENT"). This is DIRECTED diversity (attack the specific repeated mistake), where escalation @@ -306,8 +313,17 @@ function auditSafeCtx(ctx, overrides = {}) { * @property {Verdict|null} verdict * @property {boolean} incomplete * @property {boolean} halted - * @property {string} [blocker] - (BA-11) set to `'governance-deny'` when this node stopped because its Loop - * short-circuited a consecutive-policy-deny spin (not a model failure). Mirrors `RecurseResult.blocker`. + * @property {string} [blocker] - Set when this node stopped for a specific non-model reason (mirrors + * `RecurseResult.blocker`): `'governance-deny'` (BA-11) — its Loop short-circuited a consecutive-policy-deny + * spin; `'broken-sensor'` (BA-15) — the caller's `refineLeaf.sensor` threw or returned a malformed verdict; + * `'broken-verifier'` (BA-15) — the caller's `opts.evaluate` did (the default Evaluator path is never labeled). + * @property {string} [blockerDetail] - (BA-15) with a `broken-*` blocker: what the arbiter did (threw with + * which message, or which malformed shape it returned) — the actionable half of the label. + * @property {{blocker: string, blockerDetail?: string, blockerTask?: string}} [blockerFrom] - (BA-15) a + * DESCENDANT's blocker, surfaced here so an aggregating node still reports the fault upward. Deliberately + * SEPARATE from this node's own `blocker` (which means "THIS node's arbiter/Loop broke"): stamping a + * descendant's label onto every ancestor made the receipts tree accuse nodes whose sensor never ran, and + * re-labelled a parent `governance-deny` when only one child was denied. `blockerTask` names the culprit. * @property {object|null} tokens - The worker Loop's `metrics.tokens`. * @property {{iterations: number, passed: boolean, temperatures: (number|null)[], rejectedBuffer: boolean}} [refineLeaf] - (BA-8) when * this leaf ran as a bounded refine loop: how many attempts it took and whether the deterministic sensor finally @@ -341,6 +357,18 @@ function auditSafeCtx(ctx, overrides = {}) { * @property {string} [blocker] - Present when `incomplete` for a specific, actionable reason. `'governance-deny'` * (BA-11): the worker's Loop short-circuited after N consecutive policy denials rather than burn to the * budget cap — the caller can widen scope / re-gate / escalate instead of reading it as a model failure. + * `'broken-sensor'` (BA-15): the caller's `refineLeaf.sensor` threw or returned a malformed verdict — the + * ARBITER is faulty, not the model; fix the sensor and re-run (`receipts.blockerDetail` says what it did). + * `'broken-verifier'` (BA-15): same fault class at the verify slot — the caller's `opts.evaluate` threw + * (non-Halt) or returned a malformed verdict; the default Evaluator path is never labeled (its failures are + * provider-class faults). For both `broken-*` blockers `best` preserves the model's last non-empty output + * (BA-5) — the arbiter broke, so treat it as best-effort work rather than a graded pass. + * @property {string} [blockerDetail] - (BA-15) with a `broken-*` blocker: what the arbiter did — the + * ACTIONABLE half of the label, surfaced on the result (not only in `receipts`) so a caller branching on + * `blocker` can report the cause without walking the receipts tree. + * @property {string} [blockerTask] - (BA-15) when the blocker was INHERITED from a descendant in a nested run: + * which sub-task actually broke. Without it a nested failure reports "a sensor broke" with no way to find + * which one. * @property {RecurseNode} receipts - The audit node for this call (RC-10). */ @@ -586,16 +614,16 @@ async function recurse(task, ctx = {}, opts = {}) { const missingSlices = node.spawned.filter(c => c.incomplete).map(c => c.task); if (missingSlices.length > 0) { node.incomplete = true; - return { incomplete: true, best: result, missingSlices, receipts: node }; + // BA-15: carry a child's blocker up, so a nested broken sensor still names itself at the top. + return incompleteWithBlocker(node, result, { missingSlices }); } // Verify: a SEPARATE-context judge, never the generator grading itself. Runs when a contract is given, the // caller supplied a verifier, OR the task is critical (the forced-verify safety rail). const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. + return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; @@ -603,12 +631,221 @@ async function recurse(task, ctx = {}, opts = {}) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - return { incomplete: true, best: result, receipts: node }; + // BA-15: a halt AFTER the children ran (e.g. mid-synthesize) must still name a child's broken sensor — + // otherwise the caller reads a governance problem where their own arbiter crashed. + return incompleteWithBlocker(node, result); } throw err; } } +/** Valid tri-state `Verdict.status` values a caller arbiter (sensor/verifier) may return in lieu of a boolean `pass`. */ +const SENSOR_STATUS = new Set(['satisfied', 'needs_revision', 'failed']); + +/** + * BA-15 — a TYPED signal that a caller arbiter (the `refineLeaf.sensor` or the `opts.evaluate` verifier) + * broke (threw non-Halt, or returned a malformed verdict). Classified by `instanceof` + `.tag`, NOT by + * re-parsing an `Error.message` prefix — so an intermediate layer that rewords the message (the exact class + * of the prior loop.js HaltError-wrapping bug) or a Loop/provider error whose text happens to begin + * `broken-sensor: ` can never be mis-labeled. Module-local: it is always thrown AND caught inside recurse.js + * (never propagates to a caller), so it needs no `errors.js` entry or public export. + */ +class BrokenArbiterError extends Error { + /** + * @param {'broken-sensor'|'broken-verifier'} tag - which seam broke. + * @param {string} detail - what the arbiter did (surfaced as `receipts.blockerDetail`). + * @param {{cause?: any}} [options] - `cause` preserves the original error (stack/type) for debugging. + */ + constructor(tag, detail, options = {}) { + super(`${tag}: ${detail}`, options.cause !== undefined ? { cause: options.cause } : undefined); + this.name = 'BrokenArbiterError'; + this.tag = tag; + this.detail = detail; + } +} + +/** + * BA-15 — validate a caller arbiter's return at a close seam (the `refineLeaf.sensor` AND the caller + * `opts.evaluate` verifier). A verdict is well-formed iff it is an object carrying a boolean `pass` OR a + * valid tri-state `status` (the two shapes `refine`/callers branch on). Returns `null` when well-formed, + * else a short description of the malformation ("named, never coerced" — a garbage verdict otherwise reads + * as pass:false with critique:null at the sensor seam, or rides a converged-shaped return at the verify slot). + * @param {any} v + * @returns {string|null} + */ +function verdictShapeFault(v) { + if (v === null || typeof v !== 'object' || Array.isArray(v)) { + return `returned ${v === null ? 'null' : Array.isArray(v) ? 'an array' : `a ${typeof v}`}`; + } + // A PRESENT `pass` counts whatever its type (`1`/`0`/`'yes'` are a long-standing yes-no convention, and + // `refine` has always branched on its TRUTHINESS). BA-15 exists to catch a verdict carrying NO usable + // signal — not to reject one that answers clearly in a different dialect: demanding a strict boolean + // silently flipped previously-CONVERGING adopter sensors to a permanent first-attempt block. + if (v.pass != null || SENSOR_STATUS.has(v.status)) return null; + const keys = Object.keys(v).slice(0, 5).join(', '); + return `returned an object with neither a usable \`pass\` nor a valid \`status\` (keys: ${keys || 'none'})`; +} + +/** Upper bound on a `blockerDetail` fragment — it rides into receipts and, via a wired gate, onto disk. */ +const DETAIL_MAX = 200; + +/** + * BA-15 — a readable one-liner for ANYTHING thrown by a caller arbiter. `String(err)` on a non-Error throw + * (a test harness rejecting with a raw `{code:'ENOENT', path}` result is the common case) yields the useless + * `[object Object]`, defeating `blockerDetail`'s entire purpose: naming what broke so the operator can fix it. + * @param {any} err + * @returns {string} + */ +function describeThrown(err) { + const clamp = (s) => (s.length > DETAIL_MAX ? `${s.slice(0, DETAIL_MAX)}… (truncated)` : s); + if (err instanceof Error && typeof err.message === 'string' && err.message) return clamp(err.message); + if (err === null || typeof err !== 'object') return clamp(String(err)); + // Deliberately NOT a whole-object dump. A thrown non-Error is typically a spawn/exec RESULT, which routinely + // carries a full stdout buffer and an env snapshot — and `blockerDetail` rides into `receipts`, which a wired + // gate serializes VERBATIM into a plaintext audit log (the F16/BA-1 lesson: never let caller data of unknown + // shape reach the audit unfiltered). Take only the conventional diagnostic fields, clamped. + try { + const picked = ['name', 'code', 'errno', 'syscall', 'path', 'status', 'signal', 'message'] + .filter(k => typeof err[k] === 'string' || typeof err[k] === 'number') + .map(k => `${k}=${String(err[k])}`) + .join(' '); + if (picked) return clamp(picked); + } catch { /* a throwing accessor / Proxy trap — fall through to the type tag */ } + return Object.prototype.toString.call(err); +} + +/** + * BA-15 — run a caller arbiter (the `refineLeaf.sensor` or the `opts.evaluate` verifier) and NAME a broken one + * rather than let it launder. A non-`HaltError` throw or a malformed return (per {@link verdictShapeFault}) is + * re-thrown as a typed {@link BrokenArbiterError} the call sites classify by `instanceof` + `.tag` (NEVER by + * re-parsing the message text); a `HaltError` passes straight through (clean + * governance exit, BA-2). One helper so the sensor and verifier seams stay byte-identical (same guidance + * string, same fault taxonomy) — a divergence between them was the copy-paste risk this replaces. + * A verdict that carries a valid `status` but no boolean `pass` is NORMALIZED to add `pass = status === + * 'satisfied'` (the same derivation as `evaluator.js`): the advertised `{status}` contract must actually + * WORK — `refine.js` stops the leaf on `verdict.pass`, so a bare `{status:'satisfied'}` would otherwise never + * satisfy, burn every iteration, and report `passed:false` (a satisfied close mislabeled as non-recovery). + * @template {{pass?: boolean, status?: string}} T + * @param {'broken-sensor'|'broken-verifier'} tag - the fault channel, carried on the thrown error's `.tag`. + * @param {() => (T | Promise)} call - invokes the arbiter (already bound to its result/ctx args). + * @returns {Promise} the arbiter's well-formed verdict (with `pass` derived from `status` when absent). + */ +async function runArbiter(tag, call) { + const who = tag === 'broken-sensor' ? 'sensor' : 'evaluate'; + const shape = `a ${who === 'sensor' ? 'sensor' : 'verifier'} must return {pass} or {status: 'satisfied'|'needs_revision'|'failed'}`; + /** A throw from the arbiter's own body (sync or async rejection) — "the sensor threw". */ + const threw = (err) => { + if (err instanceof HaltError || err instanceof BrokenArbiterError) return err; + // `cause` keeps the original error (stack + type) reachable for debugging; `detail` stays the short label. + return new BrokenArbiterError(tag, `${who} threw: ${describeThrown(err)}`, { cause: err }); + }; + /** A throw from READING the value the arbiter returned — "the verdict is unreadable", a different fault. */ + const unreadable = (err) => { + if (err instanceof HaltError || err instanceof BrokenArbiterError) return err; + return new BrokenArbiterError(tag, `${who} returned a verdict whose properties could not be read: ${describeThrown(err)} — ${shape}`, { cause: err }); + }; + + // Typed `any` deliberately: this block probes an UNTRUSTED caller return (it may be a promise, a plain + // verdict, or a hostile Proxy), so the generic `T | Promise` narrowing does not apply until it is settled. + /** @type {any} */ + let raw; + try { + raw = call(); + } catch (err) { throw threw(err); } + + // `await`ing directly would conflate two different faults: the await PROBES `.then` on the returned value, + // so an accessor-backed/Proxy verdict throws during the await and gets reported as "the sensor threw" — + // sending the operator hunting a `throw` in a sensor that returned perfectly normally. Probe the thenable + // separately (a throw HERE is the returned value being unreadable) and only then await (a rejection THERE + // is genuinely the arbiter's body failing). + let thenable; + try { + thenable = raw !== null && (typeof raw === 'object' || typeof raw === 'function') && typeof raw.then === 'function'; + } catch (err) { throw unreadable(err); } + + /** @type {any} */ + let v = raw; + if (thenable) { + try { + v = await raw; + } catch (err) { throw threw(err); } + } + // The SHAPE INSPECTION is guarded SEPARATELY: reading `.status`/`.pass`/`Object.keys` on a returned Proxy or + // accessor-backed object can itself throw. Unguarded that escapes untyped (the very uncaught crash this seam + // prevents, one step later) — but folding it into the call's own catch is also wrong: it reports "the sensor + // threw" for a sensor that RETURNED NORMALLY, sending the operator hunting a `throw` that does not exist. + // The fault is in the returned VALUE, so it is named as such. + try { + const fault = verdictShapeFault(v); + if (fault) throw new BrokenArbiterError(tag, `${who} ${fault} — ${shape}`); + // Derive `pass` from a status-only verdict so the advertised `{status}` shape actually gates `refine`, which + // branches on `verdict.pass`. NEVER mutate the caller's object — and never flatten it either: an object + // spread copies OWN enumerable properties only, so a class-instance verdict whose `status`/`critique` are + // PROTOTYPE getters passes the shape check above and then comes out the other side with those fields + // ERASED (critique lost ⇒ every retry re-sends the plain task with zero feedback — the exact burn BA-15 + // exists to prevent). Copy descriptors onto the SAME prototype so accessor-backed fields survive. + if (v.pass == null && typeof v.status === 'string') { + const copy = Object.create(Object.getPrototypeOf(v), Object.getOwnPropertyDescriptors(v)); + Object.defineProperty(copy, 'pass', { + value: v.status === 'satisfied', writable: true, enumerable: true, configurable: true, + }); + return /** @type {T} */ (copy); + } + return v; + } catch (err) { throw unreadable(err); } +} + +/** + * BA-15 — surface a CHILD's blocker at the PARENT. A nested tree aggregates a dead child into + * `{incomplete, missingSlices}`; without this the child's `broken-sensor` (or `governance-deny`) label is + * dropped at the first parent, so a top-level caller branching on `result.blocker` sees nothing and debugs + * the model instead of its own sensor — the exact laundering BA-15 exists to close, reintroduced one level up. + * `broken-sensor` wins over `governance-deny`: it names a fault in the CALLER's own code, which is both more + * actionable and cheaper to fix than a gate decision. + * + * `blockerTask` names WHICH descendant broke. Without it a nested run reports "a sensor broke" with no way to + * find which one — half a fix. (It is also why the label is NOT stamped onto the parent's own `blocker`; see + * {@link incompleteWithBlocker}.) + * + * `'broken-verifier'` is deliberately NOT matched here: `forChild` strips `evaluate`, so a child never runs a + * caller verifier and no child node can carry that label. Matching it would be unreachable code asserting a + * capability that does not exist. + * @param {RecurseNode[]} spawned - this node's child receipts. + * @returns {{blocker: string, blockerDetail?: string, blockerTask?: string}|null} + */ +function inheritedBlocker(spawned) { + const pick = spawned.find(c => c.incomplete && c.blocker === 'broken-sensor') + || spawned.find(c => c.incomplete && c.blocker); + if (!pick || !pick.blocker) return null; + return { + blocker: pick.blocker, + ...(pick.blockerDetail && { blockerDetail: pick.blockerDetail }), + ...(pick.task && { blockerTask: pick.task }), + }; +} + +/** + * BA-15 — the ONE shared "this node is incomplete because a descendant was" return. Used by every aggregating + * path (worker / partition / fanout) on BOTH its `missingSlices` branch AND its `HaltError` catch: the halt + * branches originally skipped the inherit entirely, so a gate tripping AFTER a broken-sensor child (e.g. mid- + * synthesize) dropped the label and the caller read a governance problem where their own sensor had crashed. + * + * The inherited label rides the RETURNED result (the caller-facing API — that surfacing is the whole point) + * but is recorded on the node as `blockerFrom`, NOT by overwriting `node.blocker`. `node.blocker` means "THIS + * node's own arbiter/Loop broke"; stamping a descendant's label onto every ancestor made the receipts tree + * accuse nodes whose sensor never ran, and re-labelled a parent `governance-deny` when only one child was + * denied — pointing the operator at the wrong node to re-gate. + * @param {RecurseNode} node + * @param {any} best + * @param {{missingSlices?: string[]}} [extra] - extra fields for the non-halt aggregation branch. + * @returns {RecurseResult} + */ +function incompleteWithBlocker(node, best, extra = {}) { + const inherited = inheritedBlocker(node.spawned); + if (inherited) node.blockerFrom = inherited; + return /** @type {RecurseResult} */ ({ incomplete: true, best, ...extra, ...(inherited || {}), receipts: node }); +} + /** * BA-8 leaf-refine — run a DEFINITE leaf as a bounded generate→sense→regenerate loop (relayfact F17). Reuses the * existing `refine.js` primitive (the Outcomes iterate→grade→revise port): each attempt is a FRESH leaf Loop @@ -665,6 +902,11 @@ async function recurseRefineLeaf(task, ctx, opts, state) { // a dropped attempt is stored as `null` ("provider default"). Indexed by iteration (refine calls once each). /** @type {(number|null)[]} */ const effectiveTemps = []; + // BA-15/BA-5: the last attempt's text, kept OUTSIDE refine so a broken-sensor stop can still preserve the + // model's work — when the ARBITER breaks, the work was never judged; destroying it would punish the model + // for the caller's fault (refine's own history is lost on the throw). + /** @type {string|null} */ + let lastAttemptText = null; // One attempt = a fresh leaf Loop (no spawn tool: a retry is a direct correction, not a re-decomposition) at the // iteration's temperature, with the GAP fed forward as fresh feedback. A governance halt → throw so refine stops. const attempt = async ({ iteration, critique, history }) => { @@ -700,15 +942,29 @@ async function recurseRefineLeaf(task, ctx, opts, state) { const dropped = /** @type {{temperatureDropped?: boolean}} */ (out).temperatureDropped; effectiveTemps[iteration] = dropped ? null : temperature; accrueTokens(out.metrics ? out.metrics.tokens : null); + // BA-5: capture BEFORE the throws. The Loop returns its last non-empty text on EVERY terminating path + // (halt/deny/truncation/refusal), so a throw placed above this line would discard the text of the very + // attempt that terminated — leaving `best:null` on a FIRST-attempt halt, the exact loss the catch-branch + // preservation exists to prevent. (Caught by review: the earlier tests only halted on attempt 2, where a + // prior clean attempt had already populated this.) + lastAttemptText = out.text || lastAttemptText; if (typeof out.error === 'string' && out.error.startsWith('halt:')) throw new HaltError('refine-leaf attempt halted', { rule: out.error.slice('halt:'.length) }); if (out.error) throw new Error(out.error); // a non-halt worker fault → honest incomplete return out.text; }; + // BA-15: the sensor call is WRAPPED (via runArbiter) so a broken arbiter is NAMED, never coerced. A non-Halt + // throw (the caller's test runner crashed — ENOENT, syntax error in the harness) and a malformed return are + // the same fault class: "didn't judge", which must never collapse into "judged-and-failed" (the model's + // fault) or a bare {incomplete} (indistinguishable from a provider death). The throw stops refine at the + // FIRST broken close — retrying against a broken arbiter burns every remaining attempt for nothing (each + // retry would carry critique:null, i.e. the plain task again). HaltError passes through (governance, BA-2). + const evaluate = (result, c) => runArbiter('broken-sensor', () => sensor(result, { task, context: opts.context, contract: c.contract })); + try { const outcome = await refine({ attempt, - evaluate: (result, c) => sensor(result, { task, context: opts.context, contract: c.contract }), + evaluate, contract: typeof opts.contract === 'string' ? opts.contract : undefined, maxIterations, }); @@ -722,9 +978,8 @@ async function recurseRefineLeaf(task, ctx, opts, state) { // Optional rubric layer on top of the deterministic sensor (RC-7): forced for critical, or a contract/override. const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. + return await verifyOrBlock(task, result, ctx, opts, node); } // No rubric layer ⇒ the sensor's final verdict IS the node verdict (a non-pass is surfaced, not hidden). node.verdict = outcome.verdict || null; @@ -733,23 +988,43 @@ async function recurseRefineLeaf(task, ctx, opts, state) { node.tokens = tokensSum; // record whatever attempts DID spend, on both the halt and fault paths // The refineLeaf receipt must ride EVERY terminating path, not just the clean one (same invariant as BA-10's // `temperatureDropped`): a leaf that ran attempts then halted/faulted still spent tokens and may have engaged - // the buffer. `effectiveTemps[iteration]` is set BEFORE each attempt's throw, so it reflects every attempt - // made; `passed:false` because the catch is only reached on a throw (a pass returns from the try above). - node.refineLeaf = { iterations: effectiveTemps.length, passed: false, temperatures: effectiveTemps.slice(), rejectedBuffer: bufferUsed }; + // the buffer. `effectiveTemps[iteration]` is set BEFORE each attempt's throw, so it reflects every attempt made. + // NOT unconditional: the refine loop may have COMPLETED (receipt already written, possibly `passed:true`) and + // the throw come from the verify layer below it — clobbering that to `passed:false` would report a sensor that + // never closed when it did, misattributing a verify-slot halt to a non-converging leaf. + if (!node.refineLeaf) { + node.refineLeaf = { iterations: effectiveTemps.length, passed: false, temperatures: effectiveTemps.slice(), rejectedBuffer: bufferUsed }; + } + // BA-5: EVERY terminating branch preserves the model's last non-empty attempt (`lastAttemptText`), never + // `null` — matching the plain-worker path (`best: out.text || null`) and the "bounds PRESERVE work" + // invariant. A refine leaf is the sole cross-attempt text channel in a ralph-style retry; dropping it on a + // halt/deny/fault (as this path did pre-BA-15) loses attempt N's only bridge to attempt N+1. if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - return { incomplete: true, best: null, receipts: node }; + return { incomplete: true, best: lastAttemptText, receipts: node }; } // BA-11: a deny-spin inside a refine attempt (the Loop short-circuited after N consecutive governance // denials, rethrown at recurse.js as `denied:`) is a LABELED governance block, not a model fault. if (typeof err?.message === 'string' && err.message.startsWith('denied:')) { node.incomplete = true; node.blocker = 'governance-deny'; - return { incomplete: true, best: null, blocker: 'governance-deny', receipts: node }; + return { incomplete: true, best: lastAttemptText, blocker: 'governance-deny', receipts: node }; + } + // BA-15: the caller's SENSOR broke — a faulty arbiter, not a model failure. Named (like BA-11's + // governance-deny) so the caller fixes the sensor instead of debugging the model; `best` preserves the + // model's last non-empty attempt (BA-5) — the arbiter, not the model, is at fault. + if (err instanceof BrokenArbiterError && err.tag === 'broken-sensor') { + node.incomplete = true; + node.blocker = 'broken-sensor'; + node.blockerDetail = err.detail; + // `blockerDetail` rides the RESULT too, not just receipts: it is the actionable half of the label (WHAT + // the sensor did), and a parent reading a child's return — or a caller branching on the documented + // `{blocker}` shape — should not have to walk the receipts tree to get it. + return { incomplete: true, best: lastAttemptText, blocker: 'broken-sensor', blockerDetail: err.detail, receipts: node }; } node.incomplete = true; - return { incomplete: true, best: null, receipts: node }; + return { incomplete: true, best: lastAttemptText, receipts: node }; } } @@ -800,6 +1075,10 @@ async function recurseScan(task, ctx, opts, state) { }; } + // BA-5: hoisted so a halt thrown BELOW the scan (e.g. from the verify slot) still returns the finished, + // code-counted result as `best` instead of destroying it — re-running a scan re-pays every window judge call. + /** @type {{count: number, matchedIds: string[]}|null} */ + let scanResult = null; try { const scan = await scanCount(task, corpus, { provider, @@ -812,6 +1091,7 @@ async function recurseScan(task, ctx, opts, state) { node.scan = { window: scan.window, passes: scan.passes, scanned: scan.scanned, matched: scan.count }; // Structured, CODE-counted result — the count is authoritative; matchedIds carry the evidence (RC-10). const result = { count: scan.count, matchedIds: scan.matchedIds }; + scanResult = result; // RC-9: a dead window means we did NOT see every slice → the count is a floor, not the answer. Report it // incomplete with the partial as `best`, never a clean pass over a hole. @@ -824,16 +1104,21 @@ async function recurseScan(task, ctx, opts, state) { // structured count against the goal/contract (an isolated grader, never the scanner itself). const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. + return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; } catch (err) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - return { incomplete: true, best: null, receipts: node }; + // BA-5, PARTIAL by construction: a halt from the VERIFY slot lands here with the scan already finished, + // so the counted result is returned instead of null (it previously survived only in receipts, forcing a + // full re-scan). A halt DURING `scanCount` — the likelier case, since a token cap trips after many window + // judge calls — still yields `null`: `scanCount` throws without surfacing the windows it did judge, so + // there is nothing here to preserve. Fixing that needs `scanCount` to return its partial union on halt + // (a retrieval-side change, not this seam's); tracked as a known limit rather than papered over here. + return { incomplete: true, best: scanResult, receipts: node }; } throw err; } @@ -891,6 +1176,8 @@ async function recursePartition(task, ctx, opts, state) { // (never `undefined`) across every dispatch path. const childResults = []; + /** @type {{count: number, matchedIds: string[]}|null} */ + let partitionResult = null; try { // 2b) Pre-wave checkpoint — width (the cost) is now known. A governance HaltError halts BEFORE any worker // spends (bounds the burst to zero); a plain deny is advisory (allowlist-safe), same contract as fanout. @@ -935,25 +1222,30 @@ async function recursePartition(task, ctx, opts, state) { if (child.incomplete) missingSlices.push(label); } const result = { count: matched.size, matchedIds: [...matched] }; + partitionResult = result; // BA-5: a halt below this (e.g. verify) returns the full result, not a count-only rebuild node.partition.matched = matched.size; if (missingSlices.length > 0) { node.incomplete = true; - return { incomplete: true, best: result, missingSlices, receipts: node }; + // BA-15: carry a child's blocker up (see inheritedBlocker). + return incompleteWithBlocker(node, result, { missingSlices }); } const wantVerify = critical || typeof opts.contract === 'string' || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. + return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; } catch (err) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - const best = childResults.length ? { count: new Set(childResults.flatMap((v) => (v && Array.isArray(v.matchedIds) ? v.matchedIds : []))).size } : null; - return { incomplete: true, best, receipts: node }; + // Prefer the finished result (a halt from the verify slot lands here with it already computed); else + // rebuild a best-effort count from whatever slices did return (BA-5, never a bare null). + const best = partitionResult + || (childResults.length ? { count: new Set(childResults.flatMap((v) => (v && Array.isArray(v.matchedIds) ? v.matchedIds : []))).size } : null); + // BA-15: a halt must not swallow a child's broken-sensor label (see incompleteWithBlocker). + return incompleteWithBlocker(node, best); } throw err; } @@ -987,6 +1279,11 @@ async function recurseFanout(task, ctx, opts, state) { const childResults = []; const contract = typeof opts.contract === 'string' ? opts.contract : null; + // BA-5: declared OUTSIDE the try so a halt from the verify slot (which lands in the catch below, with the + // reduce already computed and PAID FOR) returns the finished reduce instead of a lossy re-join of the raw + // child strings — a different TYPE from the documented reduce output, and an 'merge' strategy's LLM call + // thrown away. Mirrors `scanResult`/`partitionResult` on the sibling paths. + let result; try { // 1) Decompose into exactly `count` independent parallel steps (the NB-2 Planner seam). A non-Halt planner @@ -1057,7 +1354,6 @@ async function recurseFanout(task, ctx, opts, state) { // 4) NB-3 reduce over the slice results. Unlike Family A there is no parent closing turn, so we ALWAYS // reduce: a `synthesize` FUNCTION is the deterministic code-reduce (§9.1); a string runs the built-in // reducer; unset defaults to lossless `'concat'`. (`childResults` always has `count` entries.) - let result; if (typeof opts.synthesize === 'function') { result = await opts.synthesize({ task, text: null, results: childResults, children: node.spawned, ctx }); } else { @@ -1077,24 +1373,27 @@ async function recurseFanout(task, ctx, opts, state) { // 5) Honest completeness (RC-9): any missing slice → incomplete, with the partial reduce as `best`. if (missingSlices.length > 0) { node.incomplete = true; - return { incomplete: true, best: result, missingSlices, receipts: node }; + // BA-15: carry a child's blocker up (see inheritedBlocker). + return incompleteWithBlocker(node, result, { missingSlices }); } // 6) Verify (RC-7): forced for critical, or when a contract/override is supplied. const wantVerify = critical || contract != null || typeof opts.evaluate === 'function'; if (wantVerify) { - const verdict = await verify(task, result, ctx, opts); - node.verdict = verdict; - return { result, verdict, receipts: node }; + // `return await` (not a bare `return`) is load-bearing here — see verifyOrBlock's JSDoc. + return await verifyOrBlock(task, result, ctx, opts, node); } return { result, verdict: null, receipts: node }; } catch (err) { if (err instanceof HaltError) { node.halted = true; node.incomplete = true; - // best-effort partial: whatever slices we did collect, losslessly joined (no LLM — the gate already tripped) - const best = childResults.length ? childResults.filter(v => v !== '').join('\n\n') : null; - return { incomplete: true, best: best || null, receipts: node }; + // BA-5: prefer the FINISHED reduce when the halt landed after it (the verify slot); only fall back to a + // lossless join of the raw slices when the halt tripped before/during the reduce itself. Never a bare null. + const rejoined = childResults.length ? childResults.filter(v => v !== '').join('\n\n') : null; + const best = result !== undefined ? result : (rejoined || null); + // BA-15: a halt must not swallow a child's broken-sensor label (see incompleteWithBlocker). + return incompleteWithBlocker(node, best); } throw err; } @@ -1161,6 +1460,16 @@ function buildSpawnTool(ctx, opts, depth, maxDepth, node, childResults) { // silently dropped or faked. The same declared value is collected for the NB-3 reducer. const value = child.incomplete ? (child.best == null ? '' : child.best) : (child.result == null ? '' : child.result); childResults.push(value); + // BA-15: a child blocked by a BROKEN ARBITER must not read to the parent model as a generic failure. + // Collapsed into a bare `[incomplete]`, the parent cannot tell "the model failed" from "the judge is + // broken", so it re-spawns the identical subtask against the identical broken sensor — re-running a + // full leaf attempt each time, up to the Loop's round limit. That is BA-15's own spend-burn ("retrying + // against a broken arbiter carries zero feedback") reintroduced one level up, so the tool result says + // so explicitly and tells the model not to retry. + if (child.incomplete && child.blocker === 'broken-sensor') { + const where = child.blockerTask ? ` (in sub-task: ${child.blockerTask})` : ''; + return `[blocked: broken-sensor] The CHECK that judges this subtask is itself broken${where} — ${child.blockerDetail || 'no detail available'}. Retrying will hit the same broken check: do NOT re-delegate this subtask; report it as blocked. Partial output: ${String(value)}`.trim(); + } if (child.incomplete) return `[incomplete] ${String(value)}`.trim(); return String(value); }, @@ -1184,7 +1493,19 @@ function verify(task, result, ctx, opts) { // and an agentic critic needs the path to exercise the artifact. A caller `evaluate` gets the RAW task (it owns // its own context); only the default isolated grader is contextualized. if (typeof opts.evaluate === 'function') { - return Promise.resolve(opts.evaluate(result, { contract, task })); + // BA-15 (verifier seam): the CALLER-supplied verifier is wrapped exactly like the refineLeaf sensor — a + // non-Halt throw or a malformed return is a faulty ARBITER, tagged so the call sites label it (pre-fix a + // throw crashed the whole run on the plain-worker path / laundered to a bare {incomplete} under refineLeaf, + // and a garbage verdict rode a CONVERGED-shaped {result, verdict} out). The default Evaluator path below is + // NOT wrapped: it constructs well-formed Verdicts by design, and its failures are provider-class faults. + // Capture the narrowed reference in a const: `typeof opts.evaluate === 'function'` does NOT survive into the + // nested async closure (TS re-widens `opts.evaluate` to possibly-undefined there → TS2722/TS18048). + // BOUND to `opts`, because a bare `const evaluate = opts.evaluate` DETACHES the method: the call used to be + // `opts.evaluate(...)` (receiver `opts`), and a caller passing a class method (`evaluate: grader.check`) + // would suddenly get `this === undefined` and throw on its first `this.x` read — a working (if degraded) + // integration flipped to a hard `broken-verifier` by an unrelated typecheck fix. + const evaluate = opts.evaluate.bind(opts); + return runArbiter('broken-verifier', () => evaluate(result, { contract, task })); } const provider = ctx.provider || opts.provider; const evaluator = new Evaluator({ provider }); @@ -1199,4 +1520,39 @@ function verify(task, result, ctx, opts) { ); } +/** + * BA-15 (verifier seam) — run the verify slot, converting a `BrokenArbiterError` into a LABELED + * `{ incomplete, blocker:'broken-verifier' }` return with `best` preserving the result the arbiter failed to + * judge (BA-5: the work exists — best-effort, not a graded pass). One helper so all five dispatch paths (worker / + * refineLeaf / scan / partition / fanout) get identical semantics. Anything else (HaltError, a default- + * Evaluator provider fault) rethrows to the caller's own catch, exactly as before. + * + * MUST be called as `return await verifyOrBlock(...)` from inside each caller's `try` — a bare + * `return verifyOrBlock(...)` returns the promise and exits the `try` before it settles, so a verifier + * `HaltError` would escape the caller's own catch instead of landing as a clean `{ incomplete, halted }` + * (proven by `poc/ba15-broken-sensor.mjs` [E4]). + * @param {string} task + * @param {any} result + * @param {RecurseCtx} ctx + * @param {RecurseOptions} opts + * @param {RecurseNode} node + * @returns {Promise} + */ +async function verifyOrBlock(task, result, ctx, opts, node) { + try { + const verdict = await verify(task, result, ctx, opts); + node.verdict = verdict; + return { result, verdict, receipts: node }; + } catch (err) { + // Only a typed BrokenArbiterError is a caller-verifier fault; everything else (HaltError, a default- + // Evaluator provider fault) rethrows to the caller's own catch. Classified by type, never by message text. + if (!(err instanceof BrokenArbiterError)) throw err; + node.incomplete = true; + node.blocker = 'broken-verifier'; + node.blockerDetail = err.detail; + // `blockerDetail` rides the RESULT too (see the sensor-side sibling) — the actionable half of the label. + return { incomplete: true, best: result, blocker: 'broken-verifier', blockerDetail: err.detail, receipts: node }; + } +} + module.exports = { recurse }; diff --git a/test/recurse.test.js b/test/recurse.test.js index 46a21c5..87b87ac 100644 --- a/test/recurse.test.js +++ b/test/recurse.test.js @@ -661,6 +661,413 @@ describe('recurse — leaf retry-with-sensor (BA-8 / refineLeaf, relayfact F17)' }); }); +describe('recurse — broken-sensor blocker (BA-15, "a broken arbiter is named, never coerced")', () => { + // Pre-BA-15 pathologies these tests are mutated against: (a) a sensor THROW fell through to the bare + // {incomplete, best:null} — byte-identical to a provider death, and the model's work was destroyed; (b) a + // MALFORMED verdict ({}, 'ok', {ok:true}) read as pass:false with critique:null, so refine burned every + // remaining attempt re-sending the PLAIN task (zero feedback), then the run surfaced as an honest-looking + // model non-recovery — or worse, a converged-shaped {result, verdict:{}}. + const OK = () => ({ status: 'satisfied', pass: true, score: 1, critique: '', suggestions: [] }); + + it('a THROWING sensor → labeled {incomplete, blocker:\'broken-sensor\'} with the detail, work preserved (BA-5)', async () => { + const sp = scriptedProvider(() => ({ text: 'the model actually did the work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new Error('ENOENT: vitest not found'); } }, + }); + assert.equal(out.incomplete, true); + assert.equal(out.blocker, 'broken-sensor', 'the faulty arbiter is NAMED (pre-fix: unlabeled, read as model failure)'); + assert.equal(out.receipts.blocker, 'broken-sensor', 'receipts mirror the blocker (BA-11 pattern)'); + assert.match(out.receipts.blockerDetail, /sensor threw: ENOENT: vitest not found/, 'the detail says WHAT broke'); + assert.equal(out.best, 'the model actually did the work', 'best preserves the unjudged attempt — the sensor broke, not the model'); + assert.equal(sp.calls.length, 1, 'stops at the first broken close'); + }); + + it('a MALFORMED verdict stops the loop at the FIRST broken close — attempts are not burned with zero feedback', async () => { + const sp = scriptedProvider(() => ({ text: 'attempt' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({}), maxIterations: 3 }, + }); + assert.equal(out.blocker, 'broken-sensor', 'a garbage verdict is named, never coerced to pass:false'); + assert.match(out.receipts.blockerDetail, /neither a usable `pass` nor a valid `status`/); + assert.equal(sp.calls.length, 1, 'pre-fix: 3 attempts burned, every retry a feedback-free re-send of the plain task'); + assert.equal(out.result, undefined, 'never a converged-shaped return riding a garbage verdict'); + }); + + it('malformed shapes are each caught: undefined, string, array, {ok:true}', async () => { + for (const bad of [() => undefined, () => 'ok', () => ['pass'], () => ({ ok: true })]) { + const sp = scriptedProvider(() => ({ text: 'attempt' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { refineLeaf: { sensor: bad } }); + assert.equal(out.blocker, 'broken-sensor', `sensor ${bad} must be flagged`); + assert.equal(sp.calls.length, 1, 'one attempt only'); + } + }); + + it('a genuine PROVIDER fault stays distinguishable — incomplete WITHOUT the broken-sensor label', async () => { + const sp = scriptedProvider(() => { throw new Error('connection reset by peer'); }); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { refineLeaf: { sensor: OK } }); + assert.equal(out.incomplete, true); + assert.equal(out.blocker, undefined, 'a model/provider fault must NOT be pinned on the sensor'); + }); + + it('a HaltError thrown by the sensor stays a clean governance halt, never relabeled broken-sensor', async () => { + const sp = scriptedProvider(() => ({ text: 'attempt' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new HaltError('budget cap', { rule: 'budget.maxCostUsd' }); } }, + }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true, 'governance halt recorded'); + assert.equal(out.blocker, undefined, 'a governance halt is not a sensor fault'); + }); + + it('well-formed verdict shapes are untouched: minimal {pass} retries with critique; {status:\'failed\'} is terminal', async () => { + // Minimal {pass:false, critique} (no status) must still drive a retry carrying the critique — the + // validator accepts BOTH documented shapes, not just the full Verdict. + let n = 0; + const sp = scriptedProvider(() => ({ text: `attempt_${++n}` })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: (r) => (String(r).includes('_2') ? { pass: true } : { pass: false, critique: 'NEEDS_V2' }), maxIterations: 3 }, + }); + assert.equal(out.blocker, undefined, 'minimal {pass} shape is valid'); + assert.ok(lastUser(sp.calls[1].messages).includes('NEEDS_V2'), 'the critique still reaches the retry'); + assert.equal(out.receipts.refineLeaf.passed, true); + + const sp2 = scriptedProvider(() => ({ text: 'attempt' })); + const out2 = await recurse(SIMPLE_TASK, { provider: sp2.provider }, { + refineLeaf: { sensor: () => ({ status: 'failed', critique: 'wrong approach' }) }, + }); + assert.equal(out2.blocker, undefined, 'status-only shape is valid'); + assert.equal(sp2.calls.length, 1, 'terminal failed still stops after one attempt'); + assert.equal(out2.receipts.refineLeaf.passed, false, 'honest non-pass, not a blocker'); + }); + + it('a status-only {status:\'satisfied\'} verdict STOPS the leaf and reports passed — not burned as never-passed', async () => { + // The advertised contract lets a sensor return {status:'satisfied'} (no boolean pass). refine.js stops on + // verdict.pass, so pre-fix this ran all maxIterations (pass undefined → never stops) and reported + // passed:false — a satisfied close mislabeled as non-recovery + 3× the token spend. runArbiter now derives + // pass = status==='satisfied', so it stops on the first attempt and reports passed:true. + let n = 0; + const sp = scriptedProvider(() => ({ text: `attempt_${++n}` })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ status: 'satisfied' }), maxIterations: 3 }, + }); + assert.equal(out.blocker, undefined, 'a satisfied status-only verdict is valid, not a blocker'); + assert.equal(sp.calls.length, 1, 'stops on the first satisfied close (pre-fix: burned all 3)'); + assert.equal(out.receipts.refineLeaf.passed, true, 'reported as passed (pre-fix: false — pass was undefined)'); + }); + + it('the HaltError branch preserves the model\'s last attempt as best (BA-5), not null', async () => { + // Pre-fix the HaltError / governance-deny / generic-fault branches returned best:null even though the leaf + // had produced work — a BA-5 violation, inconsistent with the plain-worker path (best: out.text). They now + // return lastAttemptText, matching the broken-sensor branch and the plain worker. + let call = 0; + const sp = scriptedProvider(() => { call += 1; if (call >= 2) throw new HaltError('cap', { rule: 'budget' }); return { text: 'attempt-1 work' }; }); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { refineLeaf: { sensor: () => ({ pass: false, critique: 'retry' }) } }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true); + assert.equal(out.best, 'attempt-1 work', 'a halt on the retry preserves attempt 1 (pre-fix: best:null)'); + }); + + it('a FIRST-attempt halt still preserves that attempt\'s text (BA-5 capture-before-throw)', async () => { + // Review-caught regression in the previous fix: `lastAttemptText` was assigned AFTER the halt/error throws, + // so the attempt that actually terminated had its text discarded — best:null on a first-attempt halt. The + // earlier test only halted on attempt 2 (where a prior clean attempt had already populated it), so it + // passed while the single-attempt case was broken. The Loop returns its last non-empty text on every + // terminating path, so the capture must precede the throws. + const sp = scriptedProvider(() => ({ text: 'partial work' })); + const ctx = { provider: sp.provider, onLlmResult: () => { throw new HaltError('budget cap', { rule: 'budget.maxCostUsd' }); } }; + const out = await recurse(SIMPLE_TASK, ctx, { refineLeaf: { sensor: () => ({ pass: false, critique: 'x' }) } }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true); + assert.equal(out.best, 'partial work', 'the halting attempt\'s own text survives (pre-fix: null)'); + }); + + it('a child\'s broken-sensor blocker propagates to the PARENT in a nested tree (BA-15 anti-laundering)', async () => { + // A parent aggregates a dead child into {incomplete, missingSlices}; pre-fix the child's blocker label was + // dropped there, so a top-level caller branching on result.blocker saw nothing and would debug the model + // instead of its own broken sensor — the laundering BA-15 exists to close, reintroduced one level up. + const sp = scriptedProvider(decomposingHandler()); + const out = await recurse(COMPLEX_TASK, { provider: sp.provider }, { + maxDepth: 2, + refineLeaf: { sensor: () => { throw new Error('ENOENT: harness missing'); } }, + }); + assert.equal(out.incomplete, true, 'a broken sensor at the leaf makes the tree incomplete'); + assert.equal(out.blocker, 'broken-sensor', 'the child\'s label reaches the top (pre-fix: undefined)'); + // The parent's OWN `blocker` stays clear: it means "THIS node's arbiter broke", and the parent's never ran. + // Stamping the descendant's label onto every ancestor made the receipts tree accuse innocent nodes, so the + // inherited label rides `blockerFrom` (with `blockerTask` naming the culprit) instead. + assert.equal(out.receipts.blocker, undefined, 'the parent is not accused of its child\'s fault'); + assert.equal(out.receipts.blockerFrom.blocker, 'broken-sensor', 'recorded on the parent as INHERITED'); + assert.equal(typeof out.blockerTask, 'string', 'and the culprit subtask is named'); + }); + + it('a verify-slot halt after a PASSING sensor does not clobber the refineLeaf receipt to passed:false', async () => { + // verifyOrBlock is awaited INSIDE the try, so a halt at the verify slot reaches the catch even though the + // refine loop already completed and wrote an honest passed:true receipt. Overwriting it would report a + // sensor that never closed when it did. + const sp = scriptedProvider(() => ({ text: 'FIXED answer' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: (r) => ({ pass: String(r).includes('FIXED') }) }, + contract: 'must be fixed', + evaluate: () => { throw new HaltError('cap', { rule: 'budget' }); }, + }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true, 'the governance halt is recorded'); + assert.equal(out.receipts.refineLeaf.passed, true, 'the sensor DID close — receipt preserved (pre-fix: false)'); + }); + + it('the generic-fault branch preserves the model\'s last attempt as best (BA-5), not null', async () => { + let call = 0; + const sp = scriptedProvider(() => { call += 1; if (call >= 2) throw new Error('provider exploded'); return { text: 'attempt-1 work' }; }); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { refineLeaf: { sensor: () => ({ pass: false, critique: 'retry' }) } }); + assert.equal(out.incomplete, true); + assert.equal(out.blocker, undefined, 'a plain provider fault is not a labeled blocker'); + assert.equal(out.best, 'attempt-1 work', 'the fault branch preserves attempt 1 (pre-fix: best:null)'); + }); + + it('the refineLeaf receipt rides the broken-sensor path (every-terminating-path invariant)', async () => { + const sp = scriptedProvider(() => ({ text: 'attempt' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw new Error('boom'); } }, + }); + assert.ok(out.receipts.refineLeaf, 'receipt present'); + assert.equal(out.receipts.refineLeaf.passed, false); + assert.equal(out.receipts.refineLeaf.iterations, 1, 'records the attempt that ran before the broken close'); + }); + + // ---- Verifier seam (same fault class at the verify slot: a caller opts.evaluate is an arbiter too). + // Pre-fix pathologies: a THROWING caller verifier CRASHED the whole run as an exception on the plain-worker + // path (and laundered to a bare {incomplete} under refineLeaf); a GARBAGE return rode a CONVERGED-shaped + // {result, verdict:{}} out. The default Evaluator rubric path is deliberately NOT labeled (well-formed by + // construction; its failures are provider-class faults) — narrowest guard. + + it('a THROWING caller verifier (opts.evaluate) → labeled broken-verifier, work preserved, never a crash', async () => { + const sp = scriptedProvider(() => ({ text: 'the actual work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => { throw new Error('rubric grader exploded'); }, + }); + assert.equal(out.incomplete, true, 'pre-fix this line was unreachable — recurse() threw'); + assert.equal(out.blocker, 'broken-verifier', 'the faulty verifier is NAMED'); + assert.match(out.receipts.blockerDetail, /evaluate threw: rubric grader exploded/); + assert.equal(out.best, 'the actual work', 'the unjudged result is preserved as best (BA-5)'); + }); + + it('a GARBAGE caller verdict never rides out converged-shaped', async () => { + const sp = scriptedProvider(() => ({ text: 'the actual work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => ({}), + }); + assert.equal(out.blocker, 'broken-verifier', 'pre-fix: {result, verdict:{}} — a converged shape riding garbage'); + assert.equal(out.result, undefined, 'no result field on the labeled incomplete'); + assert.match(out.receipts.blockerDetail, /neither a usable `pass` nor a valid `status`/); + }); + + it('under refineLeaf a broken verifier is labeled broken-VERIFIER (distinct from broken-sensor)', async () => { + const sp = scriptedProvider(() => ({ text: 'the actual work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: OK }, + contract: 'must do the thing', evaluate: () => { throw new Error('grader dead'); }, + }); + assert.equal(out.blocker, 'broken-verifier', 'the label says WHICH arbiter broke (sensor passed, verifier died)'); + assert.equal(out.best, 'the actual work', 'work preserved'); + assert.equal(out.receipts.refineLeaf.passed, true, 'the sensor receipt is honest — the leaf loop itself succeeded'); + }); + + it('a verifier HaltError is a clean governance halt — the return await regression guard', async () => { + // verifyOrBlock is called as `return await ...` INSIDE each path's try: a bare `return ` exits + // the try before settling, so the HaltError would escape recurse() as an uncaught throw (caught live by + // the POC's [E4] control arm). This test fails if the await is dropped. + const sp = scriptedProvider(() => ({ text: 'work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => { throw new HaltError('budget cap', { rule: 'budget.maxCostUsd' }); }, + }); + assert.equal(out.incomplete, true, 'a clean incomplete, not a thrown run'); + assert.equal(out.receipts.halted, true, 'recorded as a governance halt'); + assert.equal(out.blocker, undefined, 'never relabeled broken-verifier'); + }); + + it('a well-formed FAILING verdict from opts.evaluate stays judged-and-failed — {result, verdict}, no blocker', async () => { + const sp = scriptedProvider(() => ({ text: 'the actual work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => ({ status: 'needs_revision', pass: false, score: 3, critique: 'gap', suggestions: [] }), + }); + assert.equal(out.blocker, undefined); + assert.equal(out.result, 'the actual work', 'a judged failure still returns the result + verdict'); + assert.equal(out.verdict.pass, false); + }); + + it('the broken-verifier label rides the FANOUT dispatch path too (one helper, five paths)', async () => { + const sp = scriptedProvider(fanoutHandler()); + const out = await recurse(COMPLEX_TASK, { provider: sp.provider }, { + count: 2, evaluate: () => { throw new Error('grader dead on fanout'); }, + }); + assert.equal(out.blocker, 'broken-verifier', 'the fanout path labels a broken verifier identically'); + assert.ok(out.best, 'the reduced slices are preserved as best'); + }); + + // ---------------------------------------------------------------- round-3 review regressions + // Each of the following reproduces a defect the third review round found in BA-15's OWN fixes, validated + // against the pre-fix code by `poc/ba15-round3-validate.mjs` before being fixed here. + + it('a PROTOTYPE-backed verdict keeps its status/critique (spread would erase accessor fields)', async () => { + // A class-instance verdict passes the shape check by reading `.status` through a getter, but an object + // spread copies OWN enumerable props only — so status/critique came out ERASED: the critique never reached + // the retry prompt (zero-feedback burn, the exact thing BA-15 exists to stop) and `verdict.status` read + // undefined to the caller. Regression-proof: revert to `{...v, pass}` and both asserts go red. + class V { + constructor(s, c) { this._s = s; this._c = c; } + get status() { return this._s; } + get critique() { return this._c; } + } + const sp = scriptedProvider(() => ({ text: 'the work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + contract: 'must do the thing', evaluate: () => new V('satisfied', 'looks good'), + }); + assert.equal(out.verdict.status, 'satisfied', 'prototype getter survives normalization'); + assert.equal(out.verdict.critique, 'looks good', 'the critique is not erased'); + assert.equal(out.verdict.pass, true, 'and `pass` is still derived from the status'); + }); + + it('a TRUTHY non-boolean `pass` is accepted, not hard-blocked as a faulty arbiter', async () => { + // `{pass: 1}` is a long-standing yes/no convention and `refine` has always branched on truthiness, so a + // strict-boolean gate silently flipped previously-CONVERGING adopter sensors to a permanent first-attempt + // block. BA-15's job is a verdict with NO usable signal — not one answering clearly in another dialect. + const sp = scriptedProvider(() => ({ text: 'attempt output' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => ({ pass: 1, critique: '' }), maxIterations: 3 }, + }); + assert.equal(out.blocker, undefined, 'a truthy pass is a real verdict, not a broken arbiter'); + assert.equal(out.incomplete, undefined, 'the leaf converges as it did before BA-15'); + assert.equal(out.receipts.refineLeaf.passed, true, 'and is recorded as an honest pass'); + }); + + it('a NON-ERROR throw still yields an actionable blockerDetail (never [object Object])', async () => { + const sp = scriptedProvider(() => ({ text: 'work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => { throw { code: 'ENOENT', path: '/usr/bin/vitest' }; } }, + }); + assert.equal(out.blocker, 'broken-sensor'); + assert.doesNotMatch(out.receipts.blockerDetail, /\[object Object\]/, 'the detail must identify something'); + assert.match(out.receipts.blockerDetail, /ENOENT/, 'the thrown value is described'); + }); + + it('blockerDetail is BOUNDED and does not dump the thrown object wholesale (audit-log safety)', async () => { + // A thrown non-Error is typically a spawn/exec RESULT carrying a full stdout buffer and an env snapshot. + // blockerDetail rides into `receipts`, which a wired gate serializes VERBATIM to a plaintext audit log — + // so a whole-object JSON dump writes caller secrets to disk (the F16/BA-1 leak class). Only conventional + // diagnostic fields are taken, clamped. Mutation-proof: restore `JSON.stringify(err)` and both go red. + const sp = scriptedProvider(() => ({ text: 'work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { + sensor: () => { throw { code: 'ENOENT', stdout: 'x'.repeat(50000), env: { SECRET_TOKEN: 'sk-live-abc123' } }; }, + }, + }); + assert.ok(out.receipts.blockerDetail.length < 400, 'the detail is bounded, never an unbounded buffer dump'); + assert.doesNotMatch(out.receipts.blockerDetail, /sk-live-abc123/, 'caller secrets never reach the audit record'); + assert.match(out.receipts.blockerDetail, /ENOENT/, 'while still naming the actual fault'); + }); + + it('an UNREADABLE returned verdict is named as such, not reported as the sensor throwing', async () => { + // A sensor returning a Proxy whose accessors throw RETURNED NORMALLY — reporting "sensor threw" sends the + // operator hunting a `throw` that does not exist. NB the fault surfaces during the `await`'s own `.then` + // probe, so the two cases must be separated around the await, not just around the shape inspection. + const sp = scriptedProvider(() => ({ text: 'work' })); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { + refineLeaf: { sensor: () => new Proxy({}, { get() { throw new Error('trap'); } }) }, + }); + assert.equal(out.blocker, 'broken-sensor'); + assert.match(out.receipts.blockerDetail, /could not be read/, 'the fault is the RETURNED VALUE, not a throw'); + assert.doesNotMatch(out.receipts.blockerDetail, /threw: trap/, 'must not accuse the sensor body'); + }); + + it('a method-reference verifier does not hard-fail as a broken arbiter (receiver restored)', async () => { + // `const evaluate = opts.evaluate` DETACHED the method; the call had always been `opts.evaluate(...)`, so + // `this` was the OPTIONS object. Detached, `this` became undefined and the first `this.x` read THREW — + // flipping a working (if degraded) integration to a hard broken-verifier. Binding back to `opts` restores + // the prior behavior exactly. NOTE the honest limit that restores: `this` is the options object, never the + // grader instance, so a method depending on its own fields still reads undefined. A caller wanting its own + // receiver must bind (`evaluate: g.check.bind(g)`) — asserted below so the contract is not overstated. + class Grader { + constructor() { this.threshold = 0.5; } + // UNGUARDED on purpose: detached (`this === undefined`) this THROWS, which is the regression. Bound to + // `opts` it reads undefined and degrades to false — exactly the pre-BA-15 behavior being restored. + check() { return { pass: this.threshold > 0, critique: '' }; } + } + const sp = scriptedProvider(() => ({ text: 'work' })); + const g = new Grader(); + const out = await recurse(SIMPLE_TASK, { provider: sp.provider }, { contract: 'c', evaluate: g.check }); + assert.equal(out.blocker, undefined, 'a method reference must not become a broken-verifier (pre-fix: it did)'); + assert.equal(out.verdict.pass, false, 'degraded exactly as before: `this` is opts, so threshold is undefined'); + + const bound = await recurse(SIMPLE_TASK, { provider: scriptedProvider(() => ({ text: 'work' })).provider }, { + contract: 'c', evaluate: g.check.bind(g), + }); + assert.equal(bound.verdict.pass, true, 'a caller-bound method sees its own fields — the documented way'); + }); + + it('a HALT after a broken-sensor child still names the child\'s blocker (halt catch inherits too)', async () => { + // The halt branches skipped the inherit entirely, so a gate tripping AFTER the children ran (here: mid- + // synthesize) dropped the label — the caller read a governance problem where their own sensor had crashed. + const sp = scriptedProvider(decomposingHandler()); + const out = await recurse(COMPLEX_TASK, { provider: sp.provider }, { + maxDepth: 2, + refineLeaf: { sensor: () => { throw new Error('child sensor boom'); } }, + synthesize: () => { throw new HaltError('budget cap', { rule: 'budget' }); }, + }); + assert.equal(out.incomplete, true); + assert.equal(out.receipts.halted, true, 'it really was a halt'); + assert.equal(out.blocker, 'broken-sensor', 'the child\'s label survives the halt (pre-fix: undefined)'); + }); + + it('the spawn boundary tells the parent a child was BLOCKED, not just incomplete', async () => { + // Collapsed to a bare `[incomplete]`, the parent model cannot tell "the model failed" from "the judge is + // broken", so it re-delegates the same subtask into the same broken sensor, burning a full leaf attempt + // each time — BA-15's own spend-burn, one level up. + const sp = scriptedProvider(decomposingHandler()); + await recurse(COMPLEX_TASK, { provider: sp.provider }, { + maxDepth: 2, + refineLeaf: { sensor: () => { throw new Error('child sensor boom'); } }, + }); + const toolResults = sp.calls.flatMap(c => c.messages.filter(m => m.role === 'tool').map(m => m.content)); + const blocked = toolResults.find(t => typeof t === 'string' && t.includes('broken-sensor')); + assert.ok(blocked, 'the parent model is told the ARBITER broke, not merely that the child failed'); + assert.match(blocked, /do NOT re-delegate/i, 'and is told retrying hits the same broken check'); + }); + + it('a broken-sensor child OUTRANKS a governance-denied sibling, and the parent is not accused', async () => { + // Two rules that had NO test before: (a) broken-* wins over governance-deny (a fault in the CALLER's own + // code is more actionable than a gate decision); (b) the inherited label rides `blockerFrom`, never the + // parent's own `blocker`. Driven through the public API, not an exported internal. + // slice 0 → spins on a denied tool ⇒ governance-deny (and is FIRST, so a naive single `find` picks it) + // slice 1 → its sensor throws ⇒ broken-sensor + // Mutation-proof for (a): collapse inheritedBlocker to one `find(c => c.incomplete && c.blocker)` and the + // first assert goes red (governance-deny wins on array order). + const editTool = { name: 'edit', description: 'edit a file', execute: async () => 'edited' }; + const sp = scriptedProvider((messages, tools, options, i) => { + if (isPlanner(messages)) { + return { text: JSON.stringify([0, 1].map(j => ({ id: `s${j}`, action: `SLICE ${j}: count the alpha records`, dependsOn: [] }))) }; + } + if (isVerify(messages)) return { text: SATISFIED }; + // slice 0's worker keeps calling the denied tool → the Loop's deny-spin guard short-circuits it + if (lastUser(messages).includes('SLICE 0')) return { toolCalls: [{ id: 'e' + i, name: 'edit', arguments: { n: i } }] }; + return { text: 'slice 1 output' }; + }); + const out = await recurse(COMPLEX_TASK, { + provider: sp.provider, + policy: (tool) => (tool === 'edit' ? '[deny: fs.writeScope] out of scope' : true), + }, { + count: 2, + tools: [editTool], + refineLeaf: { sensor: (_r, c) => { if (String(c.task).includes('SLICE 1')) throw new Error('slice-1 sensor boom'); return { pass: true }; } }, + }); + + assert.equal(out.incomplete, true); + assert.equal(out.blocker, 'broken-sensor', 'broken-sensor outranks the FIRST-listed governance-deny sibling'); + assert.match(out.blockerTask, /SLICE 1/, 'and names WHICH sub-task broke'); + assert.equal(out.receipts.blocker, undefined, 'the parent is not accused of its child\'s fault'); + assert.equal(out.receipts.blockerFrom.blocker, 'broken-sensor', 'the inherited label is recorded separately'); + }); +}); + describe('recurse — topology knob & capability-scrub (RC-11 / RC-12 / NB-4)', () => { it('RC-11: maxDepth=1 ⇒ flat fan-out — the child is offered NO spawn tool (no nesting)', async () => { const sp = scriptedProvider(decomposingHandler({ subtask: 'leaf subtask' }));