fix: BA-15 broken-arbiter blockers (sensor + verifier seams)#21
Merged
Conversation
…d, never coerced into a model failure
Adopter feedback (forbidden-zone audit of the close chain) validated against
shipped code with a deterministic pre-fix POC (poc/ba15-broken-sensor.mjs):
Sensor seam (refineLeaf.sensor):
- a THROWING sensor fell through to a bare {incomplete, best:null} —
byte-identical to a provider death, the model's unjudged work destroyed
- a MALFORMED verdict ({}, 'ok', {ok:true}) read as pass:false with
critique:null, so refine burned every attempt re-sending the PLAIN task
(zero feedback), then surfaced converged-shaped as {result, verdict:{}}
Verifier seam (caller opts.evaluate; found by a follow-up probe that
falsified the first-pass "no live laundering path" assessment):
- a THROWING caller verifier CRASHED the whole recurse() run plain-worker
(and laundered to a bare {incomplete} under refineLeaf)
- a GARBAGE return rode out converged-shaped as {result, verdict:{}}
Fix (recurse.js only, refine.js untouched): both caller arbiters are
wrapped — a non-Halt throw or a malformed return (neither boolean `pass`
nor valid tri-state `status`) stops at the FIRST broken close and returns
a labeled {incomplete, blocker:'broken-sensor'|'broken-verifier'} +
receipts.blockerDetail, with `best` preserving the unjudged work (BA-5).
One verifyOrBlock helper covers all five dispatch paths; the default
Evaluator rubric path is never labeled (provider-class faults — narrowest
guard); HaltError passes through clean on both seams. `return await` is
load-bearing at the call sites (the POC's Halt-control arm caught an
un-awaited returned promise escaping the try's catch mid-refactor).
+13 mutation-checked tests; suite 756 pass / 0 fail; typecheck clean.
New public surfaces (MINOR): blocker 'broken-sensor'|'broken-verifier',
receipts.blockerDetail.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf
…back finding #2, F26 borrow) Token-free audit over the poc/ evidence archive + tests: per knob, does ≥1 one-knob pair with a differing OUTCOME CLASS exist, or only mechanism/wiring proof? Citations verified against the archive. - 12 knobs outcome-class proven (retrieval, window, passes, count, partition/workerBudget, synthesize, context, refineLeaf temps/buffer, maxDepth-as-capability, retrieval:'tools', maxConsecutiveDenials) - 2 flagged unproven-for-outcome, confirming the feedback's exact suspects: persona (adoption proven, outcome delta never) and the capability-scrub prompt WORDING (presence mutation-tested, effect never toggle-tested); plus contract, NB-5 wording, maxIterations, maxIdenticalToolErrors noted mechanism-only - Disposition: flags are documentation, not build orders (lean bar); the F26 version+prompt-hash receipt is PARKED (no consumer today) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf
…ona EFFECT, scrub wording NULL) Prereg'd probes (pre-worded readouts, code-scored evidence-only outcomes, harness-confound guards), claude-haiku-4-5, 8 trials/arm: - persona → OUTCOME-PROVEN (poc/audit2-persona-outcome.mjs): through the shipped recurse() (maxDepth:0 single-shot leaf), a risk-averse-SRE stance flipped a borderline deploy judgment SHIP 8/8 → HOLD 8/8. Two prior cells were NO-HEADROOM (haiku security-sweeps any find-the-risk task unprompted) — recorded, not discarded. - capability-scrub prompt wording → MEASURED-NULL, bracketed (poc/audit2-scrub-wording.mjs): "PREFER DIRECT ACTION" on/off with an identical spawn tool — trivial task 0/8 delegation both arms, policy-example-mirroring task 8/8 both arms (3 spawns every trial). The tool contraction carries the scrub, not the prose. Wording retained (one model, two task shapes — toy-fixtures caution); the load-bearing claim is downgraded to measured-null. Zero shipped-code changes. Method note recorded: first cells were NO-HEADROOM — the harness had to create the precondition before any verdict meant anything (§2.C red-team-precondition rule). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf
…BA-5 best, typed arbiter error
A /code-review workflow lost 7 of 16 agents to a spend limit (one whole
correctness finder + six verifiers), so its 3-cleanup-finding verdict sat
on top of un-adjudicated candidates. Recovered all 13 raw candidates from
journal.jsonl and verified them against the code by hand. Five were real:
1. npm run typecheck was FAILING (TS2722/TS18048): the caller-verifier
async IIFE broke `typeof opts.evaluate === 'function'` narrowing inside
the closure. CI-gating and publish-blocking. My earlier "typecheck
clean" claim was false — asserted from a `&& echo` whose echo never
fired. Hoisted to a narrowed const; now verified by exit code.
2. A status-only {status:'satisfied'} verdict burned every iteration:
the advertised contract accepts it but refine.js stops on verdict.pass,
so a satisfied close never stopped and reported passed:false.
runArbiter now derives pass = status==='satisfied' (copy, no mutation).
3. BA-5 violation: refineLeaf's halt/deny/generic-fault branches returned
best:null while the plain-worker path preserved best:out.text. All
three now preserve the last non-empty attempt.
4. A detached JSDoc block — the new helpers were inserted between
recurseRefineLeaf's doc comment and the function, silently dropping
that function's @param typing. Helpers moved above the doc.
5. The magic-string fault channel (the workflow's top-ranked finding):
'broken-sensor: ' prefixes encoded into Error.message and re-parsed by
startsWith, replaced with a typed module-local BrokenArbiterError
(instanceof + .tag/.detail) — no public export, never escapes recurse.
Also: defensive `instanceof HaltError` rethrow in verifyOrBlock, both
arbiter wrappers factored into one runArbiter, five duplicated comments
consolidated onto verifyOrBlock's JSDoc, and the "unjudged work" wording
softened to "last non-empty output" (best can be a prior attempt when the
latest is empty).
+3 regression tests; suite 759 pass / 0 fail; typecheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf
…round 1's own BA-5 fix
Re-ran the code-review workflow on the corrected code; this time all 19
agents completed (round 1 lost 7 to a spend limit), so the verdict is
trustworthy. Four confirmed defects, mutation-proven:
1. The BA-5 preservation missed the terminating attempt.
`lastAttemptText` was captured AFTER the halt/error throws, so a
FIRST-attempt halt discarded its own text and still returned
best:null. Round 1's test only halted on attempt 2 — where a prior
clean attempt had already populated it — so a too-weak test let a
broken fix look fixed. Capture now precedes the throws.
2. recurseScan/recursePartition destroyed a finished result: round 1's
`return await` routed a verifier HaltError into their catches, which
returned best:null, discarding a fully code-counted scan. Both now
preserve the computed result.
3. A child's blocker was laundered by its parent — the
{incomplete, missingSlices} aggregation dropped a child's
broken-sensor/broken-verifier label, so a nested broken sensor never
named itself at the top (BA-15's own failure mode, one level up).
Shared inheritedBlocker now carries it at all three sites.
4. A verify-slot halt after a PASSING sensor clobbered the refineLeaf
receipt to passed:false, reporting a close that did happen as one
that didn't.
Also: verdict shape inspection moved inside runArbiter's try (a throwing
accessor on a returned Proxy escaped untyped), and BrokenArbiterError
carries `cause`.
Accepted behavior change documented: opts.evaluate is now held to its
documented Verdict return; a loose object or bare boolean (always a
contract violation, previously silent) now yields
{incomplete, blocker:'broken-verifier'}. No shim — one would reopen the
laundering hole. Documented open item: instanceof HaltError is
realm-sensitive at 23 pre-existing sites.
+4 mutation-proven tests; suite 762 pass / 0 fail; typecheck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf
…ns this branch introduced
Rounds 1-2 ran /code-review medium (5 then 4 findings); round 3 ran xhigh
(6 finders + gap sweep, clean 40/40 agents) and found 15. The yield rose
because the review got DEEPER, not because the code got worse — worth
stating plainly, since "the last round found less" was my own (wrong)
argument for convergence.
Every finding was reproduced against the unfixed branch by
poc/ba15-round3-validate.mjs BEFORE any edit; every fix is mutation-proven.
Regressions BA-15 itself introduced:
1. The status-only normalization used an object SPREAD (own-enumerable
only), so a class-instance verdict with prototype getters passed the
shape check then came out with status/critique ERASED — the critique
never reached the retry prompt, reinstating the zero-feedback burn for
a shape the validator blessed. Now a descriptor copy onto the same
prototype.
2. A strict-boolean `pass` gate hard-blocked previously-CONVERGING
sensors returning {pass:1}. A PRESENT `pass` now counts whatever its
type — refine has always branched on truthiness, and BA-15 exists to
catch a verdict with NO usable signal, not one answering in another
dialect.
3. Round 1's typecheck fix DETACHED opts.evaluate, so a method-reference
verifier went from this===opts to this===undefined and threw. Re-bound
to opts; the honest limit (this is never the grader instance) is tested
rather than overstated.
BA-15's guarantee failing at boundaries it did not cover:
4. The three HaltError catches never called inheritedBlocker (only the
missingSlices branches did), so a gate tripping after a broken-sensor
child dropped the label.
5. The spawn tool flattened a blocked child into a generic [incomplete],
letting a parent re-delegate into the same broken judge — BA-15's own
spend-burn, one level up.
6. Object.assign(node, inherited) stamped the label onto EVERY ancestor:
receipts accused nodes whose sensor never ran, and one denied child
re-labelled its parent governance-deny. The inherited label now rides
receipts.blockerFrom + a new blockerTask naming the culprit.
7. recurseFanout never got round 2's BA-5 hoist and discarded a computed
reduce on a verify-slot halt.
8. blockerDetail rode only receipts, not the result.
9. Diagnostics: a non-Error throw rendered as [object Object]; a verdict
whose accessor throws was reported as "the sensor threw". Separated
around the await itself, which probes .then and so fires the accessor
before any shape check runs.
10. inheritedBlocker's 'broken-verifier' arm was UNREACHABLE (forChild
strips evaluate) while CHANGELOG and JSDoc asserted it worked. Dead
arm deleted, claims corrected.
All six aggregation/halt paths now route through one incompleteWithBlocker,
which also removes the 3x copy-paste whose next edit would have missed a path.
KNOWN LIMIT, documented not fixed: a halt DURING scanCount still returns
best:null — scanCount throws without surfacing the windows it judged, so
nothing exists at this seam to preserve. The comment implying BA-5 coverage
was corrected rather than left to mislead.
Two of the harness's own checks were CONFOUNDED and caught only because
every fix was mutation-proven, not merely re-run: a malformed Planner stub
ended the run before the path under test (a false REPRODUCED), and a
defensively-rewritten test stopped exercising the crash it existed to catch.
+28 mutation-checked tests; suite 889 pass / 0 fail; typecheck clean by
exit code; both POCs exit 0. Docs synced same commit (CHANGELOG, CLAUDE.md,
context, RLM_PRD).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… the audit log /security on the round-3 changes caught a leak the round-3 fix itself introduced. describeThrown used JSON.stringify on a non-Error throw, and a thrown non-Error is typically a spawn/exec RESULT carrying a full stdout buffer and an env snapshot. blockerDetail rides into receipts, which a wired gate serializes VERBATIM into a plaintext audit log — so this wrote caller secrets to disk. Measured: a 200KB detail string containing an sk-live- token. Same leak class as F16/BA-1's audit-ctx provider strip. Now takes only conventional diagnostic fields (name/code/errno/syscall/path/ status/signal/message), clamped to 200 chars — still names the fault (code=ENOENT) without dumping the payload. Measured after: 25 chars, no secret. Prototype pollution via the new descriptor-copy path was probed separately and is clean. +1 mutation-proven regression test (restore the JSON.stringify dump and it goes red). Suite 890 pass / 0 fail; typecheck clean by exit code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eams)
A faulty ARBITER is named, never coerced into a model failure. A
refineLeaf.sensor or caller opts.evaluate that throws (non-Halt) or returns a
malformed verdict stops the run at the FIRST broken close and returns a
labeled {incomplete, blocker:'broken-sensor'|'broken-verifier'} with
blockerDetail, best preserving the model's unjudged work (BA-5).
Three review rounds (the third at xhigh, a clean 40/40) plus a /security pass;
all findings reproduced against unfixed code before being touched, all fixes
mutation-proven.
MINOR: new public surfaces (blockerDetail/blockerTask on RecurseResult,
receipts.blockerFrom) and documented behavior changes.
Suite 890 pass / 0 fail; typecheck clean by exit code.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adopter feedback ("faulty primitives" / forbidden-zone audit of the close chain) validated against shipped code, then fixed. A caller arbiter that breaks — the
refineLeaf.sensoror a calleropts.evaluateverifier that throws or returns a malformed verdict — is now a named blocker, never coerced into a model failure.Pre-fix pathologies (all reproduced by
poc/ba15-broken-sensor.mjs, deterministic/offline, pre-fix run exits 1):{incomplete, best:null}, byte-identical to a provider death; work destroyed{}) → refine burned every attempt with zero feedback, then surfaced converged-shaped{result, verdict:{}}recurse()run (plain worker) / bare{incomplete}(refineLeaf){result, verdict:{}}Fix (recurse.js only): both caller arbiters wrapped; first broken close stops the loop; labeled
{incomplete, blocker:'broken-sensor'|'broken-verifier'}+receipts.blockerDetail;bestpreserves the unjudged work (BA-5). OneverifyOrBlockhelper covers all five dispatch paths. Default Evaluator rubric path never labeled (narrowest guard).HaltErrorclean on both seams —return awaitload-bearing (POC's Halt-control arm caught the un-awaited-promise regression mid-refactor).Evidence
[Unreleased], CLAUDE.md,bareagent.context.md, README, RLM_PRD dated entry (including the corrected "no live laundering path" first-pass assessment)MINOR bump when released (new public
blockervalues +receipts.blockerDetail).🤖 Generated with Claude Code
https://claude.ai/code/session_016qq7RsZQw1SrqHNM9sDpmf