feat: discipline hardening — verifier isolation, MODE: light triage, structural fail-streak - #3
Open
Tespera wants to merge 3 commits into
Open
feat: discipline hardening — verifier isolation, MODE: light triage, structural fail-streak#3Tespera wants to merge 3 commits into
Tespera wants to merge 3 commits into
Conversation
…ling groups on a shared event With two hooks registered on the same event (e.g. two PostToolUse/Bash entries), processing the second spec re-pruned the event list and silently dropped the group just added for the first. Latent with a single spec per event; bites as soon as a second one exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…boration, opt-in acceptance replay
Closes the honor-system gap in evidence-on-close. Today the Close Guard
checks that an '-- evidence:' note EXISTS and is substantive — but the note
is self-reported: a model can write 'evidence: pytest 21/21' without ever
running pytest, and the guard passes. The most-needed check (did the
acceptance actually run?) was the one thing the hooks couldn't see.
Three pieces:
1. Evidence Logger (new PostToolUse/Bash hook, passive): appends every
command's real outcome {cmd, exit, output tail} to .fable/evidence.jsonl.
Evidence is machine-written from tool results, not typed by the model.
Fabricating now requires a visible act (hand-editing the log) instead of
a plausible sentence. Rotates at 512KB; records even while PAUSED.
2. Citation corroboration (Close Guard): a checked card that cites a
`command` in its evidence note must have a successful run of that
command in the log — never ran, or never exited 0, blocks the stop.
Prose-only evidence (screenshots etc.) keeps the existing substantive-
string rule; projects without a log (pre-logger) are untouched (fail-open).
3. Acceptance replay (opt-in, 'REPLAY: on' ledger line): before the round
may end, re-run each cited acceptance — 'passed once' is not 'still
passes'; a later card silently breaking an earlier one is caught at the
door. Budgeted (30s/cmd via FABLE_REPLAY_TIMEOUT, 120s total) so a heavy
suite can't hang the stop; off by default because replay costs real time.
All existing invariants preserved: fail-open everywhere, .fable/ opt-in,
loop-safe, stdlib only. Tests: tests/test_evidence.py (14 cases) +
test_install updated for the fifth hook; existing suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…structural fail-streak Three quality mechanisms, one theme: make the existing levers bind where they currently rely on willpower. 1. Verifier information isolation (VERIFIER_PROMPT.md + SKILL.md): a verifier that reads the worker's transcript, summary, or claimed status grades the *story*, not the work — contaminated context is why self-critique underperforms. Hard rules for the dispatcher: verifier gets ONLY the SPEC excerpt + artifact; no expected verdict; 2-3 *different-lens* verifiers over N identical ones; never weaker than the implementer. 2. MODE: light (ledger directive): triage for small rounds. Full ceremony (SPEC + cards + evidence + replay) earns its cost on long, multi-file, hard-to-reverse work; on a small immediately-verifiable round it is overhead the model learns to game. Light keeps the honesty rules (evidence-on-close, corroboration, ceiling, fail-streak) and drops the ceremony (design gate, open-cards-block-stop). Choosing the weight is part of the discipline. 3. Structural fail-streak: the attribution-ladder reminder stays advisory at 3, but at 6 consecutive failures insight has demonstrably not worked — every further failure now exits 2 with a demand to stop retrying, distill '-- tried: <ruled out>' into the card, and restart from a fresh context. The note (or a success) resets the guard: the distillation is the exit, so the lesson survives the context that learned it. Grinding in a failure-polluted context makes models dumber; this makes the documented restart-fresh rule mechanical instead of aspirational. Invariants preserved: fail-open, .fable/ opt-in, loop-safe, stdlib only. Tests: tests/test_discipline.py (13 cases); all existing suites green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three mechanisms, one theme: several of fable-mode's best rules currently bind by willpower — this makes them bind by structure.
1. Verifier information isolation (docs/template — the missing half of lever 3)
Adversarial self-check only works when the verifier cannot inherit the worker's beliefs. A verifier that reads the worker's transcript, summary, or claimed status ends up grading the story, not the work — the same context contamination that makes self-critique weak, smuggled in through the prompt. That's why 'fresh-context verifiers beat self-critique' sometimes doesn't hold in practice: the context was fresh, the information wasn't.
VERIFIER_PROMPT.mdnow carries hard rules for the dispatcher: the verifier receives ONLY the SPEC excerpt + the artifact — never the worker's notes, explanations, claimed evidence, or an expected verdict; prefer 2-3 verifiers with different lenses (correctness / edges / integration) over N identical refuters — diversity catches failure modes redundancy can't; never a verifier weaker than the implementer. SKILL.md lever 3 states the principle in one line.2.
MODE: light— triage as a first-class ledger directiveFull ceremony (SPEC + cards + evidence + replay) earns its cost on long, multi-file, hard-to-reverse work. Applied to a small, immediately-verifiable round it is pure overhead — and worse, it trains the model to game the format (hollow cards, ritual evidence). The existing states don't quite cover this: idle still demands a live card for any detailed fan-out, and PAUSED turns the honesty rules off entirely.
MODE: lightkeeps what must never be off (evidence-on-close, citation corroboration, model ceiling, fail-streak) and drops the ceremony (design gate, open-cards-block-stop). One auditable line, same pattern as ROUTING/TIER/PAUSED. Choosing the round's weight becomes part of the discipline — maximum ceremony everywhere is not rigor, it's noise.3. Structural fail-streak — the restart-fresh rule, mechanized
SKILL.md already says it: grinding in a context stuffed with failed attempts makes models dumber — restart fresh. But the enforcement is an advisory reminder, and a model deep in a grind is precisely the one that ignores advice: each failure makes the next fix attempt more tempting, not less.
Two rungs now: at 3 consecutive failures, the attribution ladder (advisory, unchanged). At 6, insight has demonstrably not worked, so the reset becomes mechanical: every further failing command is answered with exit 2 — stop retrying; distill
-- tried: <hypotheses ruled out, dead ends>into the current card; restart the card from a fresh context that reads only SPEC + LEDGER. Writing the note (or one passing command) resets the guard. The-- tried:note is the point: it forces the grind's one real product — knowing what doesn't work — out of the doomed context and into external memory, so the fresh context doesn't re-walk the same dead ends. Escalation only via more failures, so a healthy debug loop (fail, learn, fix, pass) never hits it.Invariants & tests
Fail-open everywhere,
.fable/opt-in, loop-safe, stdlib only, POSIX.tests/test_discipline.py— 13 cases: light keeps evidence enforcement and ceiling while dropping gate/stop-block, full-mode regression; streak advisory under 6, hard at 6+,-- tried:resets, success resets, PAUSED off. All existing suites green.🤖 Generated with Claude Code