feat: machine-written evidence — Evidence Logger hook, citation corroboration, opt-in acceptance replay - #2
Open
Tespera wants to merge 2 commits into
Open
Conversation
…ling groups on a shared event With two hooks registered on the same event (e.g. two PostToolUse/Bash entries), processing the second spec re-pruned the event list and silently dropped the group just added for the first. Latent with a single spec per event; bites as soon as a second one exists. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…boration, opt-in acceptance replay
Closes the honor-system gap in evidence-on-close. Today the Close Guard
checks that an '-- evidence:' note EXISTS and is substantive — but the note
is self-reported: a model can write 'evidence: pytest 21/21' without ever
running pytest, and the guard passes. The most-needed check (did the
acceptance actually run?) was the one thing the hooks couldn't see.
Three pieces:
1. Evidence Logger (new PostToolUse/Bash hook, passive): appends every
command's real outcome {cmd, exit, output tail} to .fable/evidence.jsonl.
Evidence is machine-written from tool results, not typed by the model.
Fabricating now requires a visible act (hand-editing the log) instead of
a plausible sentence. Rotates at 512KB; records even while PAUSED.
2. Citation corroboration (Close Guard): a checked card that cites a
`command` in its evidence note must have a successful run of that
command in the log — never ran, or never exited 0, blocks the stop.
Prose-only evidence (screenshots etc.) keeps the existing substantive-
string rule; projects without a log (pre-logger) are untouched (fail-open).
3. Acceptance replay (opt-in, 'REPLAY: on' ledger line): before the round
may end, re-run each cited acceptance — 'passed once' is not 'still
passes'; a later card silently breaking an earlier one is caught at the
door. Budgeted (30s/cmd via FABLE_REPLAY_TIMEOUT, 120s total) so a heavy
suite can't hang the stop; off by default because replay costs real time.
All existing invariants preserved: fail-open everywhere, .fable/ opt-in,
loop-safe, stdlib only. Tests: tests/test_evidence.py (14 cases) +
test_install updated for the fifth hook; existing suites green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The gap this closes
fable-mode's strongest idea is evidence, not adjectives — but today the evidence note is self-reported. The Close Guard checks that
-- evidence:exists and is substantive; it cannot check that it is true. A model under pressure to close a card can write-- evidence: pytest 21/21without ever running pytest, and every gate passes. The one thing the enforcement layer most needs to know — did the acceptance actually run? — is exactly what it can't currently see. In other words: the gates verify form; this PR starts verifying substance.What this adds
1. Evidence Logger (new PostToolUse/Bash hook, passive). Appends every command's real outcome
{ts, cmd, exit, tail}to.fable/evidence.jsonl. The key property: this file is written by the hook from actual tool results, never typed by the model. Fabricating evidence stops being a plausible sentence and becomes a visible, auditable act (hand-editing a log). Rotates at 512KB. Records even while PAUSED — pausing enforcement shouldn't create evidence gaps.2. Citation corroboration (Close Guard). A
- [x]card that cites acommandin its evidence note must have a successful run of that command in the log. Never ran → blocked. Ran but exited non-zero → blocked. This turns the honor system into a receipts system for the common case (command-based acceptance), while staying fully backward compatible:acceptance:declaration) are not treated as citations.3. Acceptance replay (opt-in via a
REPLAY: onledger line). Before the round may end, re-run each cited acceptance. Rationale: 'passed once' is not 'still passes' — card N's fix routinely breaks card N-2's acceptance, and milestone-only checks miss it. Budgeted (30s/command viaFABLE_REPLAY_TIMEOUT, 120s total) so a heavy suite can never hang the Stop hook; off by default because replay costs real wall-clock time and should be a deliberate choice.Why it fits this project
It's the same philosophy the README already states — discipline enforced by code, not willpower — applied to the layer that was still willpower. All design invariants preserved: fail-open everywhere,
.fable/opt-in, loop-safe viastop_hook_active, stdlib only, POSIX.Tests
tests/test_evidence.py— 14 cases: recorder writes real cmd/exit, inert without opt-in, records while paused; corroboration allows cited+ran, blocks never-ran and ran-but-failed, fail-open without a log, prose evidence unchecked, acceptance-part not treated as citation, PAUSED regression; replay pass/fail/off-by-default.test_install.pyupdated for the fifth hook. All existing suites green (47+27 guards/inject).🤖 Generated with Claude Code