fix(mp-75): the FP rate excludes trials that could not have fired — and 0/8 is withdrawn - #46
Merged
Merged
Conversation
[M] The harness reported verdicts only, so a scenario the model answers
identically every run scored `unchanged` and was counted `clean`. The
temperature-0 held-out suite therefore published `0/8 = 0%` -- which reads as
"the engine did not cry wolf" and means "there was nothing to cry wolf about".
[M] `stats.py:128-129` early-exits at p=1.0 when the two sides are
distributionally identical, so such a trial scores `unchanged` REGARDLESS of
any change to the engine. [M] sweeping ALPHA with all three effect-size floors
forced to 0.0, an invariant pair stays `unchanged` until ALPHA >= 1.0: it
discriminates exactly one calibration.
A trial is now excluded iff the engine reports `unchanged` at confidence
exactly 1.0. Under ADR-0001 that confidence is min(p) across every signal, so
it holds precisely when no channel measured any effect. SOUND BY CONSTRUCTION:
a flagged verdict never carries `unchanged`-confidence, so this can never
remove a false positive from the rate. Excluded counts are published beside the
rate, never folded in, with the one-sided 95% upper bound and a loud banner
when nothing was scored. ADR-0022.
THE FIRST IMPLEMENTATION OF THIS WAS WRONG AND fp-guardian BLOCKED IT. It
excluded on PER-SIDE variance ("each side is unimodal, so nothing varied"),
which is not the engine's firing condition: [M] two internally-invariant sides
that DIFFER are the engine's lowest-p, highest-confidence configuration. That
predicate scored 1 of 7 cases correctly, excluded `regression` at conf
0.992-0.996 on all four gating channels, and dropped ~9.1% of tool-channel FP
mass at q=0.5 -- removing the MOST confident alarms while keeping the marginal
ones. It deleted exactly what the metric exists to count. The root cause was a
side-local reading of a pairwise early-exit; the same misreading was in the
BACKLOG row's own prescription and in a `[M]` citation, both corrected here.
The decision is now `fp_outcome()` + the `FP_OUTCOMES` table -- data, not
control flow. [M] this was forced: with the loop inline, replacing the
exclusion with `if False:` deleted MP-75's entire point and left 271 tests
green. [M] 5 mutants now caught, including counting no-effect trials in the
denominator (the original bug) and excluding flagged verdicts.
[M] Argument-only jitter is excluded today, because no gating signal on main
reads tc.arguments (argkey.py is on the unmerged MP-04 branch). Correct now --
such a trial cannot fire -- and it means spending a key on
examples/calibration/arg_*.json before MP-04 merges buys a run of no-effect
rows. Recorded in ADR-0022.
docs/fp-measurement.md: the headline `0 false alarms in 8 scored trials` and
the Phase-0 DoD's "measured 0% false-positive rate" are WITHDRAWN. The latter
already contradicted the same document's rule four paragraphs above ("reported
as 0/8 and never as 0%"). The engine did not change; the accounting did.
[M] 281 passed, ruff + black clean, mypy 8 (unchanged baseline). No live call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Second fp-guardian review returned BLOCKED on 4 conditions. All addressed. C4 THE ARM WAS STILL NOT PINNED -- I had MOVED the untested code, not shrunk it. [M] substituting `classify(r.verdict)` for `fp_outcome(r)` at the call site restored the exact pre-MP-75 accounting (`0/8 = 0%` straight back) and left all 281 tests green, because classify's three return values are all valid FP_OUTCOMES keys. Extracting `fp_report()` killed that -- and then reverting the interval to the closed form, and deleting the interval line, both survived for the same reason one level up, so `fp_summary()` came out too. Each extraction was forced by a live mutant. The lesson is worth more than the code: pinning a pure function does not pin its caller. [M] 7/7 mutants now die, including M1/M15/M16/M17/M19 which all survived the previous round. C5 THE PUBLISHED INTERVAL WAS WRONG FOR EVERY k > 0. `1 - alpha**(1/n)` is the Clopper-Pearson bound only at k=0. [M] at 1/8 it printed 31.2% against a true 47.1%; at 4/8 it printed 31.2% -- an upper bound BELOW the 50% observed rate, a self-contradicting number about the north-star metric, in the direction that flatters it. This harness exists to be aimed at tool-using scenarios at temp>0, which is exactly where k > 0 is expected. Now a real bisection bound, verified against 0/8=31.2%, 1/8=47.1%, 2/8=60.0%, 4/8=80.7%, 1/6=58.2%, plus a property test that the bound never falls below the observed rate for any k<=n<=12. C3 `0/8` WAS STILL THE HEADLINE OF README.md -- withdrawn there (5 places), in docs/fp-measurement.md's results block, and in diff/__init__.py:75 where it was cited as justification for MIN_SEMANTIC_DELTA. That floor's evidence is restated as TWO independent conditions, not three: the held-out re-validation contributed 0 scored trials. The constant does not move; only the claimed evidence for it. Also repaired a blockquote I had mangled. C4b THE LABEL AND ADR OVERSTATED. `unchanged` at confidence 1.0 is strictly BROADER than "no effect measured": [M] the mean statistic is one-sided, so a refusal rate 100%->0% is a 100-point swing at p=1.0; and at N=5 the tool channel's p is exactly 1.0 across the whole |i-j|<=1 band -- 16 of 36 cells, 10 with genuinely differing sides. All have p=1.0 > ALPHA so the exclusion stays sound, but the label now reads `could not have fired at ALPHA=0.05`, and ADR-0022 records that `scored` is ~half a naive expectation, so a calibration run needs about double the trials for a given interval. Also: classify() is now total over DiffVerdict (a fifth member silently became a scored clean trial -- the ADR-0018 failure shape); the coverage bucket moved into FP_OUTCOMES so fp_tally has one source of truth; provider errors are counted and printed instead of vanishing from a banner that claimed "every scenario was unmeasured or invariant"; a test pins that the recall arm never adopts the FP arm's exclusion (an ADR-stated invariant with zero coverage). Filed MP-77: nothing tests calibration-vs-held-out disjointness, and MP-04 is about to fit an argument floor on the set the FP run now targets. [M] 299 passed, ruff + black clean, mypy 8 (unchanged baseline). No live call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The `0/8` withdrawal corrected one row of a four-row table and left the identical defect in the row beneath it. [M] I re-scored the four "noisy-but-equivalent" golden pairs (`tests/test_diff.py:111-119`) through `diff_scenario`: ALL FOUR return `unchanged` at confidence 1.0000, so the published `0/4` is also `0/0`. A reviewer checking the row below the withdrawal, with no key and one Python call, would have found the error I had just publicly withdrawn. The whole table is now re-scored or explicitly marked un-re-scored; rows 3-4 are flagged as not quotable as an FP rate. Other surviving assertions of the withdrawn claim, all corrected: `.github/modelpin.yaml` (the "<=31% at n=8" arithmetic, verbatim, in the repo's own CI config), `examples/report-suite/README.md`, `examples/calibration/ README.md`, `scripts/calibrate_thresholds.py`, `tests/test_report_suite.py`. THREE CLAIMS I WROTE MYSELF WERE WRONG: - "Re-run today, the same command prints 0/0" -- no run happened. It was RE-SCORED from recorded verdicts, no API call. Claiming a measurement that did not occur is the ADR-0009 sin, in a paragraph about honest measurement. - "the floor rests on TWO conditions" -- itself an overclaim. [M] the two calibration runs share their 6 scenarios and 6 perturbation strings, differ only in the CANDIDATE model, and neither result file records which judge arbitrated (`_calibration_*.json` store only model/runs/rows; `--judge` defaults to gpt-4o-mini). Since this floor gates the judge's own output, the judge is the factor that would have had to vary. It rests on ONE condition. - "equivalent pairs land at 0.0, changes at >=0.8, empty gap around 0.5" -- those are the SELF-JUDGE numbers, which an adversarial audit already demoted as circular. [M] on the independent-candidate run `explain_concept` scores 0.20 equivalent and 0.60 changed, and `define_term`'s CHANGED pair scores 0.0. Real gap 0.20-0.60. My previous commit made this newly load-bearing by citing the labeled set as "the independent evidence" two lines above numbers that were not the independent set's. Also: the detection paragraph claimed a resisted prompt-injection was "correctly left unchanged" -- [M] the harness scores that a MISS (2 of 3, not 2 of 2) and citing it re-imports on the recall side the exact error MP-75 removed on the FP side. Detection now carries its N and its interval, since applying maximal rigor to one arm and none to the other is its own defect. "structural floors are FP-validated by the held-out suite" was false twice over ([M] 0 scored trials, AND at N=5 the floors are inert -- the p-gate is strictly stricter, they first bind at N=9/11/12). Phase-0 "complete" and `v0.1.1` and "201 tests" were stale (now 0.1.2 / 299). CHANGELOG gains the MP-75 entry: `ops/` is gitignored, so ADR-0022 is not a citable authority for a reader and the README no longer pretends it is. Filed MP-78: the README overstates the Drift Map, which MP-75 just made the ONLY remaining detection evidence. [M] engine diff is comment-only; no threshold moved. 299 passed, ruff + black clean. No live API call. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… [M]
fp-guardian CLEARED on the third pass. [M] direction verified exhaustively:
over 24 verdict x confidence cells exactly ONE moves (unchanged@1.0: clean ->
no-effect); across 20,000 random tallies the new rate was never lower than the
old and the numerator was identical every time. No case loses a false positive.
Engine is AST-identical to main -- the only change under modelpin/ is a comment.
Its 45-mutant battery left 9 survivors. Four were real and are now dead:
- M43/M44: `main()` is the CALLER now, and it was not pinned. Inlining
`fp_tally([classify(...)])` there restored the pre-MP-75 accounting AGAIN
(299 green); replacing the summary print-loop with `pass` made every
published number vanish (299 green). That is the FOURTH consecutive time an
extraction only MOVED the untested code one level up. Closed with two grep
tests over main()'s source -- the same technique as the recall-arm guard,
because main() needs a provider and ADR-0006 forbids the live call.
- M14: `classify`'s `raise` is unreachable today, which is exactly why nothing
noticed it being softened back to a `return "clean"` fallthrough.
- M22: the exclusion label could be reverted to "no effect measured" freely.
A FALSE [M] I PUBLISHED, in the commit whose thesis was "my published claims
were wrong": I wrote "neither result file records which judge arbitrated". [M]
Both do. `examples/calibration/results/result-{independent-judge,selfjudge}
.json` each carry `"judge": "gpt-4o-mini"`. I had read `scripts/_calibration_*
.json` -- working files, not the committed artifacts -- and cited those wrong
filenames publicly. The conclusion is UNCHANGED and now better supported: the
judge provably did not vary, so "one condition" holds a fortiori.
AND THE SURVIVING EVIDENCE WAS ITSELF ON THE OLD ACCOUNTING. I published "0
false positives in 6 pairs, 95% upper bound 39.3%" -- [M] 5 of those 6
equivalent pairs return p=1.00 and could not have fired. The honest figure is
0/1, upper bound 95.0%. The self-judge run scores 0 trials outright. That is
the same error MP-75 exists to fix, one row below where I had just fixed it.
Other guardian findings, all applied: the operator-facing line and
`measurable()`'s docstring still claimed "measured no effect on any channel" --
[M] false, golden pairs 3 and 4 (tests/test_diff.py:117-119) have genuinely
different tool distributions and still score p=1.00 everywhere. The label now
says `p=1.00 everywhere; could not have fired`. An all-provider-error run was
told to go fix its temperature; it now gets its own diagnosis. `or (None, None,
None)` replaced with an explicit None check. ADR-0022 records that the
predicate is really `min(p) >= 0.9995` (the engine rounds to 3 places), and its
"all pure, all pinned" consequence is corrected to the four-level story.
Filed: MP-78 (README overstates the Drift Map -- now the only detection
evidence left), MP-79 (the RECALL arm has ZERO behavioural coverage: `detected
+= 1` leaves 299 green, so a dead engine would report 100% detection), MP-80
(the semantic sweep is flat 0.1-0.9 -- recorded so nobody later reads that as
slack; ADR-0002 governs and the floor stays).
[M] 302 passed, ruff + black clean, engine comment-only. No live API call.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samarthputhraya
added a commit
that referenced
this pull request
Aug 25, 2026
… of 39 die The detection arm was inline in main(), which needs a live provider, so no test reached it. [M] bug-reproducer at 58c87dc: eight of nine mutants left the whole 302-test suite green -- `detected += 1` (the harness prints Detection: 3/3 for an engine that flagged nothing), `caught = True`, deleting the ADR-0018 abstention branch, and replacing the entire loop body with `continue`. The lone kill was a source grep, and it landed on the mutant with the LEAST effect on the numbers. Since #46 withdrew the false-positive claim (ADR-0022), detection is the only half of the DoD the project still asserts, and it was the undefended half. Extracted symmetric to what MP-75 did to the FP arm: RECALL_OUTCOMES -> recall_outcome -> recall_tally -> recall_report -> recall_summary, plus a shared module-level build_row(). main() now computes nothing. The two arms share only the key "unmeasured", so feeding one arm's decision to the other's table raises KeyError rather than scoring a plausible wrong number -- that is what defeated the FP arm's equivalent mutant, where classify() returned three strings that were all valid FP_OUTCOMES keys. [M] mutation-sentinel, 39-mutant battery re-cut against this tree: 38 killed. The survivor deletes a source comment. Every original defect class is closed; the original nine anchors cannot be re-run at all, because the extraction deleted every line they mutate. Three of those kills close a hole that was SYMMETRIC in the false-positive arm and older than this change: deleting either arm's per-scenario print loop, or discarding its lines at the call site, left the suite green. The lines' content was pinned; their consumption was not. The second is a NameError that would crash a paid run at the very end. fp-guardian BLOCKED this twice on text, both times correctly: - The MISSED note asserted "the perturbation did change the behaviour". It does not -- the injection changes the INSTRUCTION, and whether behaviour changed is what the arm measures. [S] docs/fp-measurement.md:74: decline_pii, 1 of the 3 entries in PERTURBATIONS, returned `unchanged` on the run of record because the model RESISTED it. Printing "real failure to detect" over a correct true negative is standing pressure to loosen MIN_SEMANTIC_DELTA, and MP-80 records [M] that the semantic sweep is flat from 0.1 to 0.9. The accounting is unchanged; the claim is not. ADR-0023 records the corrected rationale, which ADR-0022 itself got wrong. - A follow-up row cited as MP-80 did not exist; MP-80 is the semantic-floor row, so the citation pointed a future reader at a floor-LOWERING discussion. Now MP-82. The engine did not change: `git diff -- modelpin/` is empty and ALPHA / MIN_TOOL_TVD / MIN_REFUSAL_DELTA / MIN_SEMANTIC_DELTA / DEFAULT_RUNS are byte-identical to main. [M] fp-guardian compared the new arm against the old over all 4,913 row-triples of {None} u DiffVerdict x {0.0, 0.5, 0.9995, 1.0}: zero divergences in (detected, checked, unmeasured), and the FP arm is bit-identical. Three source-slicing guards keyed on printed banner text, so any copy-edit would ValueError all three -- one of them MP-75's round-4 FP call-site protection. They now key on [ARM:FP] / [ARM:RECALL] comment markers, with a test that each occurs exactly once (introducing them, the marker text appeared in its own explanatory comment and truncated the FP slice to 40 characters). [M] 332 passed (was 302) * 91% coverage * ruff/black clean * mypy 8, unchanged. Follow-ups filed, not absorbed: MP-81 (docs/fp-measurement.md publishes 2/2 where the harness computes 2/3 -- README.md:227 is already correct and holds the two sentences to copy), MP-82 (the arm prints a bare fraction while README.md:231 already publishes the 95% lower bound: 13.5% at 2/3), MP-83 (ADR ids in public copy are unresolvable -- /ops/ is gitignored). Refs #47 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samarthputhraya
added a commit
that referenced
this pull request
Aug 25, 2026
`docs/fp-measurement.md:68,76` published "Detection: every perturbation that actually changed behavior was caught" and "2/2 real behavior changes were flagged". [M] The harness scored that same run 2/3, before and after MP-79. The doc's own table lists three perturbations and shows `decline_pii` as `unchanged`; `recall_outcome` maps that to a MISS. The third was removed from the denominator by argument -- the model resisted, so behaviour did not change -- which is exactly the exclusion ADR-0023 rejects, and for its stated reason: a perturbed pair reading `unchanged` is EITHER a blind engine OR a resisting candidate, this arm cannot tell them apart, and one permitted to exclude on that basis lets a dead engine post 0/0. Since #46 withdrew the FP claim this was the project's only surviving quantitative DoD claim -- and MP-79 had just made the tool print `2/3` plus a MISSED note contradicting it, in the same sdist. What changed: - the section publishes what the harness prints: 2/3, the summary block verbatim, and [M] the 95% one-sided lower bound of 13.5%. Copied from `README.md:227,231`, correct since #46's claims-auditor pass. - the resistance reading is demoted to a labelled `> **Corrected**` note, the house pattern the three MP-75 withdrawals on this page already use. - the Methodology bullet stops calling the injection "a controlled behavior change" -- ADR-0023 consequence 3, same defect one section up. [M] Two guards tie the doc to the function whose output it quotes, deriving the tally from the doc's own table so they survive PERTURBATIONS growing (MP-82). Four mutants, each leaves the suite RED: restore the withdrawn 2/2 claim in live prose 1 failed hand-adjust the quoted block 2/3 -> 3/3 1 failed drop the decline_pii row from the table 1 failed reword recall_summary, leave the doc stale 2 failed The last is drift in the other direction -- nothing previously connected this document to the harness, which is how it stayed wrong across two ADRs. [M] No number the tool computes changed: `git diff c76d4d2 -- modelpin/ scripts/` is empty. 334 passed (was 332); ruff clean; black 55 unchanged; mypy 8 in 4 files, baseline. Gates PENDING, not yet run: claims-auditor (public claim) + fp-guardian (restates ADR-0022's invariant on a public surface). Refs: #48, ADR-0022, ADR-0023 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samarthputhraya
added a commit
that referenced
this pull request
Aug 25, 2026
[M] The rebase landed this stack on a tree it had never been tested against, and main had moved twice underneath it (#45 added tests/test_baseline_scenario_collision.py, #46 squash-landed MP-75). `pytest --collect-only` now reports 386, not 382, and tests/test_recall_arm.py::test_the_readmes_test_count_is_the_real_one shells out to that command -- so the README's claim was the one thing the rebase broke. Taken from a real --collect-only, never by hand: that number drifted 42 across several releases before MP-81 pinned it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
samarthputhraya
added a commit
that referenced
this pull request
Aug 25, 2026
…ogle adapter can bill Vertex (#50) * test(mp-79): pin the recall arm -- 8 of 9 mutants survived it, now 38 of 39 die The detection arm was inline in main(), which needs a live provider, so no test reached it. [M] bug-reproducer at 58c87dc: eight of nine mutants left the whole 302-test suite green -- `detected += 1` (the harness prints Detection: 3/3 for an engine that flagged nothing), `caught = True`, deleting the ADR-0018 abstention branch, and replacing the entire loop body with `continue`. The lone kill was a source grep, and it landed on the mutant with the LEAST effect on the numbers. Since #46 withdrew the false-positive claim (ADR-0022), detection is the only half of the DoD the project still asserts, and it was the undefended half. Extracted symmetric to what MP-75 did to the FP arm: RECALL_OUTCOMES -> recall_outcome -> recall_tally -> recall_report -> recall_summary, plus a shared module-level build_row(). main() now computes nothing. The two arms share only the key "unmeasured", so feeding one arm's decision to the other's table raises KeyError rather than scoring a plausible wrong number -- that is what defeated the FP arm's equivalent mutant, where classify() returned three strings that were all valid FP_OUTCOMES keys. [M] mutation-sentinel, 39-mutant battery re-cut against this tree: 38 killed. The survivor deletes a source comment. Every original defect class is closed; the original nine anchors cannot be re-run at all, because the extraction deleted every line they mutate. Three of those kills close a hole that was SYMMETRIC in the false-positive arm and older than this change: deleting either arm's per-scenario print loop, or discarding its lines at the call site, left the suite green. The lines' content was pinned; their consumption was not. The second is a NameError that would crash a paid run at the very end. fp-guardian BLOCKED this twice on text, both times correctly: - The MISSED note asserted "the perturbation did change the behaviour". It does not -- the injection changes the INSTRUCTION, and whether behaviour changed is what the arm measures. [S] docs/fp-measurement.md:74: decline_pii, 1 of the 3 entries in PERTURBATIONS, returned `unchanged` on the run of record because the model RESISTED it. Printing "real failure to detect" over a correct true negative is standing pressure to loosen MIN_SEMANTIC_DELTA, and MP-80 records [M] that the semantic sweep is flat from 0.1 to 0.9. The accounting is unchanged; the claim is not. ADR-0023 records the corrected rationale, which ADR-0022 itself got wrong. - A follow-up row cited as MP-80 did not exist; MP-80 is the semantic-floor row, so the citation pointed a future reader at a floor-LOWERING discussion. Now MP-82. The engine did not change: `git diff -- modelpin/` is empty and ALPHA / MIN_TOOL_TVD / MIN_REFUSAL_DELTA / MIN_SEMANTIC_DELTA / DEFAULT_RUNS are byte-identical to main. [M] fp-guardian compared the new arm against the old over all 4,913 row-triples of {None} u DiffVerdict x {0.0, 0.5, 0.9995, 1.0}: zero divergences in (detected, checked, unmeasured), and the FP arm is bit-identical. Three source-slicing guards keyed on printed banner text, so any copy-edit would ValueError all three -- one of them MP-75's round-4 FP call-site protection. They now key on [ARM:FP] / [ARM:RECALL] comment markers, with a test that each occurs exactly once (introducing them, the marker text appeared in its own explanatory comment and truncated the FP slice to 40 characters). [M] 332 passed (was 302) * 91% coverage * ruff/black clean * mypy 8, unchanged. Follow-ups filed, not absorbed: MP-81 (docs/fp-measurement.md publishes 2/2 where the harness computes 2/3 -- README.md:227 is already correct and holds the two sentences to copy), MP-82 (the arm prints a bare fraction while README.md:231 already publishes the 95% lower bound: 13.5% at 2/3), MP-83 (ADR ids in public copy are unresolvable -- /ops/ is gitignored). Refs #47 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(mp-81): publish 2/3 -- the doc had hand-adjusted the harness to 2/2 `docs/fp-measurement.md:68,76` published "Detection: every perturbation that actually changed behavior was caught" and "2/2 real behavior changes were flagged". [M] The harness scored that same run 2/3, before and after MP-79. The doc's own table lists three perturbations and shows `decline_pii` as `unchanged`; `recall_outcome` maps that to a MISS. The third was removed from the denominator by argument -- the model resisted, so behaviour did not change -- which is exactly the exclusion ADR-0023 rejects, and for its stated reason: a perturbed pair reading `unchanged` is EITHER a blind engine OR a resisting candidate, this arm cannot tell them apart, and one permitted to exclude on that basis lets a dead engine post 0/0. Since #46 withdrew the FP claim this was the project's only surviving quantitative DoD claim -- and MP-79 had just made the tool print `2/3` plus a MISSED note contradicting it, in the same sdist. What changed: - the section publishes what the harness prints: 2/3, the summary block verbatim, and [M] the 95% one-sided lower bound of 13.5%. Copied from `README.md:227,231`, correct since #46's claims-auditor pass. - the resistance reading is demoted to a labelled `> **Corrected**` note, the house pattern the three MP-75 withdrawals on this page already use. - the Methodology bullet stops calling the injection "a controlled behavior change" -- ADR-0023 consequence 3, same defect one section up. [M] Two guards tie the doc to the function whose output it quotes, deriving the tally from the doc's own table so they survive PERTURBATIONS growing (MP-82). Four mutants, each leaves the suite RED: restore the withdrawn 2/2 claim in live prose 1 failed hand-adjust the quoted block 2/3 -> 3/3 1 failed drop the decline_pii row from the table 1 failed reword recall_summary, leave the doc stale 2 failed The last is drift in the other direction -- nothing previously connected this document to the harness, which is how it stayed wrong across two ADRs. [M] No number the tool computes changed: `git diff c76d4d2 -- modelpin/ scripts/` is empty. 334 passed (was 332); ruff clean; black 55 unchanged; mypy 8 in 4 files, baseline. Gates PENDING, not yet run: claims-auditor (public claim) + fp-guardian (restates ADR-0022's invariant on a public surface). Refs: #48, ADR-0022, ADR-0023 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(mp-81): clear both gates -- the correction was itself incomplete claims-auditor and fp-guardian both returned LAND WITH FIXES, and converged independently on the finding that matters: 5d09db3 corrected the Results section and left the SAME defect 60 lines below in the SAME FILE, while its CHANGELOG claimed the file was fully corrected. REQUIRED, both gates: - `## Detection (control)` still called the injections "three controlled regressions" (ADR-0023 consequence 3) and attributed `decline_pii` "(comply -> ... semantic change)" -- the forbidden partition stated as fact, on the one scenario the run shows the model RESISTED, next to an "Expected verdict" line naming the channel a reader would loosen to close the published gap. Rewritten, plus the anti-pressure paragraph fp-guardian asked for: [M] MP-80 records the semantic sweep FLAT from 0.1 to 0.9, so lowering MIN_SEMANTIC_DELTA buys no detection and spends FP headroom. - The [M] on 13.5% had no pasteable command and the tool never prints it (MP-82), so a reader could not reproduce it. Now carries `1 - upper_bound_95(1, 3)`, plus the exchangeability premise both gates flagged: Clopper-Pearson treats the three as exchangeable trials and by construction they are not. - My new Methodology bullet said an abstention "reached no verdict". It reaches `insufficient_evidence`; "never reached a verdict" is the harness's phrase for PROVIDER ERRORS -- wrong word in the sentence whose whole job is keeping the two categories apart. Fixed, and ADR-0022's exclusion added so the asymmetry has something to contrast with. - "Phase-0 DoD: detection met", flat, no interval, no mention of the miss. CROSS-SURFACE, and this commit created it: `README.md:231` published a lower bound FOR the withdrawn 2/2 denominator ("22.4% at 2/2, 13.5% at 2/3"), offering the excluded reading as co-equal on the surface most readers meet first. 5d09db3's CHANGELOG asserted README already carried the corrected framing; :227 did, :231 did not. Both the README line and my claim are fixed. GUARDS -- the first cut read as coverage and provided less than it looked. [M] Three holes, each found by RUNNING the battery, not by reading the tests: positional `split("> **Corrected")[0]` -- re-asserting a withdrawn claim BELOW the note passed (fp-guardian) the ban was one-directional -- `2/2` refused, flattering `3/3` allowed (fp-guardian) still section-scoped after both fixes -- the same withdrawn sentence one line ABOVE the headline passed all 37 (found re-running the battery) Now 7 tests pinning the quoted block, the prose headline, the interval, the scenario set against PERTURBATIONS, and the correction note's existence. [M] Ten mutants, each RED, including the self-consistent forgery fp-guardian built (flip the table row + edit the fraction + drop the NOTE together). [M] What these tests still CANNOT do, now said in their own docstrings rather than left to read as fidelity: no artifact of the detection run is committed, so they pin internal consistency and the scenario set, never that the table matches a run that happened. Filed MP-83. Deliberately NOT fixed: `scripts/fp_measurement.py:54` cites a doc line this commit moved. Fixing it would touch scripts/ and break the [M] that no engine or harness code changed. Filed MP-84b. Not changed, on fp-guardian's authority over claims-auditor's #9: "nothing changed for the engine to see" stays. It is the wording ADR-0023 blesses at README.md:227 and it is disclaimed in the same sentence. [M] git diff c76d4d2 -- modelpin/ scripts/ = 0 bytes. 339 passed (was 334); ruff clean; black 55 unchanged; mypy 8 in 4 files, baseline. Refs: #48, ADR-0022, ADR-0023, MP-83, MP-84b Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(mp-81): state the FP arm's own exclusion where the reader meets it Two gaps left after the gate pass, both in Methodology. fp-guardian F4, second half: the Detection bullet now says "Contrast the false-positive arm, which ADDITIONALLY excludes a trial that could not have fired" -- but the FP bullet three lines above never stated any exclusion, so "additionally" referred back to nothing. ADR-0022's rule now sits in the FP arm's own bullet, where it belongs, and the contrast lands. claims-auditor #13: this page is the public "Full writeup" (README.md:222) and [M] `.gitignore:35` = `/ops/`, so every `ADR-nnnn` it cites is unreadable to the audience it is written for. The arguments are all stated inline already; now the page says so, instead of leaving pointers that look load-bearing and resolve to nothing. Not a new pattern -- [M] CHANGELOG.md already carries 9 such citations -- but this page is the one that leans on them for its central justification. [M] /verify full gate on this tree: 339 passed; coverage 91% / 1541 stmts, no module dropped; ruff clean; black 55 unchanged; mypy 8 in 4 files (baseline); statistical dead zone UNMOVED (binary cannot fire at N=2, tool cannot at N=2 or N=3); offline README quickstart exits 1 with 2 regressions and writes the report. Refs: #48, ADR-0022, ADR-0023 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(mp-81): clear the SECOND gate pass -- I introduced two of these myself Both gates re-reviewed fedf6d2 and converged again. Two of the required findings are defects the FIRST round of fixes introduced. REQUIRED, both gates independently: 1. The FP-arm gloss I wrote -- "a trial where the engine measured no effect on any channel" -- is the exact reading `measurable()`'s own docstring forbids: "That is NOT the same as 'no effect was measured', and the distinction is load-bearing." [M] Golden pairs 3 and 4 have genuinely different tool distributions and are still excluded, because 5 runs a side cannot separate them. Understating the exclusion understates how far the project is from a real FP number -- `scored` is ~half what a naive reading expects. Fixed on docs/ AND README, which carried the same gloss. 2. "A miss here is a false NEGATIVE" -- I removed one forbidden assertion and introduced another. It entails "the behaviour changed and the engine missed it": ADR-0023 consequence 2, in the unflattering direction, and the same sentence fp-guardian blocked from the operator string during MP-79. Now "a miss is never a false alarm -- it is either a false negative or a correct true negative, and this arm cannot tell which", and the phrasing is banned. REQUIRED, fp-guardian: 3. The `3/3` tripwire was a bare substring: [M] unsatisfiable beside the verbatim guard (a legitimate perfect score puts "3/3" inside the harness quote the other guard requires), so its cheapest repair would have been to stop scanning fenced blocks -- silently gutting the banned-claims guard. [M] It also fired on `13/30`, `23/30`, `3/31`; this doc's own next step is ">=30 labeled pairs". Now a state assertion over the doc's tally. [M] A legitimate perfect score turns exactly ONE test red: the deliberate review ADR-0023's falsifier calls for. 4. My blockquote filter stripped EVERY quoted line. [M] fp-guardian inserted a `> **In brief.**` pull-quote under `## Results` carrying all three withdrawn claims, most-read position on the page, 37 green. Only a quote opening with the `**Corrected` marker is exempt now. This is the THIRD iteration of this one guard and every previous fix was real -- recorded in its docstring. 5. The anti-pressure paragraph's [M] was over-scoped. [M] I re-derived the sweep myself rather than trusting the report: flat 0.1-0.9 at recall 4/6 holds for the INDEPENDENT-JUDGE run only; the self-judge run is 5/6. The run is now named, and the headroom clause -- which was [A] riding inside an [M] -- is replaced by the argument that actually supports it: [M] the floors first bind at N=9/11/12 (ADR-0002), where this run cannot see. 6. That same measurement falsified a claim 15 lines below it: "absorbed by the floor + p-gate -- the conservative floor earns its keep". [M] It was not the floor. `explain_concept` (delta 0.20, p=0.50) clears even a 0.1 floor and is stopped by the p-gate alone -- which is WHY the sweep is flat. Corrected in place rather than left standing next to its own counterexample. REQUIRED, claims-auditor -- both stale, both in README, neither guarded: "detection DoD met" -> contradicted the file it links to two words later "299 tests passing" -> [M] 42 short; actual 341 Also: "calibrated" now says what it means (confirmed FP-safe, NOT fitted -- the set cannot discriminate the value); the MP-79 CHANGELOG entry stopped asserting the no-change side of the partition its own MP-81 entry withdraws 35 lines below; line-number citations replaced with quoted phrases. [M] README was guarded for NOTHING. Seven edits each left the suite green, including deleting the withdrawal paragraph outright. Now pinned for the detection fraction, the interval and the withdrawal's existence -- mutants 13 and 14 below. The rest is MP-85. [M] FOURTEEN mutants, each RED, re-run against the FINAL tree, not against the version the gates saw. 9 guards. Filed MP-85 (README claims), MP-86 (Drift Map ships without ADR-0009 provenance), MP-87 (README upgrades the Drift Map's own wording; its one clean pair is our own Unicode bug). [M] git diff c76d4d2 -- modelpin/ scripts/ = 0 bytes. 341 passed; coverage 91% /1541; ruff clean; black 55 unchanged; mypy 8 in 4 (baseline); statistical dead zone unmoved. Refs: #48, ADR-0002, ADR-0022, ADR-0023, MP-83, MP-85, MP-86, MP-87 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(mp-81): clear the THIRD gate pass -- one guard asserted nothing at all fp-guardian returned BLOCK with 6 must-fix items. Every one is real, and the most important is a guard I added in round 2 that could not fail. THE ONE WORTH READING. Round 2 added a ban on "a miss IS a false negative". [M] It was written through a shell heredoc that collapsed `\b` into the BACKSPACE control character, so the pattern shipped with three literal 0x08 bytes and matched nothing, ever. It read as coverage, passed review, and passed its own green suite while asserting nothing. Meanwhile the claim it was written to catch was ALREADY IN THE FILE -- `docs/fp-measurement.md`, 65 lines below its own correction, "recall ... is imperfect (4/6 - the safe failure direction: a miss is a false *negative*, not a false alarm)" -- invisible to a substring scan for two further independent reasons: markdown emphasis (`*negative*`, not the `_negative_` the list held) and a hard line wrap between "is" and "a false". Three defects stacked in one guard. The scan is now whitespace-normalised and emphasis-stripped, and [M] the claim is also FALSE on our own data: of the two misses behind 4/6, `define_term` came back judged fully equivalent at p=1.00 -- indistinguishable from the candidate simply not following the instruction, which is the exact thing ADR-0023 says this arm cannot adjudicate. OTHER REQUIRED: - `## Detection (control)` claimed "anything else, `unchanged` included, scores a miss and is never excluded". FALSE in the LOOSENING direction: an abstention IS excluded (ADR-0018), and this test file's own header records that deleting the `unmeasured` block left 302 green. Fixed, and now guarded positively -- [M] a literal ban was trivially evaded ("scores a miss; is never excluded"), so the guard requires the CITATION in that section. - The blockquote filter did not reset on blank lines, so a pull-quote adjacent to a correction note inherited its exemption. [M] Same bypass, green again one paragraph lower. - "So scored is roughly half ... twice the trials" generalised a cell count over the tool channel's p-grid at N=5 into a calibration budget. [M] This document's own two measured runs say 0 of 8 and 1 of 6 scored -- 6x to unbounded, not 2x. Scoped, with the measured figures beside it, plus the soundness property (a flagged verdict never carries unchanged-confidence, so the exclusion can never remove a false positive). - "well-evidenced first calibration" -- contradicted three ways by this same changeset, including a re-scoring to 0/1 at a 95% upper bound of 95.0%. - A guard docstring claimed a `3/3` ban that round 2 had moved to the tripwire. [M] FIFTEEN mutants, each RED, run against the final tree in one process (scratchpad/battery.py): every withdrawn claim above, below and adjacent to the note; the emphasis+wrap form; block, headline, interval, table row, vocabulary, ADR-0018, the note's deletion, a stale harness reword, both README claims, and the full self-consistent forgery with the bound recomputed. Two apparent survivors were battery bugs, not holes, and are reported as such: `is **withdrawn**` occurs twice in README so a single-replace left one, and a `scripts/` diff was line-endings only. [M] Also swept every touched file for control characters after finding the backspaces: none remain. [M] git diff c76d4d2 -- modelpin/ scripts/ = 0 bytes. 342 passed; coverage 91% /1541; ruff clean; black 55 unchanged; mypy 8 in 4 (baseline); dead zone unmoved. Refs: #48, ADR-0002, ADR-0018, ADR-0022, ADR-0023 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(mp-81): the ban policed grammar, not meaning -- go sentence-level A reword battery against the round-3 guards found 11 survivors. Six matter little; five were the SAME forbidden claim, respelled. [M] The literal ban caught "every miss is a false negative" and let through "misses are false negatives", "a miss here means a false negative", and "the miss is, in truth, a false negative". It was policing grammar, not meaning. My first repair -- a document-wide positive check that the disclaimer is present -- failed differently and WORSE: the disclaimer can sit in one section while the assertion sits in another, which is precisely the shape of the defect MP-81 started from. It went green on all three rewordings. The rule is now per-sentence: tie a miss to "false negative" and you must disclaim it in the same breath. Any phrasing that does passes; any phrasing that does not is red. [M] All five rewordings now die, and neither compliant sentence false-alarms. Also closed: - [M] "this arm excludes nothing whatsoever, not even an abstention (contra ADR-0018)" went green against the citation-presence guard added in round 3 -- the citation was still there, the claim was still false. Now also refuses "excludes nothing" in that section. - [M] README's test count is pinned to the collected count. That number was 42 short for several releases; it cannot drift again. Correcting it by hand in round 2 was not a fix, it was a reset. [M] SIX MUTANTS STILL SURVIVE and are named in the CHANGELOG rather than left for a reader to find: dropping the 0/1 upper bound or the not-fitted gloss from README; removing the soundness property, the 0-of-8 / 1-of-6 planning figures, or restoring "the conservative floor earns its keep" in the doc; and re-asserting a withdrawn claim INSIDE a `> **Corrected**` note, where quoting it is the whole point and no guard can read intent. Pinning every sentence of a document from a unit test is the wrong shape -- these are disclosed and tracked (MP-85), not papered over. What IS pinned is every number, every fraction, and every claim about what the harness measured. [M] 30 mutants across two batteries against the final tree; 24 red, 6 named survivors. 11 guards. 343 passed; coverage 91%/1541; ruff clean; black 55 unchanged; mypy 8 in 4 (baseline). git diff c76d4d2 -- modelpin/ scripts/ = 0 bytes. No control characters in any touched file. Refs: #48, ADR-0018, ADR-0023, MP-85 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(adr-0024): record the doc-guard scope decision MP-81 decided how far unit tests guard published documents: every number, fraction, interval and claim about what the harness measured is pinned and derived from the code; prose is not; and the resulting gaps are named. That fork will be re-litigated by the next person who hits a false alarm or wants to pin more, so it is a record rather than a habit. ADR-0024 carries the rationale and the six accepted holes; CHANGELOG and NOW.md now point at it instead of restating it. [M] 343 passed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(mp-77): declare fit vs score per scenario, and make it binding The north-star metric is the false-positive rate, so a threshold fitted on the same scenarios a rate is scored on makes that rate in-sample. Nothing tested that. `tests/test_report_suite.py:93` was the repo's only disjointness test and it covered report-suite vs held-out only. The row asked for a calibration-vs-held-out test. [M] Those two are already disjoint (13 ids vs 8), so that test is green on day one and blind. The live leak was INSIDE `examples/calibration/`: its README declared the whole directory a tuning set, while README-arguments.md authored the seven `arg_*.json` files to price a false-positive rate and docs/fp-measurement.md aimed the FP-establishing run at them. One directory, two opposite roles, nothing machine-readable. `examples/roles.json` now declares a role per scenario -- fit / score / public / fixture -- and `tests/test_suite_roles.py` enforces it. `arg_*` is `score`: no threshold may be fitted on it, not MIN_TOOL_ARG_TVD nor any successor. ADR-0025. The guard took five versions. Each was green, reviewed, and believed closed when the next break was found, so the ten breaks are recorded in the module docstring rather than tidied away: - roles enforced on ids, not content -> hash the content - the held-out suite's own role unpinned -> EXPECTED_ROLES - the sweep one directory deep -> rglob - the hash keyed on the raw file dict -> hash the parsed model - ...which failed too, because `Scenario.input` is `dict[str, Any]`: the same `"_note"` key moved one level down beat it -> canonicalise `input` against what the adapters actually read - `assertions: {}` vs absent hashed apart while behaving identically - path -> {roles} let a scenario move between calibration's two entries freely -> EXPECTED_MEMBERS pins membership - content compared fit-vs-others only, so a score set could be laundered into the fixture path -> compare every role pair but public/fixture - the adapter pin matched `repr()` and tested black, not the adapter -> ast - `kind` was hashed but never read, so flipping it bought a byte-equivalent copy a fresh identity -> exclude it, pinned by a test that reopens the exclusion if `kind` ever becomes behavioural Docs corrected where they had drifted or overclaimed: - [M] the held-out suite does NOT "declare no tools" -- 3 of 8 do, as bare strings that normalise to an empty parameters block. Temperature 0 is the reason it cannot score the argument channel. - [M] "Zero scenarios combine tool use with temperature > 0" was falsified when MP-54 landed; the new text would have contradicted it in the same file. - [M] The arg_* trial projection was published wrong twice, both flattering: 3-4 trials via the 2x heuristic fp-measurement.md forbids by name, then 1-2 using only the most favourable of three runs. Pooled across all three runs of record (0 of 8, 1 of 6, 0 of 6 = 1 of 20, 5.0%), seven scenarios project to 0.35 expected scored trials -- the modal outcome is an abstention, not a rate. That is the measured case for MP-89's --repeats. - report-suite's README said the integrity test "only requires" disjointness from the held-out suite. That is now false; it lists what is actually enforced. [M] 356 passed - ruff clean - black clean - mypy 8-in-4 (unchanged baseline). Follow-ups filed: MP-91 (top-level fields still use a denylist, which is how `kind` slipped), MP-92 (pin _RESERVED_FILES). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mp-82): the detection arm publishes the bound the README already published The two arms of scripts/fp_measurement.py were asymmetric in the direction that flatters detection. fp_summary printed its rate AND a one-sided 95% Clopper-Pearson upper bound; recall_summary printed a bare fraction and no interval -- while README.md:233 and docs/fp-measurement.md:131 ALREADY published the detection floor, and the test suite recomputed it on every run to guard them. The interval existed on every surface except the executable, which is the one surface a user runs. [M] The floor is the same helper by complement, 1 - upper_bound_95(misses, checked): 13.5% at the 2/3 run of record, 36.8% at a hypothetical 3/3. Verified exact rather than plausible -- the binomial tail at each published floor is 0.050000. [M] The accounting did not move: 0 divergences in recall_tally, fp_tally, fp_summary, classify, fp_outcome, recall_outcome and upper_bound_95 over an exhaustive old-vs-new differential. Presentation only. Three things the row did not ask for, each forced by evidence: - The bound says what it is a rate OVER. The denominator includes decline_pii, which the model resisted, and ADR-0023 forbids this arm asserting that any perturbation changed behaviour. "the true rate" alone has a weaker antecedent here than in the FP arm, whose preceding line names its rate. - A corrupt tally raises instead of publishing. [M] detected > checked is the one input in the whole space that overstates: 1 - upper_bound_95(-1, 3) reaches neither guard and prints 100.0%, certainty of perfect detection. - No interval when nothing was checked, mirroring the FP arm's gate. [M] The guards were wrong before they were right, which is the real finding. An adversarial pass showed a mutant that keeps the exact bound at checked == 3 and prints the raw point estimate everywhere else passing all 360 tests -- four of five new guards exercised only n=3, which is also len(PERTURBATIONS), so three distinct `n=` mutants coincided there too. And ADR-0022's percentage ban went structurally blind: "100%" in "100.0%" is False, so the substring check stopped seeing every value the new line can produce. Guards now span n = 1, 2, 3, 5, 6, 8, pin the label and both caveat halves and the order as literals so a doc re-copy cannot launder them, and ban any published percentage equal to the observed rate. [M] 15 of 15 mutants die, each with a code-level killer. Docs: the verbatim block is re-cut from what the harness prints; the sentence promising this work is replaced; the CHANGELOG entry no longer misquotes the FP arm, which does print a point percentage. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * feat(mp-104): the Google adapter can bill Vertex AI, not just an API key An AI Studio API key cannot spend Google Cloud credit. [M] Measured on a real account: with Cloud credit available, every current model returned 429 RESOURCE_EXHAUSTED -- "Your prepayment credits are depleted" -- and a key created fresh IN the credited project was refused identically, because prepay is a property of the billing account, not the project. So no amount of key-creation reaches that money. [M] The same project on Vertex answered normally and billed the Cloud credit. [M] Vertex rejects API keys outright ("API keys are not supported by this API. Expected OAuth2 access token"), so this could not be done by swapping a key -- it needs a different credential path. GOOGLE_GENAI_USE_VERTEXAI=true plus GOOGLE_CLOUD_PROJECT selects it, authenticated by Application Default Credentials. Those are the SDK's own documented variable names, so a user already running google-genai needs no Modelpin-specific setup. The API-key path is unchanged and remains the default. ADR-0008 holds either way: Modelpin reads only what the caller put in their own environment, and never hardcodes, stores or ships a credential. ADC is a different SHAPE of the user's own credential, not somebody else's. Guards cover the direction that would break existing users -- a stray or falsey switch must not divert someone to a backend they have no credentials for -- and the two failure messages a user will actually hit: Vertex selected with no project, and Vertex selected with no ADC (which must not read as a missing API key). [M] A key is never passed to Vertex even when both variables are set. [M] Also on the record because it is on-mission: the two doors do not serve the same catalogue. gemini-2.5-flash is 404 "no longer available to new users" on AI Studio and still served on Vertex -- exactly the retirement divergence this project exists to notice, and data/models.json still seeds gemini-2.5-pro. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(mp-104): the Vertex default could not reach a single current model provider-sdk-verifier returned BLOCK on 759e902 with two findings. Both reproduced independently before fixing. 1. THE DEFAULT LOCATION WAS THE WORST POSSIBLE ONE. [M] Free count_tokens against a real project: us-central1 gemini-2.5-flash SERVED global gemini-2.5-flash SERVED us-central1 gemini-3.5-flash 404 global gemini-3.5-flash SERVED Every gemini-3.x id 404s regionally; only the legacy 2.5 family survives there. The comment claimed us-central1 "carries the broadest model coverage" -- exactly inverted. On a tool whose PURPOSE is testing new models, that default cannot reach any model worth testing, and _explain_api_error would have reported it as "model not found - check the id" while the id was right. [M] `global` is also the SDK's own default, so the override was strictly worse than doing nothing. README and CHANGELOG were actively steering users into it and are corrected; the residency trade-off is now stated where the default is documented. 2. THE SDK COULD SILENTLY FLIP THE BACKEND UNDER US. [M] google-genai 2.19.0 reads GOOGLE_GENAI_USE_ENTERPRISE *and* GOOGLE_GENAI_USE_VERTEXAI, and the former WINS; the installed docstring calls vertexai= the "legacy flag". So with USE_ENTERPRISE=true, Modelpin took its API-key branch while the SDK, on an unpinned vertexai=, built vertexai=True against aiplatform.googleapis.com -- posting an AI Studio key to Vertex while Modelpin believed it was on AI Studio. Now both names are read, in the SDK's precedence order, and the API-key path pins vertexai=False so the SDK cannot re-decide. Also fixed, and the more embarrassing one: preflight could not detect missing credentials, and the test asserting it could was hollow. [M] The SDK only calls load_auth when `project` is ABSENT, and this path requires it, so ADC is deferred to the first REQUEST -- Client() succeeds with no credentials at all. preflight() would pass and the run would die mid-replay on a raw DefaultCredentialsError, after the user had been quoted a cost. The old test passed only because its mock raised from Client(), a shape the real SDK never exhibits. ADC is now resolved eagerly and the test raises where the failure actually lives, with an ordering assertion. [M] The suite was also passing for the wrong reason: four Vertex tests called the REAL google.auth.default(), green here only because this machine is logged in, and would have failed on any CI runner while probing the GCE metadata server -- a live call ADR-0006 forbids. Mocked in the fixture; runtime for that file fell 11.07s -> 0.59s, which is the measurement that proves it. [M] 382 pass, and the suite is green with credentials forced absent. [M] gemini-3.5-flash now answers end-to-end through the adapter on the default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(mp-104): merge the duplicate Unreleased '### Added' heading [M] `grep -n '^### ' CHANGELOG.md` showed TWO `### Added` inside one `[Unreleased]` section (lines 9 and 234). This is the same structural defect claims-auditor blocked on the MP-82 entry -- I fixed it for `### Changed` and then reintroduced it for `### Added` in the same session. The Vertex bullet now sits under the existing heading. CHANGELOG.md ships in the sdist, so this is a public surface. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs: sync the test count after rebasing onto main [M] The rebase landed this stack on a tree it had never been tested against, and main had moved twice underneath it (#45 added tests/test_baseline_scenario_collision.py, #46 squash-landed MP-75). `pytest --collect-only` now reports 386, not 382, and tests/test_recall_arm.py::test_the_readmes_test_count_is_the_real_one shells out to that command -- so the README's claim was the one thing the rebase broke. Taken from a real --collect-only, never by hand: that number drifted 42 across several releases before MP-81 pinned it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fp-guardian: CLEARED on the third pass (BLOCKED twice). claims-auditor: findings applied.
[M]302 passed · ruff + black clean ·git diff main -- modelpin/is comment-only, nothreshold moved.
The defect
scripts/fp_measurement.pycounted a scenario as a passed false-positive trial even when theengine measured no effect on any channel — a trial that could not have failed.
[M]diff/stats.py:128-129early-exits atp=1.0when two sides are distributionally identical, sosuch a trial scores
unchangedregardless of any change to the engine.The temperature-0 held-out suite therefore published
0/8 = 0%. Re-scored — not re-run, noAPI call — the same recorded verdicts give
0/0 = n/a, whose upper bound is unbounded.The fix
A trial is excluded iff the engine reports
unchangedat confidence 1.0. Under ADR-0001 thatconfidence is
min(p), so it holds exactly when every channel returnedp = 1.00. Sound byconstruction: a flagged verdict never carries
unchanged-confidence, so the exclusion cannever remove a false positive.
[M]Verified exhaustively: of 24 verdict × confidence cells exactly one moves; over 20,000random tallies the new rate was never lower than the old and the numerator was identical every
time. Monotone-harder, no case where a false positive vanishes.
What this destroys, on purpose
0/8is withdrawn fromREADME.md(headline + 4 more),docs/fp-measurement.md, the repo'sown
.github/modelpin.yaml, twoexamples/*/README.md,calibrate_thresholds.py, and a testdocstring. Phase-0's "measured 0% false-positive rate" is withdrawn — it already contradicted
that document's own rule four paragraphs above.
Three rounds of the same lesson
Every extraction only moved the untested code one level up:
if False:deleted the exclusion — 271 greenfp_outcome()classify(r.verdict)at the call site restored0/8 = 0%— 281 greenfp_report()fp_summary()fp_tally([classify(...)])inlined inmain()— 299 greenmain()Pinning a pure function does not pin its caller, and each extraction creates a new caller.
Two numeric defects the reviews caught
1 - α^(1/n)isClopper-Pearson only at k=0.
[M]at 4/8 it printed 31.2% — an upper bound below the 50%observed rate. This harness exists to be aimed at the surface where k>0 is expected. Replaced
with an exact bisection bound;
[M]verified to 1.11e-16 against an independent 60-digitreimplementation.
MIN_SEMANTIC_DELTA's evidence: three "independent conditions" → one, and that one is0/1.
[M]Both calibration runs record"judge": "gpt-4o-mini"— the judge, which thisfloor gates, provably did not vary. And
[M]5 of the surviving run's 6 equivalent pairsreturn
p=1.00, so it is 0/1 (UB 95.0%), not 0/6 at 39.3%. The threshold does not move.Filed, not fixed here
MP-79 — the recall arm has zero behavioural coverage:
[M]detected += 1leaves 299green, so a dead engine would report 100% detection. That matters more now that detection is the
only half of the DoD still asserted. Also MP-77 (calibration/held-out disjointness untested
before MP-04 fits a floor there), MP-78 (README overstates the Drift Map), MP-80 (the
semantic sweep is flat 0.1–0.9 — recorded so nobody reads it as slack; ADR-0002 governs).
🤖 Generated with Claude Code