Every repository below does the same thing to a different artefact. Take something a team already trusts, find the question it structurally cannot answer, then measure the size of what it is missing on a corpus where the true answer is known.
Not "this metric is imperfect." A drift report is a function of two samples of input data, and a decision is a function of an input and a boundary, so nothing in the report has an edge to the boundary. That is arithmetic, not an opinion, and it is the shape of all eight.
The numbers below are the uncomfortable ones. Each is produced by code in the repository it belongs to, re-measured by that repository's CI on every push, and matched against the sentence that quotes it.
| Making refusal a first class result | Every tool here can return "not checkable" and mean it. A gate that emits a green tick when the data needed to decide is absent has told its operator something false, so each one carries three verdicts and one of them is a refusal, wired to a distinct exit code. |
| Receipts as a build step | collect_metrics.py re-measures every published figure;
check_numbers.py matches each one against the phrase that quotes it, in both
directions, and fails on a number in the prose that no measurement explains.
1,259 machine checked figures across the portfolio. |
| Negative results that survive review | The most recent finding is about my own tool: the support diagnostic in
offpolicy-gap tracks the failure it was built to predict, and still gives a
team no usable threshold. That is measured, and it is in the README rather than a
footnote. |
| Repository | What it finds | The number that settles it | |
|---|---|---|---|
| 🔎 | chunk-recall-audit | The highest score any retriever could reach on your chunking, before you run one | recall reports 1.0 and the pipeline answers 0.2528 |
| 📉 | drift-decision-gap | Whether the shift you found moved a decision | PSI alone misses 76.7 percent of harmful cases |
| 🚦 | eval-churn-gate | A release that keeps its average and ruins one slice | the mean sees it 0.0 of the time, the partition catches it 1.0 |
| ⏱️ | window-parity | The one skew a windowed feature actually has | the shipped parity check detects 0.0 of it |
| 🎯 | sweep-shrinkage | How much of your sweep's winning score is real | on coin flip labels a sweep reports 0.5202 on average |
| 🧾 | prefix-cost-attribution | A bill that sums to what the server actually spent | per request billing collects 1.8378 times the real cost |
| 🔬 | run-provenance-audit | What a training run actually depends on | of 8 classes of change, the usual record catches 1 of them |
| 🎲 | offpolicy-gap | What an offline estimate of a new policy can be about | the 95% interval covers 0.26, and clipping takes it to 0.0 |
A diagnostic that works and still gives you no threshold
offpolicy-gap builds the best predictor of failure computable from a log alone, shows
it tracks the failure across 12 configurations, then measures that no single cut
separates the good intervals from the bad ones: a deficiency of 0.0126 covers and one
of 0.0104 does not. The negative result is about my own tool.
A comparison that hid the classes it existed to find
In run-provenance-audit the first model comparison rounded to nine significant figures,
which called two real classes of change harmless. What surfaced it was not a failing
test. It was that the arithmetic says a changed summation order cannot leave a float sum
bit identical, so the measurement was wrong rather than the mechanism.
A test fixture that reproduced the bug it was testing for
A perturbation set a batch size to a constant that happened to equal the test configuration's batch size, so the class reported harmless in the suite and material everywhere else. That is the exact failure mode the repository is about, inside its own fixture.
Two correct numbers for the same quantity
A command and an experiment reported different coverage rates for one configuration because each built its own seed stream. Both were reproducible. The receipts pipeline could not have caught it, and did not. What caught it was watching the screen while a demo recorded.
