An off policy evaluation audit, which measures what an offline estimate of a new policy can and cannot be about, shows that a ninety five percent interval around the standard estimator covers the truth a quarter of the time on an ordinary logged bandit, and returns exit 2 rather than a number when the target policy wants actions the log could not have produced.
- Your off policy estimate is not a noisy estimate of your policy's value. It is a precise estimate of a different quantity. A logged bandit with a candidate filter, which is what every ranking system has, leaves 0.395 of the target's mass on actions the logging policy was forbidden to take, and 0.4328 once the support floor is applied. 1731 rows carry some of that unreachable mass. Those rows appear in the sum with the forbidden actions contributing nothing, and the estimator reports what is left as if it were the answer.
- The interval does not warn you. Across 200 independent logs from the same world, the inverse propensity interval covers 0.26 of two hundred, not 0.95. snips reaches 0.785, the doubly robust reaches 0.79, and the clipped estimator, the fix everybody reaches for first, is the worst of the four: clipped covers 0.0, with an interval whose upper end of 0.0188 still excludes nominal. All four exclude 0.95, so this is not a wide interval, it is a wrong one.
- The truth can move underneath all of it without a single number changing. Shift the true reward of the 4 actions removed from the catalogue and every log is byte identical, every importance weight is identical, the support deficiency is 0.328 in every one of them, and the four estimates differ by 0.0. The true value moves by 0.1972, from 0.3368 to 0.534 at the far end, and nothing computable from the log records that it did.
- The diagnostic sees it, and it is not a threshold you can set. Across 12 configurations the support deficiency tracks the coverage at a rank correlation of -0.8311, while the effective sample size manages 0.1905. But no single cut separates the wrong intervals from the correct ones: a deficiency of 0.0126 covers and one of 0.0104 does not. The tool reports both floors and refuses to pretend one number decides it.
Off policy evaluation is how a team decides whether to ship a new ranker without shipping it. Take the log the current policy produced, reweight each logged reward by how much more likely the new policy was to have taken that action, and average. It is three lines of numpy, it is unbiased under an assumption, and the assumption is almost never checked because the log that supports the estimator does not contain what is needed to check it.
The assumption is coverage: the new policy may only put probability where the logging policy could have. Real systems break it structurally rather than by accident. A candidate generator drops items before the ranker ever scores them, a business rule removes a category, an inventory filter hides what is out of stock. Those actions have propensity exactly zero, which is a different thing from a small propensity: they appear in no row, they contribute nothing to any sum, and no quantity of the same data recovers them. What comes out is the target policy's value restricted to the region the log covers, wearing the target policy's name.
So this repository does not argue about it. It builds a contextual bandit with 4000 rows, 8 actions and 6 features where the counterfactual is known exactly, runs the 4 estimators anybody would run, and measures how often each interval contains the answer. On the shipped configuration the audit returns unsupported, exit 2, and the numbers it refuses to publish as an estimate are: a log worth 56 rows out of four thousand by Kish's effective sample size, which is a share of 0.0141 of what the report would otherwise print, a single weight of 87.19, and a true value is 0.4344 for a target policy whose logging policy scored 0.5463, against which the standard estimator is low by 0.1769 on average.
The corpus is anchored on a case whose answer is exact rather than measured. When the target policy is mixed all the way toward the logging policy the two are the same policy, every importance weight is exactly 1.0, the effective sample size is exactly the row count, and the estimate is exactly the logged mean. The suite asserts all three on the nose, the calibration sweep runs 6 mixture weights out from there, and at the anchor the interval covers 0.98 of the time with a support deficiency of 6.57e-06. That is what separates a measurement from an arrangement: a reader can check the anchor by hand from the definition of an importance weight.
The uncomfortable result is the last one. The diagnostic in this repository is the best predictor of failure I could build from a log alone, it correlates strongly with coverage, and it still does not give a team a threshold. Across the grid, one configuration whose interval is fine carries more support deficiency than one whose interval is wrong, so any single cut misclassifies something, and the interval on a perfect separation rate over twelve cells goes as low as 0.2425 would be consistent with. 116 tests, 99.7 percent line coverage, and make verify re-measures every figure quoted in this document, in the defense guide, in the policy file and in five decision records, and fails if any of them has moved.
Every line of terminal text above is real captured stdout from a command that ran, with each segment paced by that command's measured wall time. It is a replay of a captured session rather than a live screen recording, and docs/video/manifest.json lists each command with its exit code and measured duration. Higher quality MP4: docs/video/demo.mp4.
The log a ranking system produces. Exit code 2. A candidate filter removed actions the target policy wants, so there is no estimate of the target policy's value to report.
$ python -m offpolicy audit
verdict: unsupported
exit code: 2, estimator ips
what this log could not have told you
target mass on forbidden actions : 0.3950
target mass below the support floor: 0.4328
rows with any unsupported mass : 1731 of 4000
$ echo $?
2
Four thousand rows, worth fifty six. The row count is the number a report normally prints. The effective sample size is the number that decides how wide an honest interval is.
And only then, the estimates. Four confident numbers, four intervals, and each one is an interval about the covered region rather than about the policy in the heading.
The panel no real audit has. This is the one that makes the repository a corpus rather than a library: the errors nothing above this line could have seen.
The other side of the boundary. Exit code 0. Same world, same target policy, a log that explored. Without this the exit 2 above is a complaint rather than a boundary.
$ python -m offpolicy audit --policy configs/explored.yaml
verdict: estimate-usable
exit code: 0, estimator ips
$ echo $?
0
docs/screenshots/manifest.json records which report each image came from, the headings it was framed by, and the measured contrast ratio of the verdict badge in it. The capture script fails the run if a badge is invisible or falls below a contrast ratio of 4.5, because the screenshot tool is the only thing in this pipeline that can see pixels.
the diagram source, and why this is a committed image
There is no mermaid fence here, and that is a decision rather than an omission. GitHub renders mermaid itself, and when it works the source is the picture, which is the better arrangement. It does not always work: a diagram that parses under mermaid versions ten and eleven locally can still come back from GitHub as "Unable to render rich display", which is a failure inside their renderer that nothing in this repository can fix. Three smaller traps pushed the same way. A diagram with HTML labels is not well formed XML, because the labels sit in a foreignObject with unclosed br tags, and it then displays when injected into a live page and fails silently as an img src, with naturalWidth 0 and nothing in any console. An img src with a percentage width and no intrinsic height leaves the browser without an aspect ratio. And a transparent background is not theme neutral, because light node fills with dark text come out as dark grey on near black in a dark theme.
So tools/render_diagram.py emits the SVG by hand, with plain text elements, intrinsic dimensions, and one opaque rectangle covering the whole viewBox. It renders identically on GitHub, in an editor preview, in the PDF and offline. The layout it draws, which is the source in the sense that matters:
what the log contains
the context of each request
the action that was served
the reward that came back
the probability the serving policy gave that action, and the whole
distribution it was drawn from
one action per request, and no counterfactual anywhere
what the target policy wants
a different action in many contexts
sometimes an action a candidate generator removed before the serving
policy ever scored it
the reward of that action, in that context, appears in no row of the
log and no quantity of the same data brings it back
four estimators and one diagnostic, and only one of them is about the gap
ips and clipped reweight the logged rewards. Drop the unsupported
actions silently, and report what is left as the
target's value
snips and dr rescale, or model the reward and correct it. Return
a plausible number for the region with no data, and
say nothing about it
the diagnostic how much of the target sits outside what the log
could produce, and how many effective rows the rest
is worth. No estimate, no guess
so an estimate is one of two things, and the log alone says which
the estimate is of a different quantity
the target's value restricted to the covered region, reported as
the target's value
the estimate is of the right quantity
and may still be too noisy to act on, which is a question about the
effective sample size
three verdicts, and the order between them is the design decision
unsupported exit 2 more of the target's mass than the threshold
allows sits outside what the log could
produce. Outranks the other
underpowered exit 1
estimate-usable exit 0
Regenerating the image after editing that layout is one command: python tools/render_diagram.py.
The left column is what a log contains. The right column is what the policy under evaluation wants. Everything below is a measurement of the gap.
This is the section I would most want reviewed, because two of these were defaults I had already written into the policy file and one was a discrepancy between two of my own numbers.
Rejected: clipping the importance weights. It is the standard remedy, it is the first thing anybody reaches for when a weight comes back in the tens, and on this corpus it makes the interval strictly worse. Sweeping 10 weight caps is what settled it:
| weight cap | bias | rmse | coverage | mean interval width |
|---|---|---|---|---|
| 1.5 | -0.3858 | 0.3859 | 0.0000 | 0.0158 |
| 2 | -0.3769 | 0.3769 | 0.0000 | 0.0191 |
| 5 | -0.3471 | 0.3471 | 0.0000 | 0.0332 |
| 10 | -0.3213 | 0.3215 | 0.0000 | 0.0501 |
| 50 | -0.2617 | 0.2633 | 0.0000 | 0.1148 |
| 200 | -0.2253 | 0.2312 | 0.0750 | 0.1924 |
| 1000 | -0.1954 | 0.2170 | 0.2600 | 0.2932 |
| 3000 | -0.1826 | 0.2308 | 0.2600 | 0.3425 |
| 10000 | -0.1769 | 0.2530 | 0.2600 | 0.3647 |
| 30000 | -0.1769 | 0.2530 | 0.2600 | 0.3647 |
The cap of 1000 is the best by root mean square error, with an error of 0.217 there and a coverage of 0.26 there, and 0 of them reach nominal coverage at all. The tightest cap in the sweep is biased low by 0.3858, an interval a tenth as wide as the unclipped one and centred a third of the way from the truth: a cap trades variance for bias, and against a support gap the bias is the part that was already killing you. The shipped default is 10, which is the common one and which this table says is bad, and the default stays because the report has to show a reader the thing they are probably already doing. Reproduce with python experiments/exp04_what_the_clip_buys.py.
Rejected: a support floor of exactly zero. It is the principled threshold and it makes the diagnostic silent on every softmax log, because a softmax policy has strictly positive probability everywhere. A log whose tail propensity is one in a million is fully supported at a floor of zero and carries importance weights in the thousands. The loader now refuses a configured floor at or below zero, and the report prints the deficiency at both floors instead, which is ADR-005. The two are different problems: at zero the figure counts actions that were forbidden and no exploration recovers, at the configured floor it also counts actions that were permitted and never taken, which more exploration would reach if anybody would pay for it.
Rejected: two seed streams for one quantity. The coverage command and the coverage experiment each built their own replication logs, and for the same shipped configuration they reported 0.1750 and 0.2600. Both were correct and only one of them could go in this document. A coverage rate is a property of a seed stream as much as of an estimator, so the repetition now lives in the package as replication_logs and both callers use it. What surfaced it was recording the demo video and noticing that the number on screen was not the number in the draft.
Kept: the doubly robust estimator, which comes out looking good here. It reaches 0.79 coverage, the best of the four, and its reward model is fitted on a reward surface it can represent exactly, which is the most generous case rather than the fairest one. It stays because deleting the estimator that survives would make the argument look arranged, and exp03 is the answer to it: shift the rewards of the unexplored actions and its number does not move at all, because it never had any data about them.
An estimator is a function of the rows it sums over, and that bounds what it can possibly be about, whatever the standard error says. The table is mechanical rather than a matter of opinion.
| Estimator | Sums over | May therefore be about | Cannot be about |
|---|---|---|---|
ips |
logged rows, reweighted by the target over the logging probability | the target's value on the region the log covers | any action with propensity zero, which contributes nothing to the sum |
snips |
the same rows, divided by the sum of the weights rather than the row count | the same region, with the weight normalisation removing some variance | the same actions. Normalising changes the scale, not the support |
clipped |
the same rows with every weight capped | the same region, with variance traded for bias | the same actions, plus the part of the covered region the cap truncated |
dr |
the same rows, plus a reward model evaluated on every action | the covered region, plus a model's extrapolation into the rest | anything the model could not learn, since the unexplored actions appear in no training row either |
That last cell is the honest boundary and it is the one this repository ends on rather than the one it started from. The doubly robust estimator is the only one that returns a number for the unexplored region, and the number is a model's guess extrapolated from rows about other actions.
The same coverage the screenshots above are taken from, over 200 logs from the same world, against a nominal 0.95.
| estimator | interval covers | mean error | rmse | mean interval width |
|---|---|---|---|---|
ips |
0.2600 | -0.1769 | 0.2530 | 0.3647 |
snips |
0.7850 | 0.0208 | 0.1431 | 0.3659 |
clipped |
0.0000 | -0.3213 | 0.3215 | 0.0501 |
dr |
0.7900 | 0.0210 | 0.1030 | 0.2086 |
Read the first column against the last. The estimator with the narrowest interval covers least often, which is the relationship a reader's intuition has backwards: a narrow interval on a biased estimate is a confident wrong answer, and clipping produces exactly that. Read snips and dr: both are close to the truth in mean error, both under cover, and both are close for a reason that is a property of this corpus rather than of the estimators, which is what the next section is about.
The corpus is anchored on a configuration whose answer needs no measurement. Mix the target policy all the way toward the logging policy and the target is the logging policy: every importance weight is a probability over itself, so every weight is exactly 1.0, Kish's effective sample size is exactly the row count, and the inverse propensity estimate is exactly the logged mean. The sweep runs out from there.
| mixture toward the log | support deficiency | effective share | ips | snips | clipped | dr |
|---|---|---|---|---|---|---|
| 1 | 0.0000 | 1.0000 | 0.9800 | 0.9800 | 0.9800 | 0.9850 |
| 0.9 | 0.0433 | 0.9126 | 0.4100 | 0.8950 | 0.1750 | 0.9500 |
| 0.75 | 0.1082 | 0.5723 | 0.2600 | 0.7850 | 0.0000 | 0.8800 |
| 0.5 | 0.2164 | 0.1780 | 0.2550 | 0.7800 | 0.0000 | 0.7850 |
| 0.25 | 0.3246 | 0.0523 | 0.2550 | 0.8200 | 0.0000 | 0.7950 |
| 0 | 0.4328 | 0.0141 | 0.2600 | 0.7850 | 0.0000 | 0.7900 |
The top row is the anchor and it is where the machinery proves itself: at a mixture of one every estimator lands within sampling noise of nominal. One step off it, at a mixture of 0.9, the inverse propensity interval has already fallen to 0.4100 and the clipped one to 0.1750, while the support deficiency is still under five percent. That gap between the first two rows is the whole practical problem: the configurations where this fails are not exotic, they are one small step from the one case where everything is exactly right. Reproduce with python experiments/exp01_the_anchor_and_the_calibration.py.
The sharpest result in the repository, and the reason the snips and dr rows above should not reassure anybody. Take the actions the catalogue filter removed entirely and move their true reward, in logit space, across 5 reward surfaces. Those actions were never served, so no row of the log changes.
| reward shift on the blocked actions | true value | ips | snips | dr | support deficiency |
|---|---|---|---|---|---|
| -3 | 0.3368 | 0.3521 | 0.5577 | 0.5253 | 0.3280 |
| -1.5 | 0.3763 | 0.3521 | 0.5577 | 0.5253 | 0.3280 |
| +0 | 0.4344 | 0.3521 | 0.5577 | 0.5253 | 0.3280 |
| +1.5 | 0.4925 | 0.3521 | 0.5577 | 0.5253 | 0.3280 |
| +3 | 0.5340 | 0.3521 | 0.5577 | 0.5253 | 0.3280 |
Every log in that table is byte identical. Every importance weight is identical. Every estimator returns the identical number down to the last bit, so the estimates differ by 0.0 across the five. The support deficiency is 0.328 in every one of them. And the answer they are all estimates of moves by 0.1972, from 0.3368 to 0.5340.
That is what a support gap is, stated as an experiment rather than as an argument: the estimate is not uncertain about the blocked region, it is silent about it, and the silence is indistinguishable from agreement. The estimators that looked accurate in the previous table looked accurate because the actions nobody explored happened to have about the same mean reward as the ones everybody did, which is a property of that corpus and not of those estimators. Reproduce with python experiments/exp03_the_shift_the_log_cannot_see.py.
Everything above is graded against a truth this corpus happens to know. A team auditing their own log does not have one, so the question that decides whether any of this is useful is whether the numbers computable from the log alone tell them which situation they are in. The grid varies the logging policy, the exploration rate, the catalogue filter and how far the target sits from the log.
| logging policy | mixture | support deficiency | effective share | coverage | interval |
|---|---|---|---|---|---|
| eps greedy 0.9 | 0.9 | 0.0000 | 0.9328 | 0.9400 | fine |
| eps greedy 0.9 | 0 | 0.0000 | 0.1070 | 0.9550 | fine |
| eps greedy 0.5 | 0.9 | 0.0000 | 0.8773 | 0.9300 | fine |
| eps greedy 0.5 | 0 | 0.0000 | 0.0662 | 0.9300 | fine |
| eps greedy 0.15 | 0.9 | 0.0000 | 0.7137 | 0.9600 | fine |
| eps greedy 0.15 | 0 | 0.0000 | 0.0164 | 0.9300 | fine |
| softmax T 2 | 0.9 | 0.0000 | 0.8487 | 0.9350 | fine |
| softmax T 2 | 0 | 0.0000 | 0.0457 | 0.9400 | fine |
| softmax T 0.7 | 0.9 | 0.0104 | 0.5994 | 0.7100 | wrong |
| softmax T 0.7 | 0 | 0.1040 | 0.0047 | 0.5600 | wrong |
| eps greedy 0.9, filtered | 0.9 | 0.0126 | 0.9540 | 0.9200 | fine |
| eps greedy 0.9, filtered | 0 | 0.1263 | 0.1301 | 0.6250 | wrong |
3 of them under cover, and the support deficiency tracks the coverage at a rank correlation of -0.8311 while the effective sample size manages 0.1905, which is close to nothing. So the deficiency is the right diagnostic and the effective sample size is not a substitute for it.
And then the row that stops this being a clean story. A deficiency of 0.0126 covers while one of 0.0104 does not, so the two groups overlap and no single threshold on this diagnostic puts every configuration on the right side. The shipped materiality threshold of 0.05 is a decision with a cost, not a discovered constant, and even a perfect separation over twelve cells would carry an interval reaching as low as 0.2425 would allow. The report prints the deficiency and the effective sample size side by side rather than combining them into a score, for the same reason. Reproduce with python experiments/exp05_what_the_diagnostic_buys.py.
| Technology | Role in this project | Why chosen here |
|---|---|---|
| Python 3.11, 3.12, 3.13 | the whole tool | it has to install next to a serving stack, and the suite runs on all three because it asserts identities rather than values |
| numpy | the world, the policies, the log and every estimator | the only place randomness enters, and the reason a coverage study over two hundred replications finishes in seconds rather than minutes |
| PyYAML | the policy file | the support floor and the materiality threshold are the two values that decide a verdict, so they cannot live in code where a reader has to trust a diff to find them |
| Kish's effective sample size | the precision half of the verdict | it is the row count only when every weight is identical, which happens only when the target is the logging policy, which is exactly why it is the right denominator to print beside the row count |
| Wilson score intervals | every rate in this document | a coverage rate near zero has an asymmetric interval, and the normal approximation puts the lower bound of the clipped estimator's rate below zero |
| the delta method | the standard error of the self normalised estimator | it is a ratio of two random sums, so its variance is not the variance of the numerator, and treating it as though it were is a common way to publish a narrow interval |
| pytest, pytest-cov | 116 tests, 99.7 percent line coverage | the anchor identities are asserted on the nose here rather than produced by an experiment nobody reruns, and two tests walk the syntax tree to prove no diagnostic can reach the counterfactual |
| ruff | lint and format, on src, tests, tools, experiments and benchmark |
one tool, one config, and no argument about style in review |
| GitHub Actions | three Pythons, the tool run through its own action, then a pinned receipts job | the matrix catches an optional dependency imported at module scope; the action job asserts all three exit codes against three committed policy files, since a gate that only ever sees exit 0 is not testing an exit code |
| GitHub Actions composite action | distribution | action.yml makes this five lines in another repository, which is the difference between a demo and a tool |
| Playwright with Chromium | the report screenshots in tools/ |
the only way to check that a verdict badge is legible is to look at the pixels, and the capture asserts a contrast ratio before it saves |
| ffmpeg | the replay video | paced by measured wall time from a captured session, so the video cannot drift from the behaviour |
| matplotlib | the cost chart, in the evidence extra | drawn from benchmark/results/audit_latency.json and never by hand |
| cmark-gfm with Chromium | the defense guide PDF | one markdown source, two renderings, no second copy of the text to keep in step |
Prerequisites: Python 3.11, 3.12 or 3.13, and git. No API keys, no model server, no GPU, and no network access at runtime.
git clone https://github.com/srujan20/offpolicy-gap.git
cd offpolicy-gap
python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
python -m offpolicy plan # what this configuration evaluates, and how each estimator fails
python -m offpolicy audit # the log a ranking system produces. Exit 2
python -m offpolicy audit --policy configs/underpowered.yaml # support holds, the interval is too wide. Exit 1
python -m offpolicy audit --policy configs/explored.yaml # a log that explored. Exit 0
python -m offpolicy coverage # two hundred logs, and how often each interval contains the truth
make verify # lint, the suite, and every published figure re-measuredmake help lists every target. Everything is seeded and deterministic and nothing reaches the network, so any figure in this document can be reproduced from a fresh clone rather than taken on trust.
To audit your own log, the diagnostic and the verdicts transfer unchanged and what has to change is where the rows come from:
cp configs/policy.yaml my-policy.yaml # then set your action count, thresholds and replications
python -m offpolicy plan --policy my-policy.yaml
python -m offpolicy audit --policy my-policy.yaml --no-truth --json out.jsonThe honest limit is here rather than at the bottom of the page: this repository generates its log from src/offpolicy/world.py, and pointing it at yours means constructing a Log from your rows. What transfers without any work is diagnostics, estimators and audit, none of which import the world; what does not is a reader for your log format. The --no-truth flag exists so the report looks the way it will look on your data, with the grading panel absent rather than empty.
In another repository, as a step:
- uses: srujan20/offpolicy-gap@v1.0.0
with:
policy: configs/policy.yaml
underpowered-fails: "false" # exit 1 warns rather than blocks, while you widen the logaction.yml is a composite action, and the reason it exists rather than a bare run: line is the ordering between the two non zero exit codes. Exit 2 means the target policy wants actions the log could not have produced, which no amount of extra data fixes, so it fails the step by default. Exit 1 means support holds and the interval is honest and too wide, which needs a bigger logging budget than one pull request, so it defaults to a warning annotation. Exit 3 and 4 always fail, because they mean the audit did not run: exit 3 in particular means the log carried no propensity distribution, which is a question this tool cannot answer rather than an answer it does not like.
Method: benchmark/bench_audit.py times the estimators, the diagnostic and a coverage study at four log sizes from 2000 rows out to 128000 rows. The world is built outside the timed region, because constructing it is not work an audit does. 7 timed repeats per size after one untimed warm up. Hardware: 2 vCPU, 7 GB RAM container, and the interpreter the benchmark recorded for itself, Python 3.11.15.
| rows | four estimators p50 ms | the diagnostic p50 ms | diagnostic over estimators | coverage study p50 ms |
|---|---|---|---|---|
| 2000 | 1.05 | 0.35 | 0.33 | 58.94 |
| 8000 | 2.2 | 1.2 | 0.54 | 183.2 |
| 32000 | 9.52 | 4.89 | 0.51 | 690.7 |
| 128000 | 40.72 | 19.13 | 0.47 | 3059.11 |
At the largest size the estimate takes 40.7 ms and the diagnostic takes 19.1 ms, which is 0.47 times the estimator it guards. This is the one comfortable number in the repository, and it is worth stating plainly: the check costs less than the thing being checked, at every size measured, so nobody has to trade the diagnostic against latency. The total scales at a slope of 0.57 on a log log fit against the row count, which is sublinear because the fixed cost of building the target's probability matrix does not grow with the log.
The expensive thing is the coverage study, not the audit. At the largest size the coverage study takes 3059.1 ms, because it is two hundred audits rather than one. That is the honest shape of the cost: measuring whether an interval is an interval is two orders of magnitude more expensive than computing one, which is a large part of why nobody does it.
Where it degrades, honestly. Every estimator here is a vectorised pass over an array that fits in memory. A production log is on disk or in a warehouse and the same arithmetic becomes a scan, which changes the constant and not the shape: all four estimators and the diagnostic are single passes, and the diagnostic needs the full propensity row rather than the served scalar, so it reads more columns rather than more rows.
116 tests and 99.7 percent line coverage, measured with pytest --cov=offpolicy. The suite needs no network, no model server and no fixtures downloaded from anywhere: the world is generated from a seed, so a clean clone reproduces every figure.
The most valuable tests in the suite are the two that assert a property of the source rather than of a result. test_nothing_in_the_diagnostics_reads_the_true_reward_surface parses diagnostics.py and walks its syntax tree, asserting that no attribute named means or value_of is reached and that World is never imported. A diagnostic that read the true reward surface would grade beautifully on this corpus and would be unimplementable on any real log, and a docstring promising otherwise is not a check. The same walk runs over estimators.py.
Three more are written to fail in both directions. The anchor is asserted on the nose: every weight exactly 1.0, the effective sample size exactly the row count, the estimate exactly the logged mean. The verdict is asserted to be identical with and without a truth, which is what stops the audit grading itself. And the Wilson interval is asserted both to exclude 0.95 at 185 of 200 and to include it at a rate the shipped configuration reaches, because an interval that only ever excluded would make every claim in this document trivially true.
A README quotes a measurement, the code changes, the number stays, and a year later the document is confidently wrong. So the numbers in this file are not maintained by hand:
make receipts # or: python tools/collect_metrics.py --skip-tests && python tools/check_numbers.py --stricttools/collect_metrics.py runs the suite, reads its machine readable reports, runs all five experiments, reads the benchmark's JSON, and writes every resulting value to docs/metrics.json. tools/check_numbers.py then checks it both ways. Nothing in either file types a number.
Three properties of the check matter more than the idea of it, and each one is there because the version without it failed to catch something:
- Values are pinned to the phrase that makes the claim, not to the file. A metric registers an anchor such as
"covers {} of two hundred", and the check requires that exact string with the value substituted in. Searching a long document for a short number always succeeds, which is how a sentence quoting the wrong figure survives a check that reports "every number matches". - An anchor with no placeholder in it is refused at collection time. Such an anchor matches whatever the document says regardless of the value, which is a guard that cannot fail.
- The reverse direction is load bearing in CI. The forward check catches a deleted figure. The reverse check reports any number in the prose that no metric explains, and
--strictmakes that a failure. Fenced blocks, inline code, HTML attributes and link targets are excluded, so an example invocation may contain a made up count without training the reader to ignore the section.
The number scanner has been wrong twice, in both directions, and both fixes are in the file. It had no leading minus, so a rank correlation published as a negative number was scanned as its magnitude and matched nothing. Then it had no exponent, so a magnitude published as 6.57e-06 was scanned as the two tokens 6 and 06 and the repository's own measurement was reported as unexplained twice over.
One family of figures is deliberately not re-measured on every push: the timing table. A duration measured on a GitHub runner is a different measurement from one measured on the machine described above, so re-timing in CI would fail the check for the honest reason that the hardware changed. benchmark/results/audit_latency.json is the measurement, it is committed, make bench rewrites it, and its diff gets reviewed like any other file.
Every table above is guarded cell by cell, which is deliberately a weaker claim than the prose anchors: a cell is checked for its value appearing as a table cell, not for appearing in its own row. Guarding the row label too would mean generating this document rather than writing it.
Full records in docs/adr/:
- ADR-001: print the support gap above the estimate, not beside it. Why the report's section order is the argument, and what it costs a reader who wanted a dashboard tile.
- ADR-002: log the whole propensity distribution, not only the served propensity. The one change to a serving system this tool needs, why the estimators deliberately do not use it, and why a log without it gets its own exit code.
- ADR-003: anchor the corpus on a case whose answer is exact. What stops a simulated corpus being an arrangement, and what the anchor does not prove.
- ADR-004:
unsupportedoutranksunderpowered. The ordering rule, and why reporting an effective sample size on an unsupported estimate is the mistake it prevents. - ADR-005: report the deficiency at two floors. Why a threshold of zero is refused, and what the gap between the two figures tells a team about their own log.
- Choosing the policy. This repository evaluates one target policy against one log and returns a verdict about whether the evaluation means anything. It does not search a policy space, and a tool that optimised against an estimator it had just shown to be about the wrong quantity would be a strange thing to build. Trigger to reconsider: a log with enough support that the verdict is
estimate-usableacross a whole family of targets. - Confidence intervals that are correct under a support gap. There are none. The estimand changes, so an interval about the covered region is a correct interval about a quantity nobody asked for. The tool refuses rather than widening, and ADR-001 is the record of it.
- Learning a policy from the log. Off policy learning has the same support problem and a harder version of it, because the optimiser is actively drawn toward the unexplored region where the estimate is highest and least constrained. The diagnostic here would apply and the verdict logic would not.
- Sequential decisions. Every row here is one context, one action, one reward. A recommendation session is a trajectory, importance weights multiply along it, and the effective sample size collapses much faster. The mechanism is the same and the arithmetic is worse.
- Estimating the propensity when it was not logged. Fitting a model of the logging policy from the actions it took is a real technique and it cannot recover a propensity of exactly zero, which is the case this repository is about. A fitted propensity would put a small positive number where the truth is zero and the deficiency would silently go to zero with it.
- Deciding what the materiality threshold should be. The shipped 0.05 is a decision with a comment. exp05 shows that no threshold on this grid separates cleanly, which is an argument against anybody publishing one as a standard.
- Secrets. There are none to handle. No credential is read from a config file or from the environment, no network call is made at runtime, and a hygiene test walks
src,experimentsandbenchmarkasserting that no network library is imported anywhere, so the offline claim is enforced rather than promised. - What is never logged. Reports carry rates, counts, thresholds and estimates. No context vector and no per row reward leaves the process: the audit works on aggregates. A report that quoted the rows carrying the largest importance weights would be a data export with a dashboard on top, and those rows are the most identifying ones in any real log.
- The one sensitive input is the log itself. Auditing a real system means reading production interaction data, which is why the tool is designed to run inside the environment that already has it. Nothing is uploaded, and the JSON report is safe to attach to a pull request.
- The policy artifact is reviewable and inert.
configs/policy.yamlis YAML loaded withsafe_load, so a policy file cannot execute code, and a threshold change shows up as a readable one line diff in a pull request. - No pickles anywhere. Reports are JSON, text and HTML, and the policy is YAML. Nothing this tool reads can execute code.
- Supply chain. Two runtime dependencies: numpy and PyYAML. A test walks the syntax tree of every module under
srcand fails if anything else is imported at module scope, sopip install offpolicycannot quietly grow a compiled dependency tree. Playwright, matplotlib and cmark-gfm are behind an extra. - Data. The world is generated from a seeded process with a declared reward surface. No customer data, no scraped content, no licensing question, and no personal data of any kind passes through this repository.
| Failure | Detection | Behaviour | Recovery |
|---|---|---|---|
| The target policy wants actions the log could not produce | The support deficiency, computed from the logged propensity distribution | Exit 2 and the verdict unsupported, with the deficiency at both floors printed above the estimate |
Restrict the target to the covered region, or explore. More of the same data does not help, and the report says so |
| Support holds and the weights are concentrated | Kish's effective sample size as a share of the rows | Exit 1, with the effective sample size and the largest weight beside the row count | More data, or a target closer to the log. The interval is honest and too wide |
| The log recorded only the served propensity | require_full_propensities(), before anything is computed |
Exit 3, naming the missing column. No estimate is offered | Change what the serving system writes. ADR-002 is the argument for it, and the cost is one float per action per request |
| The recorded propensity is not the one the action was drawn from | The mean importance weight, which is exactly one in expectation under a correct log | Reported rather than enforced, because the mean is itself random | Check the propensity floor the serving system applies. A scaled estimate is a different bug from a restricted one |
| A support floor at or below zero | Validated at policy load | Refused, because a softmax policy has positive probability everywhere and the diagnostic would be silent on every log | Set a floor above zero. ADR-005 is the reasoning, and the shipped value is one tenth of uniform |
| A minimum effective share at or above one | Validated at policy load | Refused, because the effective sample size equals the row count only when the target is the logging policy | Set a share below one. A floor at one calls every real audit underpowered and therefore says nothing |
| A weight clip at or below one | Validated at policy load | Refused, because it caps every weight the target raises above the logging policy | Raise the clip, or drop the clipped estimator. The sweep in this README says what a cap buys |
| Fewer than two replications | Validated at policy load | Refused. Coverage is a rate, and with one log there is an outcome rather than a rate | Set at least two. The shipped value is two hundred |
| Every action blocked from the catalogue | Validated at policy load | Refused, naming the action count, because there would be no distribution left to log under | Block fewer than the action count |
| An unknown estimator name | Explicit in the audit entry point | Exit 3, naming the four that exist | Fix the name. Falling back to a default would silently change which estimator the verdict was about |
| Flaky figures across re-runs | Not possible by construction | Every seed is derived from the policy file, so identical inputs produce identical figures | If a figure changes, an input changed. make verify from a clean clone is the check |
Three, and the first is the one I would want to be asked about, because the finding is a negative result about my own tool.
The whole repository points at one deliverable: a number a team can compute from their own log that tells them whether their off policy estimate means anything. The support deficiency is that number, it correlates with coverage at -0.8311 across the grid, and every incentive I had was to stop there.
The test I could not talk myself out of was whether a threshold on it separates. Not whether it correlates, which is a statement about ranking, but whether there is a cut such that everything above it is wrong and everything below it is fine, because a cut is what an alerting rule needs. There is not. One configuration with a deficiency of 0.0126 has a perfectly good interval and one with 0.0104 does not, and the reason is that the second one fails for a variance reason on top of a small support gap while the first has plenty of effective rows.
So the report prints the deficiency and the effective sample size side by side and refuses to combine them into a score. The verdict uses a threshold because a gate has to, and the threshold sits in the policy file with a comment naming who chooses it, and exp05 is cited from that comment. What I would defend is that the honest version of a tool like this reports two numbers a human has to look at, and that a single score would have been more adoptable and less true.
The doubly robust estimator fits a reward model and uses it to fill in the actions that were not taken. The first implementation took world as an argument, because the world is where the features live, and the features are what the model regresses on.
That is a tool that cannot exist. The world is the counterfactual: it is the object that knows the true mean reward of every action in every context, and an estimator holding a reference to it would grade beautifully on this corpus and be unimplementable on any real log. It would also not have been caught by any result, because the model only ever read the feature matrix.
The fix was to move the features onto Log, where they belong: a real log has the context of each request. The estimator now reads log.features and the world is not importable from the module. What makes it stay fixed is a test that parses the source of estimators.py and diagnostics.py and fails if World appears in an import at all, because the next person to need a feature matrix will reach for the nearest object that has one.
Recording the demo video, the coverage command printed an inverse propensity coverage of 0.1750. The draft of this README, quoting the experiment, said 0.2600. Both had run that morning, both were reproducible, and both were about the shipped configuration.
The command and the experiment each built their own sequence of replication logs, from seeds that differed by the world seed. Neither was wrong. A coverage rate is a property of a seed stream as much as of an estimator, and two hundred replications is enough to make a rate stable to about a percentage point and not to five.
The fix is small and the discipline is not: replication_logs moved into the package, both callers use it, and the two numbers are now the same number by construction rather than by coincidence. What I would carry from it is that the receipts pipeline did not catch this, and could not have: both numbers were produced by code that ran. The check that caught it was looking at the screen while a video recorded.
- Read a real log.
Logis a dataclass over eight arrays and a constructor from a dataframe is an afternoon. Everything above it transfers unchanged. First metric to watch after adoption: the structural deficiency at a floor of zero, because that number is how much of the policy under evaluation the log has nothing at all to say about. - A restricted target as an output, not just a verdict. When the deficiency is material, the tool could return the target policy renormalised onto the covered region and estimate that, which is an honest number about a policy nobody wants to ship. It is one function and the reason it is not written is that naming it well is most of the work.
- Sequential logs, where importance weights multiply along a trajectory and the effective sample size collapses geometrically. The mechanism is identical and the arithmetic is worse, which makes it the natural next corpus.
- Calibrate the materiality threshold against a decision rather than a coverage rate. What a team actually wants to know is the probability that shipping this policy is worse than not shipping it, which is a different question from whether an interval covers, and it needs a loss function this repository does not have.
- Before real production use: check that your serving system records the full propensity row and not only the served value, set the support floor from your own action count rather than from the 0.0001 here, and run
coverageon a target you have already shipped, where the answer is known, before trusting it on one you have not. - First metric to watch after adoption: the gap between the deficiency at zero and at the configured floor. It is the part of the problem more exploration would fix, and its size is what turns an argument about logging budgets into an arithmetic one.