Progressive delivery for prompts and models: canary a change on a sticky slice of traffic, and let an always-valid sequential test roll it back automatically — peeking after every request without inflating the false-rollback rate the way an ordinary confidence interval would.
Everything below is reproducible from this repo with no model and no network:
uv sync --extra dev && bash demos/run_all.sh. Every number is re-derived from
the committed artifacts by
scripts/check_readme_claims.py, which CI runs
as a gate — so a stale figure here fails the build rather than quietly misleading
you.
Offline evals tell you a prompt change looks good on a golden set. The real test is production traffic, and the mechanism teams reach for is a canary: send the new prompt or model to a small slice of users, watch the quality signal, and roll back automatically if it dips. The infrastructure looks like a feature flag, but the statistics underneath are where it goes wrong.
The trap is peeking. A canary checks its metrics continuously and stops the moment they look bad. A textbook confidence interval assumes you look once, at a sample size fixed in advance. Look after every observation and reject the first time the interval excludes zero, and the probability of some look producing a false alarm compounds toward one — you roll back changes that were never actually worse. That is not a hypothetical: this repo measures it.
The one measurement that motivates the whole design — the same A/A traffic (no real difference between the two arms) run through the identical controller, with only the statistical test swapped:
| test | false rollbacks on A/A | vs nominal α = 0.05 |
|---|---|---|
| naive fixed-horizon z, peeked every step | 234 / 2000 = 11.7% | 2.3× over |
| mSPRT always-valid confidence sequence | 0 / 2000 = 0.0% | within α |
The naive interval — the one most people would reach for — rolls back a healthy candidate more than one time in ten. The always-valid sequence, checked just as often, holds at or under its nominal error rate. That gap is the reason to build this rather than bolt a t-test onto a feature flag.
Any progressive-delivery demo can show a bad candidate being pulled. The question a reader should ask is what would make the guarantee fail. Three design moves answer it, each with a control that must break if the claim is hollow:
1. The A/A false-rollback rate is measured, not asserted — against a negative
control that must fail. The headline table above is only meaningful because the
naive arm does blow past α. If it didn't, the sequential test would be solving a
non-problem. A test pins exactly that (test_naive_peeked_interval_OVER_rolls_back_on_aa):
if the naive false-rollback rate ever drops to α, the comparison is dishonest and
the build fails. The two arms share the identical controller, traffic, and seeds —
the only variable is which interval the rollback rule reads.
2. A horizon default is never miscounted as a quality rollback. The controller
has a safe default: if the bake window elapses while the verdict is still
inconclusive, it takes the operator's configured fallback. Early on, that default
was rollback, and every A/A run hit it — so a naive count of "runs that rolled
back" would have reported a catastrophic false-rollback rate that was really just
the safety default firing. Those are different events. Every terminal decision now
carries a Trigger (quality / guardrail / horizon), and the calibration
counts only quality rollbacks — the ones the sequential test is actually
responsible for. test_horizon_default_is_not_counted_as_a_quality_rollback pins
the distinction.
3. The variance proxy needs no estimation, so coverage can't be gamed by a
lucky variance. Quality here is bounded in [0, 1] (a judge score, or a 1/0
programmatic pass). By Hoeffding's lemma such a variable is 1/4-sub-Gaussian, so
σ² = 0.25 is a universal bound — coverage holds for any [0,1] quality
distribution without fitting its variance from the same data used to test it. The
price is conservatism, which is why the mSPRT arm reads 0.0% rather than exactly
5%: on Bernoulli data the bound is loose in the safe direction. For a rollback
controller, erring toward not falsely pulling a healthy candidate is the right
direction to be wrong.
- Routes traffic stickily by hashing the user id, so a user sees one variant for the whole ramp and never flip-flops as the fraction grows.
- Ramps monotonically (5% → 10% → 25% → 50%) on a schedule, so a bad candidate is exposed to the fewest users before it trips.
- Accumulates online metrics under sparse, delayed labels — only a fraction of requests get a quality label and it lands some steps later, matching how judge scoring and thumbs actually arrive.
- Rolls back on an always-valid confidence sequence (Robbins' normal-mixture mSPRT), so continuous monitoring does not inflate the false-alarm rate.
- Guards latency and refusal as secondary rails, using the same anytime-valid machinery so the guardrails don't reintroduce a peeking leak.
- Serves a zero-dependency web UI to watch the confidence sequence close in on the margin and the moment a rollback fires.
| # | Criterion | Evidence |
|---|---|---|
| 1 | Canary a change to a configurable traffic fraction, sticky assignment | router.py; test_router.py pins stickiness (users never flip back) and that the realized split matches the fraction |
| 2 | Online metrics (sampled judge scores, latency, refusal) per variant | metrics.py: sparse + delayed labels via LabelBuffer; streaming median latency |
| 3 | Sequential test (mSPRT-style) driving promote/rollback | confseq.py (normal-mixture confidence sequence) + controller.py |
| 4 | Automatic rollback on an injected regression within a bounded window | detection sweep, results/detection_sweep.json: −0.20 caught 400/400 at a median of 131 candidate labels |
| 5 | Injected-improvement scenario promotes, does not roll back | 400/400 promoted, 0 false rollbacks: results/detection_sweep.json |
| 6 | False-rollback rate on A/A traffic | 0/2000 (mSPRT) vs 234/2000 (naive): results/aa_calibration.json |
| 7 | Detection-window measurements across regression severities | the four-severity sweep table below |
| 8 | End-to-end demo on a simulated-traffic harness | tripwire canary, tripwire demo (live prompt regression, offline replay), the web UI |
At every step the controller reads the landed quality labels for both arms and
forms the anytime-valid confidence sequence for mean(candidate) − mean(baseline).
The decision reads only the interval bounds [L, U], so every verdict is
reconstructable from the numbers it prints:
quality difference (candidate − baseline)
+0.1 ┤
│ ╭────────────── U (upper bound)
0.0 ─┼───┼─────────────────────────── ← rollback margin
│ ╰────╮
│ ╰─────╮
−0.1 ┤ L (lower) ╰──────╮
│ ╰───╮
−0.2 ┤ ╰── U dips below 0 → ROLLBACK
└──────────────────────────────────────────────→ requests
- Rollback when
U < −rollback_margin: confident, accounting for peeking, that the candidate is worse. This is the safety monitor. - Promote when the bake window elapses with no rollback trigger (schedule-based,
the realistic canary discipline), or — in the eager mode — once
L > −marginproves non-inferiority. Using non-inferiority rather than strict superiority is what lets an equal-quality candidate ever ship instead of ramping forever. - Continue otherwise: ramp further and gather more labels.
The confidence sequence widens like √(log n / n) instead of the fixed-n √(1/n).
That extra √(log n) is the exact, finite price of being allowed to peek forever —
finite because the sequence still shrinks to zero, so a true regression is always
caught eventually.
The same thing in the web UI — the shaded confidence sequence narrowing over the ramp, the point estimate riding just below the margin, and the rollback firing the moment the upper bound finally clears it:
Host: Apple M4 Pro, macOS 26.5, Python 3.12. The live demo used Ollama
gemma4:31b-it-qat (30.7B, Q4_0); every other number is pure simulation and needs
no model.
2000 independent seeds of A/A traffic (baseline quality == candidate quality ==
0.8), 50/50 split, α = 0.05. Only quality-triggered rollbacks are counted; the
two arms differ only in the interval.
| test | false rollbacks | rate | within α? |
|---|---|---|---|
| mSPRT (always-valid) | 0 / 2000 | 0.0% | yes |
| naive z (peeked) | 234 / 2000 | 11.7% | no (2.3× over) |
results/aa_calibration.json.
Baseline quality 0.85, sparse (30%) delayed (25-step) labels, the 5%→50% ramp, 400 seeds per severity. The window is reported in candidate labels observed at rollback — the operationally meaningful budget, since that is what costs money and time online.
| candidate drop | rollback rate | median candidate labels at rollback | p90 |
|---|---|---|---|
| −0.20 (0.85→0.65) | 400/400 | 131 | 234 |
| −0.10 (0.85→0.75) | 400/400 | 864 | 1323 |
| −0.05 (0.85→0.80) | 52/400 (13%) | 2241 | 2579 |
| −0.02 (0.85→0.83) | 0/400 | — | — |
A large regression is caught fast and always; a regression near the noise floor is caught rarely and late, within the 20k-request horizon. That is the honest behaviour — the test does not manufacture confidence it hasn't earned, and the −0.05 row is reported as it came out rather than tuned to look decisive (see docs/findings.md for why it sits at 13% and not higher).
A genuinely better candidate (0.70 → 0.85) over 400 seeds: 400/400 promoted, 0
false rollbacks. The safety monitor does not fire on a candidate that is
actually better. results/detection_sweep.json.
The one arm that touched a real model. A ZIP-extraction feature whose downstream
parser needs exactly five digits, canaried between two prompts on
gemma4:31b-it-qat:
- baseline (strict): "Reply with ONLY the 5-digit ZIP code. No other text."
- candidate (a plausible "improvement"): "You are a friendly, helpful assistant… let them know what the ZIP code is."
Graded by a programmatic checker (re.fullmatch(r"\d{5}", answer)), harvested once
and replayed offline:
| prompt | pass rate |
|---|---|
| baseline (strict) | 24/24 = 100% |
| candidate (friendly) | 0/24 = 0% |
Every candidate response was like The ZIP code for that address is **20500**. —
the correct ZIP, wrapped in a sentence that breaks the contract. A "does the
answer contain the right ZIP" check would pass all 24; the strict checker fails
all 24, and the controller rolled the candidate back at request 416. The
completions are committed in sessions/, so tripwire demo reproduces the
rollback with no model. results/live_demo.json.
| Component | Choice | Notes |
|---|---|---|
| Language | Python 3.12 | |
| Dependencies | typer, python-dotenv |
The router, confidence sequence, controller, and simulator need no third-party code |
| Live model | Ollama gemma4:31b-it-qat |
Only touched by tripwire harvest |
| Statistics | Robbins normal-mixture mSPRT | Closed-form, dependency-free; inverse-normal via Acklam |
| Web UI | http.server + one static page |
No web framework |
| CI | GitHub Actions | Re-derives every README number; fails on drift |
tripwire/
├── src/tripwire/
│ ├── confseq.py # the mSPRT normal-mixture confidence sequence + naive control
│ ├── router.py # sticky hash-based assignment, monotone ramp
│ ├── metrics.py # per-variant stats under sparse/delayed labels; streaming median
│ ├── controller.py # the promote/rollback rule; naive-CI swap for the control
│ ├── simulator.py # deterministic seeded traffic
│ ├── experiments.py # the scenarios + aggregate measurements everything cites
│ ├── livedemo.py # the ZIP task, two prompts, programmatic checker
│ ├── replay_demo.py # runs the harvested completions through the controller, offline
│ ├── providers.py # OllamaProvider (live) + ReplayProvider (offline)
│ ├── types.py # Outcome / Decision / Trigger / ControllerConfig
│ └── webui/ # zero-dependency canary monitor
├── tests/ # incl. the negative controls: naive-must-over-rollback,
│ │ # mSPRT-within-alpha, horizon-not-counted, sticky-terminal,
│ └── test_webui.py # web-UI path-traversal vectors pinned
├── scripts/
│ ├── calibrate_aa.py # the A/A centerpiece: mSPRT vs naive
│ ├── detection_sweep.py # detection window across severities + improvement
│ ├── harvest.py # live recording (resumable)
│ ├── live_demo.py # offline replay of the harvested regression
│ ├── make_results.py # regenerate everything results/ holds
│ └── check_readme_claims.py# re-derives this README's numbers; CI fails on drift
├── sessions/ # the committed live completions (harvested once)
├── results/ # committed evidence this README cites
└── docs/
├── design.md # the statistics, and the decisions that weren't obvious
└── findings.md # every measurement that surprised me, with the mistake
Everything except harvest needs no model and no network:
uv sync --extra dev
uv run pytest -q # the suite, incl. negative controls
uv run tripwire calibrate # A/A false-rollback: mSPRT vs naive
uv run tripwire sweep # detection window across severities
uv run tripwire canary --baseline 0.85 --candidate 0.70 # one canary run, verbose
uv run tripwire demo # replay the live prompt regression
uv run python scripts/check_readme_claims.py # re-derive every number above
bash demos/run_all.sh # all of it end to endThe web UI — drag the sliders and watch the confidence sequence decide:
uv run tripwire serve # http://127.0.0.1:8014Re-harvesting from a live model (the only step that needs Ollama; resumable):
OLLAMA_TIMEOUT=600 uv run python scripts/harvest.py- The quality signal is a bounded scalar. Real online quality is a mix of
sparse judge samples, delayed task-completion, and thumbs. The design models the
sparsity and delay faithfully, but collapses the signal to one
[0,1]number per labeled request. A multi-metric decision (quality and latency and refusal jointly) is a natural extension; here latency and refusal are separate guardrails, not part of the primary sequential test. - The union bound across two arms is conservative. The difference confidence sequence runs each arm at α/2 and combines — simpler and obviously correct, but looser than a single joint martingale. The slack widens the interval, which for a rollback controller errs safe (and shows up as the 0.0% A/A rate); a self-normalized joint bound would detect small regressions with fewer labels.
- σ² = 1/4 is the worst-case variance. Coverage is guaranteed for any
[0,1]quality distribution, but a low-variance one gets a wider interval than it needs, so detection of small effects is slower than a variance-adaptive bound would be. The −0.05 severity sitting at 13% rollback within the horizon is a direct consequence — see findings. - Regressions are injected, not stumbled upon. The severities are chosen to map the detection curve, and the live demo's regression is a real but deliberately induced prompt edit. The mechanisms are disclosed, not dressed up as natural drift.
- Simulated traffic, one live task. The statistical claims come from a seeded simulator (which is exactly the "simulated-traffic harness" the spec asks for); the live arm is a single ZIP-extraction task on one model. Every number is reproducible from this repo on comparable hardware, not a property of AI features in general.
Every claim in this README is backed by a committed artifact under results/ or a
test that fails when the claim breaks. The centerpiece — that the always-valid
test holds α while the naive one doesn't — is only credible because the naive arm
is run under the identical controller and shown to fail; a design that only
demonstrated its own success would prove nothing. Where a number came out
unimpressive — the −0.05 regression caught only 13% of the time within the horizon
— it is printed as measured, with the mechanism (the conservative worst-case
variance) explained rather than tuned away.