post: a passing check is a claim about what ran, not what's true

[clarkbw]

Post: a-passing-check-that-never-ran…-jsdom-no-webgl.md

Candidate #8 from the 2026-08-11 blog-candidate review — the flagship theme essay, not the usual broke→tried→fixed single-bug post.

Thesis: a green check is a claim about what ran, not about what's true. Six independent bugs from one month, across four unrelated codebases, all the same shape — the set of things examined was smaller than the set of things claimed, and nothing in the output distinguished the two.

The six vignettes (each ends in one generalizable rule):

1. isOnLand() returning false for both "open water" and "no coastline data here" — three CHS gates audited valid by a check that never examined them (station-corrections 7d52f67, 1c06273)
2. check-slugs iterating the lock side only, so a station absent from the lock was never visited (station-corrections 99afa20 — its commit message names the shared failure class out loud)
3. A tool benchmark scoring which tool was called, not what it returned — hid a months-old all-nulls battery_state bug (signalk-mcp e837d92)
4. A CLI wrapper's selftest grepping for a warning-box title the plain-text subcommand never prints — passed while genuinely broken, caught only by running the negative control
5. A Puppeteer smoke asserting the instant the canvas appeared, ~2 s before the map finished failing (slackwater-web 32bfe09)
6. A fix no conventional check can guard because jsdom has no WebGL — delete it and tsc, the suite, the build and the smoke test all stay green (slackwater-web afa3df1, 88ec107)

Closes with a six-question checklist (#the-checklist, linked from the TL;DR).

Figures (2, hand-written SVG):

• blind-spots.svg — the six checks, each as an examined/never-examined bar with the blind spot marked. High, under the TL;DR.
• settle-then-judge.svg — the smoke-test race timeline: canvas up ≈2 s before the last failure lands.

Privacy: vignette 4 comes from the private infrastructure repo and is fully generalized — "a CLI wrapper's selftest", a neutral mytool in the snippets, no hostnames, no repo names, no infra paths. The quotable lesson ("a selftest that passes when broken is worse than no selftest") is preserved. Vignette 3's live-verification host is omitted for the same reason.

Verified: script/lint-liquid.py clean (35 posts), bundle exec jekyll build clean, both <img src="/assets/img/…"> present in _site/…/index.html and both SVGs in _site/assets/img/…. Figures rendered in headless Chrome at 880 px to confirm no text overflow. No {% raw %} guard needed — no Liquid-significant syntax in the body.

Date is the draft date (2026-08-11) — script/bump.py / script/publish.py set the real one at merge.

Worth a look before merging:

• The title is the essay's thesis rather than a symptom string ("A passing check is a claim about what ran, not what's true"). The SEO load is carried entirely by the slug, description and tags.
• Related-links footer points at the test-oracle post, the delivery-path post, and the LLM-bench post (vignette 3's benchmark, written up before its scoring blind spot surfaced).