fix(qa): the instance tests could pass an image they never exercised (L082-L086)

[robballantyne]

An audit of all 47 shipped instance tests (~8,750 lines) found 23 verified defects across 16 files, collapsing to 8 root causes. These run on rented GPUs, so each costs a rebuild cycle to find the slow way. Fixed here in one batch, with the shapes codified so the next instance is a red diff rather than a red cell.

Base ROOT + the shared derivative suites only. The unsloth-studio work is deliberately not here.

False greens — an image ships broken and reports fine

service_running used as an if guard, 7 sites. It reports a supervisord state, so if service_running x && wait_for_port p; then … else skip; fi collapses "not configured", "RUNNING but never bound its port" and "supervisord has never heard of it" into one silent pass. A jupyter that hangs without exiting binds nothing and the suite reported ALL TESTS PASSED — autorestart catches crashes, so the hang is exactly the state nothing else covers.

The first version of this fix would have broken the build, and a pre-build review caught it. Deciding "expected" from /etc/supervisor/conf.d/NAME.conf alone is wrong: syncthing.conf and tensorboard.conf ship unconditionally, but 5 of 7 QA templates carry no portal entry for them, so those services correctly exit 0 and sit EXITED. Requiring the port means a 60s wait and a hard fail on a healthy image — and runner.sh skips the whole derivative phase on any base/* failure. base-qa and pytorch-qa list every entry, which is why it looked safe when only base was considered.

The predicate is now configured AND routed AND not-serverless, which 65-conditional-services already used — and which also resolves base/67 and base/85 asserting opposite verdicts about the same six programs.

Others: wait_for_caddy_ports || echo WARN falling through to test_skip, in the two files whose own comments say that wait is what makes the skip trustworthy (and which use || test_fail at six other call sites). A check named bind-loopback passing 172.17.0.2 — the docker bridge, reachable by co-tenants — because it enumerated the bad forms instead of requiring the good ones. runner.sh reporting passed on zero discovered tests, upstream of the required-tests check. And its OPEN_BUTTON_TOKEN guard running after wait_for_client, leaving the results server unauthenticated on a mapped port for exactly the window the guard exists to close.

Checks that could not fire

runner.sh zero-test guard	printf '%s\n' on an empty array emits one blank line → mapfile len=1 → "Discovered 1 tests" → bash ""
12-provisioning stall detector	word-matched provisioner, matching the provisioner itself — pinned true for the whole of a hang, the one state it exists to detect (and 11 unrelated processes on this host)
15-boot-markers	grep -q "PATH=" satisfied by TCLLIBPATH=
27-caddy-tls	grepped lowercase for openssl's capital-V Verify return code — the cert always reported untrusted

False reds — these cost redraws and block promotes

pidof caddy matches transient caddy hash-password (cost-14 bcrypt, 43s measured) and returns "P1 P2", a string that can never appear in an ss row — a healthy image failing "no listening sockets". llama.d/10's health-wait had a terminal catch-all, so a failed supervisorctl and normal in-flight BACKOFF both hard-failed, partly defeating the file's own restart tolerance. 56-cpu-thread-limits wrote without pinning $WORKSPACE. The sglang/vllm serving tests hardcoded 18000 while their sibling contract tests derive it from *_ARGS. llama.d/11 declared TEST_TIMEOUT=1800 then waited 3600s — it could only ever be killed and reported as a bare timeout naming no check.

Five rules

L083 fail_later arity · L084 curl status fallback · L085 TEST_TIMEOUT covers its waits · L086 RUNNING is not serving · L082 tightened to reject a bare $4 — which is exactly how 67's awk passed lint while reporting the first listener on the box after shell quoting destroyed its program.

Two of these shipped broken and were caught before merge, which is the argument for running a rule against real input rather than reasoning about it. L084 was inert against the two-line form it was written from while firing on correct code. The arity counter ran past ;, ignored single quotes, and my first rebuild broke $( ) inside double quotes.

Evidence

1032 passed, 9 skipped; baseline CLEAN ✓ across 27 images. Every fix verified against the real defect — the ss peer column, the empty-array printf, the 000000 capture, the promiscuous pgrep — rather than reasoned about.

Note for the reviewer

Group 1 is the change that could turn currently-green base cells red on real hardware. It wants a live base + pytorch promote from this branch before merge, not a merge first.