Fix relay readiness and drain checks in the resilience harness

[jmchilton]

This PR description was drafted and posted by Claude (AI assistant) on jmchilton's behalf.

Summary

The resilience harness treats the relay's /messages/poll/stats as a steady-state "is a consumer attached" signal. It isn't. Both readiness and drain read it wrong, in opposite directions.

Evidence

Holding a single long poll open against ghcr.io/mvdbeek/pulsar-relay:0.2.1 and sampling the stats endpoint every 50ms:

samples: 010010000000000001000000100000001000000100000000001000001000000100000000000000000000001000000011000000101000101000000001010001001001100000000000000010110000
nonzero 25/156 = 16.0% duty cycle

A second run of the same experiment measured 59.5%, with the longest run of zeros at 0.51s. The relay registers a poll-waiter only while a long poll is parked in its storage loop, so a live consumer reads as absent in most samples.

That breaks two checks:

wait_until_consuming can time out against a healthy Pulsar. Seen on CI: TimeoutError: Pulsar did not bind relay consumers within 60.0s while the relay log shows an unbroken chain of 34 poll waiters, one every 2.004s, across the whole 60s. The harness polled every ~0.198s and got HTTP 200 every time, so it was sampling a live consumer ~300 times and reading zero each time. A fixed poll interval can beat against the relay's ~2s cycle and land in the invisible part of every one.

_wait_relay_setup_waiters_drained returns immediately, always. It took a single zero as proof the old waiter was gone. With this duty cycle it reports "drained" on the first sample even when a consumer is fully alive — so the guard it exists to provide (don't let readiness pass against a stale registration) has not been doing anything.

Changes

• Jitter the readiness poll interval so the sampling cannot lock to the relay's cycle.
• Require zeros to persist for RELAY_DRAIN_CONFIRM_SECONDS (2s, ~4x the largest observed gap) before calling the waiters drained. RELAY_DRAIN_TIMEOUT goes 7s → 10s to make room.
• Put the bind-marker state, the sample count, and the last stats body into the TimeoutError, so the next occurrence diagnoses itself instead of needing this investigation again.
• Split _relay_setup_waiter_count() into _relay_poll_stats() + _setup_waiter_count(stats) so the raw body is available for that message.
• Pin the relay image by digest (0.2.1@sha256:b677658c…), overridable with RELAY_IMAGE. The readiness checks depend on the shape of an undocumented endpoint of that image; leaving it on :latest means its behaviour can change with no commit here.

The topic-key parsing was not the problem — keys are namespaced <uuid>/job_setup and the existing endswith("/job_setup") matches correctly. Verified against a live relay.

Validation

• Reproduced both defects against a locally running relay from this compose file, and re-ran the same probes after the change: the drain check now refuses a live consumer (previously passed on the first sample) and still reports drained within ~2.2s when nothing is consuming.
• Readiness detection latency against a consumer on a 2s poll cycle, 5 trials: 0.01s, 0.01s, 0.36s, 1.09s, 2.78s.
• flake8 clean.
• test/resilience/scenarios/test_pulsar_restart.py against a real stack: 4 passed in relay mode (including test_a2_sigkill_during_execution, the CI failure above), then 12 passed across all three messaging modes.

Open question

Pinning the relay image means this harness no longer catches relay-side drift. If tracking the relay's tip was the point of :latest, a scheduled job running with RELAY_IMAGE=ghcr.io/mvdbeek/pulsar-relay:latest would give that back without letting an upstream change turn a PR red for reasons unrelated to it. Happy to switch to that shape.