PR 11 — Advanced failure and concurrency validation

[batbrainy]

Linked issue

Closes #21 — IMPLEMENTATION_PLAN.md §13's PR 11 — Advanced failure and concurrency validation.

Problem

PRs 3–8 shipped the required failure-path tests. What the plan still claims and nothing demonstrates is the tier above them: that the guarantees §8 and §9 state survive adversarial timing, and that §16's durability gate — "Docker restart policies recover crashed db/web/worker containers automatically (verified by container kills)" — is met by an actual kill rather than by a line of YAML.

Three commitments named this PR by name and were still open: PR 8 (#29) deferred "container-kill e2e, multi-poller stress, crash-window matrix"; ADR 0008 rejected Solid Queue concurrency limits keyed by source id and wrote "revisit when PR 11's multi-poller tests can measure a gap"; and the README and fixtures/github/README.md both promised the redirect corpus scenarios here, which nothing had ever consumed.

Scope

In: crash-window tests (B9's test half); multi-poller concurrency tests (§9) including ADR 0008's measurement; class-isolation and fairness stress including starved-class enrichment (S3.9's stress part); fixture-mode fail-closed and container-kill verification (D12); the two orphaned redirect scenarios.

Deliberately out: PR 9's multi-source allocation, shared-IP reconciliation and budget observability; PR 10's /status and extended logging; PR 12's README completion and clean-checkout verification. IMPLEMENTATION_PLAN.md is unedited — it is frozen, and this PR honours §15 rather than amending it. app/ carries one comment-only change, correcting a forward reference this PR resolves.

Technical decisions

The split. CI proves every claim that is a property of a file or of the application's own logic; a dated reviewer transcript proves every claim that is a property of the Docker runtime. Nothing straddles — CI runs bundle exec rspec with no path arg and no tag exclusion against a bare postgres:16 service and has no Compose stack, so a Docker-dependent example would break it. Tagging and excluding was rejected because it needs .rspec, ci.yml and config/ci.rb all edited and any one drifting silently re-enables it; a bin/ harness and a rake task were rejected on the grounds script/probe_304.sh already states about itself.

§15 step 8's command does not exercise the restart policy. docker kill is an API stop: the daemon records the container as manually stopped, and restart: unless-stopped is defined to skip exactly that case. Measured against Docker 28.3.0, all three services stayed down with RestartCount at 0 — a reviewer following the plan literally would conclude the gate is unmet. The policy is sound; it is the kill that does not test it. A SIGKILL to the container's main process from outside its PID namespace, which the daemon does not attribute to an operator, brought db, web and worker back unaided. The script performs both kills and reports both. An in-container kill is not an alternative — the kernel refuses SIGKILL to a PID namespace's own init from inside it — and the helper needs --user 0, because the app image's non-root user cannot signal postgres.

ADR 0008's measurement, with a recorded verdict. No gap. Five contended ticks produce no run row, request, debit or schedule movement; a contended tick issues no INSERT, UPDATE or DELETE at all, so a semaphore would spend writes to prevent writes that never happen; a killed session's lock frees in milliseconds where a fixed duration cannot; and key agreement comes from SourceProvisioner, not from any concurrency primitive. A grep example asserts no job declares limits_concurrency, so reversing the decision fails a test. Bounded honestly: this covers correctness and local cost, not semaphore behaviour under N real worker containers.

Fairness measured over a window, not a choice. Every existing fairness spec asks for one choice against counters set by update_all. None spends a window, so none can express §10's actual claim. The drain runs the real Fairness, Claim and BudgetLedger — and deliberately not the transport, whose sticky tail would hand every flood row octocat's body and produce 57 identity failures that obscure the sequence.

concurrent_write_spec.rb turns transactional fixtures off, the second file in the suite to do so, because what it tests is what one session can see of another's commit. An around hook is not a workaround: its ensure still runs inside the fixture transaction, so cleanup would roll back with everything else.

Testing performed

1,487 examples, 0 failures — bind-mounted and again against a freshly built image with no bind mount. RuboCop clean, Brakeman 0 warnings, bundler-audit clean. Green across three seeds; the suite must be run in its own Compose project, since sibling worktrees share github_push_ingestor_test and a concurrent run deadlocks it.

Two examples were verified by breaking the code and confirming only they fail:

• Removing event_source.reload from IngestionRunner#call fails 4 multi-poller examples and nothing else. That reload is what stops two pollers who both read a stale cadence from polling back to back — ADR 0006 rejected an alternative specifically to protect it, and nothing tested it.
• Swapping PageWriter's actor-before-repository upsert order fails exactly 1. Asserted on statement order, because both upserts share one per-envelope transaction and the positional observable is identical either way — the first attempt at this example passed under both orders and was rewritten.

New coverage, by file: spec/recovery/multi_poller_spec.rb (the stale-cadence race, the gate across both request paths including the enrichment lease being restored exactly, ADR 0008's four observations); spec/recovery/crash_window_spec.rb (an abandoned running run row nothing resurrects, a page committed one envelope deep then replayed, finish's write order, and all three crash signatures composed into one restart); spec/recovery/concurrent_write_spec.rb; spec/services/github/enrichment/fairness_stress_spec.rb; spec/services/github/fixture_mode_fail_closed_spec.rb (fail-closed proved through the real chain with a throwaway corpus, rather than by stubbing the runner to raise); spec/services/github/enrichment/redirect_boundary_spec.rb; and 18 new spec/docker_compose_spec.rb examples covering the declarations the recovery claim rests on — the named volume, the healthcheck's -h 127.0.0.1, web's puma command (a future "simplification" to bin/rails server must fail there), the test service's isolation, and that no service leaves restart implicit.

Docker verification

script/verify_recovery.sh --confirm against a fixture-mode stack; transcript committed at docs/evidence/2026-07-31-container-kill-recovery.md. push_events read 263 before and after every kill, restart and recovery; the pgdata volume is identical at both ends, so those counts are survival rather than recreation; docker compose run --rm test left both development databases byte-identical and did not trigger setup; and fixture mode failed closed in the running stack — this database holds entities from earlier live polls, and the cycle refused with a corpus gap and exit 2 rather than reaching api.github.com.

The script refuses under CI, without --confirm, against a worker not in fixture mode, with RAILS_ENV set to anything but development, or when container names do not match §15's literals. It never runs docker compose down, never drops a database, never issues psql against a _test database, and redacts logins, repository names and avatar URLs — a development database that has polled live GitHub holds real third-party identifiers no finding needs. The fixture-mode guard fired for real during development, refusing a run I had started against a live worker.

Documentation updates

README gains Crash recovery, verified and the nine-row fixture scenario matrix, replacing the disclaimer that container-kill verification "lands with PR 11; nothing above claims it" and the two now-delivered rows of the Planned-contents table. ADR 0008 records the measurement and its verdict; fixtures/github/README.md names what consumes the redirect scenarios; poll_event_source_job.rb's "Revisit in PR 11" comment now points at the answer.

Known limitations

• The container-kill result is not machine-verified on every push. A compose change that breaks recovery at runtime without changing the YAML would go unnoticed until the transcript is re-run; the new docker_compose_spec.rb examples exist to shrink that window.
• The transcript is one host, one OS, one Docker version, one date. unless-stopped is not exercised across a daemon restart or a host reboot, and a SIGKILL to postgres exercises WAL crash recovery, not disk corruption.
• No kill was timed inside an in-flight push_events insert; that guarantee is not observable from outside the process and is asserted by spec/recovery/crash_window_spec.rb instead.
• Counts were read as deltas, not the README's documented absolutes, because this database was not started from an empty volume. The script detects and reports which mode it used.
• Processing remains at-least-once execution with effectively-once persisted outcomes. No exactly-once claim.

🤖 Generated with Claude Code