PR 8 — Background processing and recovery

[batbrainy]

Linked issue

Closes #18 — IMPLEMENTATION_PLAN.md §13's PR 8 — Background processing and recovery.

Problem

Nothing fires on a schedule. docker compose up starts db, setup, web, and then waits for someone to run bin/ingest or bin/enrich by hand. Extension B's restart-safety claims — pending work preserved in business tables, work reconciled after a crash, advisory locks released on session death — are stated in the plan and unproven in the repository.

Scope

In: Solid Queue in its own queue database inside the same Postgres container (B4); the worker container; the 60-second recurring poll tick (S1.7); EnrichActorJob / EnrichRepositoryJob with post-commit enqueue; the entity-scoped reconciler (S3.8, B6); restart safety (S1.8, B5); advisory-lock release on session death, verified (B7's verification half); job IDs in logs (S4.3's job-ID half); recovery tests (D11).

Deliberately out: rich /status and queue observability (PR 10); job_id on the DEBUG github.request line (PR 10); container-kill e2e, multi-poller stress, crash-window matrix (PR 11); multi-source allocation validation (PR 9); Solid Queue concurrency limits keyed by source id — rejected with a reason in ADR 0008, not deferred silently; an id-addressed enrichment job, which the fairness design rules out.

Technical decisions

The enqueue is a hint; the entity rows are the durable record. Solid Queue's separate database means an enqueue cannot join the business transaction. Github::IngestionRunner#call dispatches once per run that created events, after every row has committed, the run row is finalized, and the source lock is released. ReconcilePendingEnrichmentsJob asks the same question of committed state every 60 seconds, so a process killed between the COMMIT and the enqueue loses the hint and never the work. ADR 0008 records this, with the rejected alternatives.

One cycle per class, not one per created event. Github::EnrichmentRunner enriches one entity per call and chooses it by §10's fairness policy under a FOR UPDATE SKIP LOCKED lease — an id-addressed job would have to bypass that ordering to honour its argument, which is how a repository flood starves actors. So N enqueues carry exactly the information one does. Enqueuing per created event would put ~2,400 argument-identical cycles an hour on a queue that can spend 40 requests, each running the age-out sweep to be told no.

A tick is not a poll. EventSource.poll_due pre-filters (mode's source_type, enabled, idle, projection arrived) and Github::PollSchedule under the source lock stays the authority — the runner re-reads the row before deciding. The source_type filter is load-bearing: a development database routinely holds a fixture source too, and a live worker polling it would raise FixtureMiss once a minute forever. db/migrate/20260731100000 adds the partial index spec/db/schema_spec.rb explicitly asked PR 8 for.

SourceBusy is INFO, never a failed execution, and no retry_on on any job. Both retry ladders already exist and are ledger-aware (Ingestion::PollState, Enrichment::EntityState); a second uncoordinated ladder would re-poll a source whose backoff was just written. The 60-second tick is the retry.

Solid Queue's schema lands as a migration and the dump. db:prepare loads a schema file only for an uninitialized database, and PR 2's version-0 db/queue_schema.rb placeholder already created schema_migrations in every development queue database — verified against one, which returned the single row 0. A schema-file-only change would have left those databases empty and crash-looped the worker on a reviewer's second docker compose up.

shutdown_timeout 20s / stop_grace_period 30s is derived from §2A's pinned defaults (open 5 + read 15 is the longest an attempt already holding the gate can run), the way RequestGate::WAIT_SECONDS is. A job still waiting for the gate has reserved nothing and is safe to kill.

Testing performed

1,383 examples, 0 failures; RuboCop clean; Brakeman 0 warnings; bundler-audit clean.

New spec/recovery/ covers §12's recovery list:

• pending-work recovery — ingest, clear the queue (the crash), reconcile: one cycle per class, no entity row touched, and the recovered work completes when the cycles run.
• duplicate delivery — the same job performed twice leaves state byte-identical, spends one request and one reservation, and reports idle rather than pretending.
• crashed worker's lease — a lease with no worker behind it (no ensure runs in a real crash): idle and free until Claim#lease_seconds elapses, then enriched, with no cleanup code anywhere.
• advisory-lock release on session death — pg_terminate_backend on a second session, not a cooperative close, for the source lock and the request gate, plus a full poll that fails SourceBusy before the kill and completes after it. The residual gap (the dying session cannot be RSpec's own pooled connection) is stated in the file.
• source contention — the tick under a held lock: no raise, no re-enqueue, no run row, no request, no budget spent, and the other source still polled.

Plus spec/jobs/ for each job, spec/services/github/enrichment/dispatch_spec.rb for the enqueue rule, and spec/queue/ — the only examples that touch github_push_ingestor_queue_test, with spec/job_boundary_spec.rb keeping it that way (mirroring network_boundary_spec.rb).

CI gains bin/rails solid_queue:check and a fixture-mode timeout --signal=TERM 20 bin/jobs supervisor smoke — boot, recurring registration, clean shutdown; it does not claim to prove a tick fired.

Docker verification

GITHUB_MODE=fixture docker compose up --build — db, setup, web, worker; the scheduler registered all three recurring tasks, and within 60 seconds the log showed job.completed for PollEventSourceJob (sources_due: 1, one run_id), ingestion.run_started/run_completed, enrichment.dispatched with the per-class summary, and enrichment.completed / enrichment.failed carrying their job ids.

Recovery drill: stopped the worker, set the entities back to pending, TRUNCATE solid_queue_jobs CASCADE (the crash), started it again at 23:31:17 — the reconciler tick at 23:32:00 rediscovered the work from the entity rows and the cycles ran. docker compose restart worker restarts cleanly (no stale pidfile). docker compose run --rm test (image, no bind mount) is green.

Documentation updates

docs/adr/0008-post-commit-enqueue-and-entity-scoped-reconciliation.md; README gets a Continuous ingestion section, the worker service in Quick start, the job log vocabulary, queue-inspection and recovery-drill commands, and a plain statement that docker compose up at the default GITHUB_MODE=live now spends real unauthenticated quota continuously. Stale forward references in source_lock.rb, enrichment_runner.rb, bin/enrich, config/database.yml, spec/support/webmock.rb and the compose header corrected. IMPLEMENTATION_PLAN.md unchanged — it is frozen.

Known limitations

• Enrichment latency after a lost enqueue is up to 60 seconds, and a deep backlog drains at the reconciler's cadence rather than in a burst; bin/enrich --limit N remains the operator's handle.
• The session-death test kills a second session, not the RSpec process's own connection; §15's container kill closes that gap and is PR 11's.
• The CI supervisor smoke proves boot and clean shutdown, not that a tick fired — the 60-second schedule would treble the step.
• Processing is at-least-once execution with effectively-once persisted outcomes. A job may run twice; the second run changes nothing. No exactly-once claim.

🤖 Generated with Claude Code