PR 6 — Poll budget and scheduling

[batbrainy]

Pull Request

Linked issue

Closes #16 — IMPLEMENTATION_PLAN.md §13, "PR 6 — Poll budget and scheduling".

Problem

A poll was an unconditional act. IngestionRunner fetched exactly one page, never sent
If-None-Match, never read a Link header, and wrote nothing to any event_sources
column. The five scheduling columns, event_sources.etag and
github_api_budget.global_blocked_until all existed and were all dead data — PR 4's
BudgetLedger read the global block but its own comment said it would never set it,
never derive class blocking for scheduling, and never handle a secondary limit.

Scope

Included — the issue's checklist, in full:

• effective_poll_time as the max of five independent components
  (cadence_due_at, poll_floor_until, retry_not_before_at, global_blocked_until,
  derived poll_class_blocked_until), in Github::PollSchedule.
• Link-header pagination with budget-bounded stops — cap / allowance / no-next-link /
  empty page — and explicitly no stop-on-known-event, in Github::Ingestion::PageLoop
  over a new Github::LinkHeader.
• Page-one ETag and its conditional request. A 304 performs no processing and its
  reservation stays debited.
• global_blocked_until for §10's three truly-global conditions only, decided by a new
  Github::RateLimitPolicy and recorded by BudgetLedger#block_globally!. Class blocking
  stays derived from the counters.
• Secondary limits handled globally from Retry-After, clamped to 60 s–1 h, with
  jittered backoff when the header is absent, zero, or an HTTP-date.
• Persisted poll state, in Github::Ingestion::PollState.
• event_sources.status vocabulary (idle | failed) with a CHECK constraint.
• The required validation gate: script/probe_304.sh plus
  docs/evidence/2026-07-30-unauthenticated-304-quota-probe.md.
• ADR 0006, README, .env.example, fixture-corpus docs, and every stale "PR 6 will…"
  comment left behind by PRs 4 and 5.

Deliberately not included — effective_enrichment_time, fairness shares and enrichment
jobs (PR 7, #17); the recurring task and worker container (PR 8, #18); the rich /status
endpoint (PR 10). Nothing fires the cadence yet — the one-shot is still the only thing
that polls.

IMPLEMENTATION_PLAN.md is untouched. It is frozen and forward-referential by design,
and §14 pins the execution summary to PR 12. The probe re-run confirmed the observation §10
records, so no amendment was required — had it contradicted it, §10's accounting decision
and ADR 0004 would both have needed revision with the tradeoff stated. That is the case
CLAUDE.md's rule is for.

Technical decisions

Five components, never one collapsed timestamp. ADR 0006 records this. A single
next_poll_at cannot answer the two questions §9 asks of it: --force could not tell which
part it may bypass, and a routine X-RateLimit-Reset on a healthy 200 would defer every
poll to the top of the hour. next_poll_at is written as a cache of the answer and is
never read back to make a decision.

--force removes exactly one term. PollSchedule::FORCEABLE is that claim in one
place. force is never passed to the page loop, the executor, the ledger, the policy or the
poll-state writer, so the other four stay binding by construction; SourceLock is acquired
before it is read; and reserve! knows nothing about it, which is a second, independent
guarantee that a forced run cannot overspend.

The cadence gate runs inside the source lock, and re-reads the row. The caller's
EventSource is loaded before mutual exclusion exists, so two processes would otherwise
both read a stale cadence_due_at, serialize on the lock, and poll back to back — defeating
the guarantee the lock exists to provide.

A run row exists iff the process tried to reach GitHub. A poll the schedule turns away
writes no ingestion_runs row and no source state. Under PR 8's recurring task the
alternative is dozens of zero-count rows an hour diluting every counter in the table. The
cost, stated in ADR 0006: skipped cycles are countable from the logs only.

A truncated walk is completed, not failed or deferred. If page 3 is denied after
pages 1–2 committed, "the attempt happened and did not produce usable events" is false, and
deferred would drop a run holding real events out of IngestionRun::SUCCESSFUL_STATUSES.
The truncation is visible in pages_fetched and stop_reason.

Two hardening fixes in BudgetLedger, both for bugs this PR's new writer makes
reachable rather than pre-existing failures. apply_observation dispatched on the
window_status string; once a global block could write a third value, a block landing on
an uninitialized window sent the next good response into snapshot.reset_at < nil →
ArgumentError, on a row neither rollover predicate could ever fix. Both it and the
enrichment eligibility guard now key on the fact (reset_at, window_initialized_at)
rather than the label.

No new columns. §7 lists every column both tables need and all of them already exist.
Rejected alternatives, with reasons, in ADR 0006: a stored poll_class_blocked_until; a
second class writing github_api_budget; a secondary_limit_count counter for exponential
escalation — see Known limitations.

One migration, and one only. The event_sources.status CHECK, because PR 6 is the PR
that defines the vocabulary and PR 3's migration comment said so. IngestionRun separately
invites a CHECK on ingestion_runs.status "whenever migrations are next touched" — that is
an invitation, not a requirement, and taking it here would put a schema change in this diff
that no behaviour in it depends on. Parked; considered and declined, not missed.

Testing performed

910 examples, 0 failures (from 764 on main), rubocop clean across 144 files, brakeman
0 warnings, bundler-audit clean.

New: link_header_spec, poll_schedule_spec (§12's effective_poll_time unit test, with
no database — including one example per constraint --force must still obey),
poll_backoff_spec, rate_limit_policy_spec, page_loop_spec (D5 and D6 in full),
poll_state_spec (the write matrix row by row).

Extended: the runner spec gains the cadence gate, --force end to end, a full multi-page
poll, and persisted poll state per terminal outcome; budget_ledger_spec gains
#block_globally! and both hardening fixes; the model and schema specs gain the vocabulary
and the derived class blocking.

Changed, and worth a reviewer's attention:

• "sent no conditional header, so this 304 came from the corpus script and not from us"
  inverts into "sends the ETag the first poll stored". PR 5 wrote that example
  precisely to be inverted here.
• "holds no application transaction across the fetch" observed only the last fetch. It
  now collects over a three-page walk — the page most likely to be wrapped in a transaction
  is the second one, after the writer has run.
• The budget-denial group split in two. An exhausted poll allowance is now caught
  pre-flight as a §9 scheduling component, which is strictly better — no lock work, no
  run row, no log line per tick. The ledger's denial path stays live and separately covered
  for what the schedule cannot see coming.

Docker verification

GITHUB_MODE=fixture …  ingest            → 4 created, 3 quarantined, 1 ignored
GITHUB_MODE=fixture …  ingest            → Ingestion deferred until … — cadence_due_at
sleep 60 && …          ingest --force    → 4 duplicates, 0 created
GITHUB_FIXTURE_SCENARIO=paginated MAX_PAGES_PER_POLL=3 …
                                          → 3 pages, 11 seen, 6 created, 1 duplicate

Two findings came out of running these rather than assuming them, and both changed the
code or the docs:

1. --force correctly refuses to bypass the server floor. The corpus sends
   x-poll-interval: 60 exactly as GitHub does, so an immediate forced replay defers with
   poll_floor_until. The README now documents the sleep 60 and explains that the wait
   is the demonstration.
2. A deferred run printed seven zero counters, which reads as a run that produced
   nothing rather than one that never happened. Suppressed.

POLL_INTERVAL_SECONDS, MAX_PAGES_PER_POLL, ENABLED_LIVE_SOURCE_COUNT and
RATE_LIMIT_RESERVE were not in the compose env anchor, so setting them in a shell had
no effect inside the container and the pagination walk was not reproducible as documented.
Added, with a spec asserting the anchor's defaults match Configuration::DEFAULTS.

Documentation updates

README (status, --force, pagination, "Why a 304 costs a request here", expected latency,
fixture rate-limit recovery, logging, request-path diagram, env table); ADR 0006; a
one-sentence evidence pointer appended to ADR 0004; .env.example; the fixture corpus
README; docs/evidence/.

Known limitations

• The probe transcript is a template. Done. The probe was run against
  api.github.com and the finding reproduced: x-ratelimit-used 4 → 5 → 6 across two
  304s, resource: core throughout. §10's validation gate is satisfied and no plan
  amendment was required. Two corrections came out of running it: the script used an empty
  bash array under set -u, which aborts on the bash 3.2 macOS ships; and the transcript's
  claim that GitHub's Date is authoritative for ordering is false — all three responses
  echo the cached representation's Date, so only the local clock orders them. Both are
  documented as observed.
• Secondary-limit backoff does not escalate across processes. §10 asks for "exponential
  backoff when the header is absent"; there is no persisted counter to be exponential in,
  and §7's github_api_budget column list is closed. In practice this is a 60-second floor
  with jitter, escalating only via GREATEST(existing, new) — which a block still in force
  makes rare, because the block itself prevents the request that would escalate it. Adding a
  column was the alternative; it would put a column §7 does not list into the schema.
• A pre-flight deferral writes no run row, so poll history is a history of attempts.
• budget.global_block_cleared records the write, not the moment the block stopped
  biting — blocking is derived from the timestamp, so an expired block is already inert
  without any write.
• Capture is bounded and sampled, not complete. MAX_PAGES_PER_POLL is 1 by default,
  documented latency is 30 s–6 h against a 5-minute cadence, and the /events window holds
  ~300 events. Pagination deepens one poll within the budget; it does not backfill.
• Another client behind the same public IP can still exhaust the window. The ledger
  coordinates this application only.

🤖 Generated with Claude Code

[batbrainy]

Reviewed the poll scheduling and budget paths. I left two inline comments on behavior that looks worth fixing before merge.