Why
At the Phase 4 cutover (CannObserv/watcher#241), the cluster loses its only adaptive
politeness mechanism, and per the boundaries charter the replacement belongs here.
Watcher today escalates on 429: report_rate_limited_for_domain raises
Domain.current_interval (capped at BACKOFF_MAX_INTERVAL = 60.0s) and decays it after a
quiet window. That loop depends on Watcher observing the 429 on its own fetch path — which
stops existing at cutover. On this side, a 429 is TransientFetchError → retried at the
reclaim cadence, and because it is transient it never becomes terminal, so no
fetch_failed fact ever fires (#9 §3 deferred non-terminal facts). Watcher cannot see
429s at all; content.fetch-policy carries only static spacing.
Post-cutover state: HostPacer enforces a fixed per-host interval. An origin that
starts rate-limiting gets the same cadence it was already getting, forever, except for the
accidental spacing of the ~60s reclaim retry on the specific command that hit the 429 —
sibling commands to the same host are not slowed at all.
Where it belongs
The charter's tests resolve this cleanly: origin tolerance is cross-command coordination
over a resource only the fetcher can see (test 2 → Replicator), and the escalation state
is in-memory derived, rebuildable by replay — cold-start-polite is the safe direction,
same as HostPacer's existing map. No database, no new stream, no policy question: the
floor stays the issuer's number (or the env default); this issue is about temporarily
rising above the floor when the origin says so.
That is also why the alternative — non-terminal fetch_failed facts feeding Watcher's
backoff, which then republishes policy — was not chosen: it routes a mechanism decision
through a policy channel and couples the reaction time to the policy republish cadence.
(#9 §3 stays deferred on its own merits; an issuer may still want visibility of a
struggling command someday. That is observability, not control.)
Sketch
HostPacer grows a report_rate_limited(host) that multiplies the host's effective
interval (bounded — Watcher uses ×2 up to 60s; matching those constants keeps cutover
behaviour-neutral), and a decay after a quiet window, mirroring Watcher's
_maybe_decay_backoff. The byte path calls it where _raise_for_status sees a 429 (and
plausibly 503 + Retry-After, which is the origin telling us the number — worth
honouring when present, capped). Policy lookups keep supplying the floor; escalation never
goes below it.
Not in scope
Refs
Why
At the Phase 4 cutover (CannObserv/watcher#241), the cluster loses its only adaptive
politeness mechanism, and per the boundaries charter the replacement belongs here.
Watcher today escalates on 429:
report_rate_limited_for_domainraisesDomain.current_interval(capped atBACKOFF_MAX_INTERVAL = 60.0s) and decays it after aquiet window. That loop depends on Watcher observing the 429 on its own fetch path — which
stops existing at cutover. On this side, a 429 is
TransientFetchError→ retried at thereclaim cadence, and because it is transient it never becomes terminal, so no
fetch_failedfact ever fires (#9 §3 deferred non-terminal facts). Watcher cannot see429s at all;
content.fetch-policycarries only static spacing.Post-cutover state:
HostPacerenforces a fixed per-host interval. An origin thatstarts rate-limiting gets the same cadence it was already getting, forever, except for the
accidental spacing of the ~60s reclaim retry on the specific command that hit the 429 —
sibling commands to the same host are not slowed at all.
Where it belongs
The charter's tests resolve this cleanly: origin tolerance is cross-command coordination
over a resource only the fetcher can see (test 2 → Replicator), and the escalation state
is in-memory derived, rebuildable by replay — cold-start-polite is the safe direction,
same as
HostPacer's existing map. No database, no new stream, no policy question: thefloor stays the issuer's number (or the env default); this issue is about temporarily
rising above the floor when the origin says so.
That is also why the alternative — non-terminal
fetch_failedfacts feeding Watcher'sbackoff, which then republishes policy — was not chosen: it routes a mechanism decision
through a policy channel and couples the reaction time to the policy republish cadence.
(#9 §3 stays deferred on its own merits; an issuer may still want visibility of a
struggling command someday. That is observability, not control.)
Sketch
HostPacergrows areport_rate_limited(host)that multiplies the host's effectiveinterval (bounded — Watcher uses ×2 up to 60s; matching those constants keeps cutover
behaviour-neutral), and a decay after a quiet window, mirroring Watcher's
_maybe_decay_backoff. The byte path calls it where_raise_for_statussees a 429 (andplausibly 503 +
Retry-After, which is the origin telling us the number — worthhonouring when present, capped). Policy lookups keep supplying the floor; escalation never
goes below it.
Not in scope
policy numbers are unaffected.
Refs
docs/contracts/replicator-boundaries.md— mechanism/policy split; the three ownership testssrc/core/rate_limiter.py(report_rate_limited_for_domain,BACKOFF_MAX_INTERVAL, decay window)