Classify redis OutOfMemoryError as transient — the capped broker made it a live publish path (archiver#129)

[gregoryfoster]

Why

The Archiver-operated broker now runs with an explicit maxmemory cap (CannObserv/archiver#128 → archiver#129). Before that change maxmemory 0 + noeviction was inert: no write was ever refused, so the "error and let the producer retry" degradation the config implied never engaged. With a cap in place, memory pressure surfaces as OOM command not allowed when used memory > 'maxmemory' on XADD, instance-wide — every producer on the broker is refused at the same moment, whichever stream caused the pressure.

That is a new error path on Replicator's publish call that did not exist when the byte path was written.

The bug

redis.exceptions.OutOfMemoryError subclasses ResponseError. ResponseError is not in src/worker/loop.py's _TRANSIENT_ERRORS, so an OOM raised by handler.py::_publish on the blob_available XADD falls through to _handle_unclassified and counts against REPLICATOR_MAX_DELIVERY_ATTEMPTS — 5 reclaims at the 60 s REPLICATOR_CLAIM_MIN_IDLE_MS cadence, so roughly five minutes of incident exhausts the ceiling.

This is precisely the bug Archiver hardened against in the same change, and for the same reason: ResponseError subclassing makes it an error of omission.

Why it has not bitten yet

Instance-wide OOM refuses the <topic>.dlq XADD too, so during a steady incident the dead-letter write fails as well, _close raises, and the message survives as a cycle failure with backoff. That is luck, not design, and it does not hold in two cases:

1. A flapping cap. Memory oscillating around maxmemory burns the delivery counter without ever completing the DLQ write. times_delivered only ever advances, so after the incident any later unclassified failure of any kind dead-letters immediately, with none of the 5-attempt grace the ceiling exists to provide.
2. The clearing edge. A command whose publish is refused at attempt ≥ 5 and whose DLQ write then succeeds as memory frees is dead-lettered and closed with fetch_failed(handler_error) — for bytes that stored successfully and are sitting on disk. The issuer is told its content will never arrive while the blob exists.

Fix

Add OutOfMemoryError to _TRANSIENT_ERRORS in src/worker/loop.py:

from redis.exceptions import OutOfMemoryError

Transient is the correct classification on the same argument the tuple's existing members carry: the command is valid, the failure is someone else's incident, and a transient failure is exempt from the delivery ceiling so a long-but-genuine outage can never silently drop good work. The durable record of intent is already the PEL — the message stays unacked, claim_stale re-runs it, and content-addressed storage makes the re-run a no-op.

Watch the tuple's existing NOTE while doing this: the gate assumes co-core-aio propagates redis exception types unwrapped.

Not in scope

• The fetch_failed reporter's swallow stays as it is. src/worker/reporter.py deliberately absorbs a failed fact-publish because the DLQ entry is already the durable record; an OOM there loses a terminal fact and the issuer's reaper is the designed backstop. Changing it would burn the delivery ceiling to reach the same DLQ minutes later.
• The content.fetch-policy reader (Consume content.fetch-policy — replace the single env default with Watcher's per-host numbers (cannobserv#285) #19) is unaffected. XREAD is not a denyoom command.

Producer-durability answer for archiver's deploy/README.md

The table currently reads unasserted for both streams. Once this lands:

Stream	Replicator's producer durability under OOM
content.fetch	n/a — Replicator does not produce on it. The only writer in this repo is scripts/seed_fetch.py, an operator tool.
content.blobs	Durable via the consumer-group PEL. _publish re-raises, the message stays unacked, claim_stale re-runs it. No DB and no outbox, by charter (docs/contracts/replicator-boundaries.md).

Acceptance

• OutOfMemoryError classified transient in src/worker/loop.py
• Test: a handler raising OutOfMemoryError returns Outcome.RETRY and does not touch the delivery counter or the DLQ
• Reply on Broker: content.fetch-policy is a fourth stream with no consumer group — monitoring gap + producer-controlled retention under noeviction (cannobserv#285) archiver#128 so the deploy/README.md cells can move off unasserted

Refs

• Broker: content.fetch-policy is a fourth stream with no consumer group — monitoring gap + producer-controlled retention under noeviction (cannobserv#285) archiver#128 · #128 fix: give noeviction a ceiling and treat broker OOM as transient archiver#129 — the broker change and Archiver's own producer hardening
• Assign operational ownership of the Redis change bus (no service operates redis-server today) archiver#109 — broker ownership
• Broker: periodic bus-health timer (XLEN + memory headroom) — deferred from #128 archiver#130 — deferred broker health monitoring
• Consume content.fetch-policy — replace the single env default with Watcher's per-host numbers (cannobserv#285) #19 — surfaced during that review