From 7cb8e9f108f1382879733097d521f50bc711d6ad Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Thu, 6 Aug 2026 14:38:33 +0000 Subject: [PATCH 1/5] #24 docs: promote the fact schemas out of the request-options subsection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `### Request options` had no sibling heading after it, so both fact schemas — `BlobAvailableEvent` and `FetchFailedEvent` — were structurally nested under a subsection about `headers` and `timeout_seconds`. A split that reads the heading tree would have relocated the wire schemas into a lookup annex. Adds `### The command`, `### The success fact`, `### The failure fact` under `## The payload`, and moves the `occurred_at` tz blockquote up to the command it constrains. Content verbatim; prove-no-loss reports 0 unaccounted lines. Co-Authored-By: Claude Opus 5 (1M context) --- .../content-fetch-issuer-contract.md | 20 ++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/docs/contracts/content-fetch-issuer-contract.md b/docs/contracts/content-fetch-issuer-contract.md index b0886f1..b37aa46 100644 --- a/docs/contracts/content-fetch-issuer-contract.md +++ b/docs/contracts/content-fetch-issuer-contract.md @@ -77,6 +77,8 @@ would shard on. ## The payload (inside `payload`) +### The command + **Command — `content.fetch`, `ContentFetchCommand`** (`co_core.pure.models.changes`): | Field | Type | Notes | @@ -89,6 +91,14 @@ would shard on. | `headers` | `dict[str, str] \| None` | co-core ≥ 0.7.3, **honoured since #11**. Merged over the fetcher's defaults, issuer wins. Guards below | | `timeout_seconds` | `float \| None` | co-core ≥ 0.7.3, **honoured since #11**. Seconds; bounded above. `None` = the driver default | +> **`occurred_at` must carry a timezone.** Since co-core v0.7.2 (cannobserv#273) it is an +> `AwareDatetime` on every payload: a **naive** value is rejected fail-loud rather than assumed +> to be UTC, because "assume UTC" corrupts the instant when a producer stamps a naive local time. +> An aware non-UTC value is normalized. A naive one fails `from_wire` inside Replicator's `read`, +> which means it dead-letters as a malformed frame and is one of the rows that stay **silent** — +> the same shape as an issuer that flattened the envelope. `datetime.now(UTC)`, not +> `datetime.now()`. + ### Request options: what Replicator will send, and what it refuses (#11) `headers` and `timeout_seconds` shape the individual fetch. Both are optional and both default to @@ -130,13 +140,7 @@ remains the sole dedupe key and the sole correlator. Two commands differing only fetch occasions (MUST-1 unchanged); a *redelivery* carrying different options is still the same command and is still deduped. -> **`occurred_at` must carry a timezone.** Since co-core v0.7.2 (cannobserv#273) it is an -> `AwareDatetime` on every payload: a **naive** value is rejected fail-loud rather than assumed -> to be UTC, because "assume UTC" corrupts the instant when a producer stamps a naive local time. -> An aware non-UTC value is normalized. A naive one fails `from_wire` inside Replicator's `read`, -> which means it dead-letters as a malformed frame and is one of the rows that stay **silent** — -> the same shape as an issuer that flattened the envelope. `datetime.now(UTC)`, not -> `datetime.now()`. +### The success fact **Fact — `content.blobs`, `BlobAvailableEvent`**: @@ -209,6 +213,8 @@ lives here rather than in Watcher. Three details are the whole value of them: > existing fact rather than a new event type, so an issuer branching on `terminal` first is already > forward-compatible with it.) +### The failure fact + **Fact — `content.blobs`, `FetchFailedEvent`** (co-core ≥ 0.7.2, cannobserv#270): | Field | Type | Notes | From 149e70a8688349a1952b71491b7ffb685acedef9 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Thu, 6 Aug 2026 15:16:57 +0000 Subject: [PATCH 2/5] #24 docs: split the issuer contract into a contract and a reference MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The contract was 14,772 tokens against the 10,000 per-doc budget, and it is the doc most likely to be read by an agent working in a sibling repo — the case where an over-budget file does the most damage, since there is no cheaper local surface to fall back on. The path sibling repos link stays the normative core, not a stub: a stub would cost every issuer-side agent a second read to reach the MUSTs, which is the cost the split exists to remove. Core keeps the frame, the payload shapes, the seven MUSTs, and the guarantee/non-guarantee pair — 14,772 -> 9,966. content-fetch-issuer-reference.md (7,349, new, equally normative) takes what an issuer looks up rather than reads through: the request-options refusal list, the reasoning behind the enriched blob_available fields and the #17 conditional-GET trap, the failure taxonomy, the four silent conditions, reading the DLQ, the provenance and trust posture, the co-core version history, what the envelope key is operationally for, and pacing at the deployed defaults. docs/plans/2026-07-31-fetch-failed-fact-settled.md takes the resolved silence-as-failure question. Archival, so out of the measured surface. Moves are verbatim. prove-no-loss --base d4a5e02 accounts for 471 of 475 lines; the four it does not are the intra-document pointers whose referent moved, so "see the taxonomy below" and "the refusal table above" were disproven by the move and are now cross-file links. Every live doc is now under budget; AGENTS.md is 5,896 of 6,000. No dead links, no orphans, every anchor resolves. Manifest updated and re-embedded in the same change, per the #19 CR #17 rule. Refs updated: AGENTS.md (Detail Docs + the Bus Conventions signpost), README.md, docs/ARCHITECTURE.md, docs/COMMANDS.md, docs/contracts/replicator-boundaries.md, src/core/errors.py (its docstring names the file for the taxonomy specifically, which is now the reference). Closes #24 Co-Authored-By: Claude Opus 5 (1M context) --- .skills/context-token-ratio | 2 +- .socraticodecontextartifacts.json | 9 +- AGENTS.md | 8 +- README.md | 6 +- docs/ARCHITECTURE.md | 2 +- docs/COMMANDS.md | 2 +- .../content-fetch-issuer-contract.md | 320 ++++-------------- .../content-fetch-issuer-reference.md | 300 ++++++++++++++++ docs/contracts/replicator-boundaries.md | 1 + .../2026-07-31-fetch-failed-fact-settled.md | 33 ++ src/core/errors.py | 2 +- 11 files changed, 429 insertions(+), 256 deletions(-) create mode 100644 docs/contracts/content-fetch-issuer-reference.md create mode 100644 docs/plans/2026-07-31-fetch-failed-fact-settled.md diff --git a/.skills/context-token-ratio b/.skills/context-token-ratio index e72716a..5d8213c 100644 --- a/.skills/context-token-ratio +++ b/.skills/context-token-ratio @@ -1 +1 @@ -2.46 +2.47 diff --git a/.socraticodecontextartifacts.json b/.socraticodecontextartifacts.json index 2cb0b8a..f1cf795 100644 --- a/.socraticodecontextartifacts.json +++ b/.socraticodecontextartifacts.json @@ -13,12 +13,17 @@ { "name": "design-plans", "path": "./docs/plans/", - "description": "Founding MVP design (2026-06-25, adopted). The command/fact contracts as shipped in co-core v0.7.0 — ContentFetchCommand (URL-addressed, command_id as the wire idempotency key) and BlobAvailableEvent (content_fingerprint, blob_uri, size_bytes, media_type, url, optional command_id) — the stream taxonomy and DLQ naming, the temp-storage interface, the two-level idempotency argument, why fingerprint parity dissolves when Replicator is the sole fetcher, explicit MVP scope cuts, and the build sequence. The founding plan's three open questions were settled 2026-07-31 in a companion doc: blob_uri is a file:// URI over a two-level sharded, content-addressed path (///.bin, constant .bin extension, temp-file + os.replace so presence means complete); the MVP command issuer is an in-repo scripts/seed_fetch.py rather than Watcher or Archiver; and the MVP boundary is stop-at-fact, with archiver-writeback reclassified from MVP+ to won't-do — Archiver consumes blob_available and writes the SourceRevision itself (archiver#118, which also carries the info_source_id contract gap). Blob retention is out of MVP scope and tracked in #5." + "description": "Founding MVP design (2026-06-25, adopted). The command/fact contracts as shipped in co-core v0.7.0 — ContentFetchCommand (URL-addressed, command_id as the wire idempotency key) and BlobAvailableEvent (content_fingerprint, blob_uri, size_bytes, media_type, url, optional command_id) — the stream taxonomy and DLQ naming, the temp-storage interface, the two-level idempotency argument, why fingerprint parity dissolves when Replicator is the sole fetcher, explicit MVP scope cuts, and the build sequence. The founding plan's three open questions were settled 2026-07-31 in a companion doc: blob_uri is a file:// URI over a two-level sharded, content-addressed path (///.bin, constant .bin extension, temp-file + os.replace so presence means complete); the MVP command issuer is an in-repo scripts/seed_fetch.py rather than Watcher or Archiver; and the MVP boundary is stop-at-fact, with archiver-writeback reclassified from MVP+ to won't-do — Archiver consumes blob_available and writes the SourceRevision itself (archiver#118, which also carries the info_source_id contract gap). Blob retention is out of MVP scope and tracked in #5. Also holds the relocated resolution of the fetch_failed open question (2026-07-31, moved out of the issuer contract in #24): the consumer arrived in watcher#241, co-core shipped FetchFailedEvent in v0.7.2, and the resolution deliberately did NOT ship non-terminal facts (#9 §3, unbounded growth on an untrimmed broadcast stream) and deliberately left the rows with no usable command_id DLQ-only, because a fact naming no command closes nothing and one naming a foreign command_id would be worse than silence." }, { "name": "issuer-contract", "path": "./docs/contracts/content-fetch-issuer-contract.md", - "description": "Normative contract for any service publishing content.fetch commands (seed_fetch.py today, Watcher from Phase 4) — and its permanent home, linked to rather than copied by issuer-side repos. Opens with the frame shape, because getting it wrong is the likeliest and quietest mistake: what lands on the stream is a co-core envelope produced by to_wire (key, payload as JSON, event_type, schema_version, occurred_at, content_type), never the model's fields flattened, and an XADD of top-level command_id/url fails from_wire into content.fetch.dlq silently. key is derived by to_wire (command_id for a command, content_fingerprint for a fact) and is not what Replicator dedupes on — that is the decoded payload.command_id — but it is what makes a DLQ entry correlatable, so an issuer can tail content.fetch.dlq with a plain XREAD and close a pending entry deterministically instead of waiting out a timeout. Then why the wire carries no info_source_id and what that pushes onto the issuer: mint a fresh ULID command_id per fetch occasion (a resource-stable id is silently deduped away for 24h by replicator:cmd:); url is one-to-many onto InfoSources and therefore not a correlation key; persist the command_id -> domain map before publishing, outbox-style; correlation must be idempotent because a crash between publish and the dedupe SET emits a second fact under the same command_id; consumers must not dedupe their inbox on content_fingerprint, which is storage identity rather than correlation identity; there is no fetch_failed fact, so silence is the only failure signal and the issuer needs its own pending-entry reaper that re-issues rather than concluding loss; blob_uri is a host-local file:// URI whose 7-day TTL runs from last fetch-reference, not last consumer read. Carries the full failure taxonomy (which conditions dead-letter to content.fetch.dlq silently, which retry indefinitely, which park in the PEL under the byte ceiling), what Replicator does and does not guarantee, the unauthenticated-capability trust posture resting on Archiver's localhost bus ACL, and the deferred fetch_failed fact as a co-core change. Written for #8." + "description": "Normative contract for any service publishing content.fetch commands (seed_fetch.py today, Watcher from Phase 4) — and its permanent home, linked to rather than copied by issuer-side repos. Split in #24 into this read-through half and the lookup half in issuer-reference; both are normative and the path sibling repos link is this one. Opens with the frame shape, because getting it wrong is the likeliest and quietest mistake: what lands on the stream is a co-core envelope produced by to_wire (key, payload as JSON, event_type, schema_version, occurred_at, content_type), never the model's fields flattened, and an XADD of top-level command_id/url fails from_wire into content.fetch.dlq silently. key is derived by to_wire (command_id for a command, content_fingerprint for a fact, command_id:occurred_at for a failure) and is not what Replicator dedupes on — that is the decoded payload.command_id. Then the payload section: the command table with the tz-aware occurred_at rule, a summary of the request-options guards (refused not adjusted, identity untouched) pointing at the reference for the refusal list itself, the BlobAvailableEvent field table with the four rules for reading the enriched fields, and the FetchFailedEvent table with its reason tokens. Then the seven MUSTs, which are the contract proper: mint a fresh ULID command_id per fetch occasion (a resource-stable id is silently deduped away for 24h by replicator:cmd:); persist the command_id -> domain map before publishing, outbox-style; url is one-to-many onto InfoSources and therefore not a correlation key; correlation must be idempotent because a crash between publish and the dedupe SET emits a second fact under the same command_id; consumers must not dedupe their inbox on content_fingerprint, which is storage identity rather than correlation identity; handle fetch_failed but keep a reaper anyway, because four conditions still produce nothing; and copy the bytes before the blob's 7-day TTL, which runs from last fetch-reference rather than last consumer read. Closes with what Replicator guarantees (store-then-publish, announce-then-ack, command_id echoed, the fingerprint is definitional, at-least-once) and what it does not (no non-terminal fact, no latency bound, no issue-rate promise under per-host pacing, no ordering, no cross-command dedupe, no retention on content.blobs). Written for #8, split for #24." + }, + { + "name": "issuer-reference", + "path": "./docs/contracts/content-fetch-issuer-reference.md", + "description": "Lookup half of the content.fetch issuer contract, split out in #24 so the contract itself stays readable start to finish. Equally normative — a rule does not become advisory by living here. Carries: the request-options rules (#11) — header names lower-cased before the merge so the issuer wins rather than httpx sending two User-Agent lines, surrounding whitespace dropped from a value and nothing else, and the full refusal list, where every entry is refused rather than adjusted with a terminal fetch_failed / invalid_request_options before any request goes out (hop-by-hop headers, host and content-length which httpx derives, any proxy-*, names that are not RFC 9110 tokens, values with any byte outside \\x20-\\x7e because a CRLF is request splitting, two names differing only in case, more than 32 headers or 8192 bytes of them, and timeout_seconds that is non-finite or over REPLICATOR_MAX_FETCH_TIMEOUT_SECONDS because the consume path is serial). The reasoning behind the six enriched blob_available fields: None means nobody said rather than the default, status_code is always 2xx on this fact so a branch on == 200 drops a 203, verbatim excludes surrounding whitespace and an over-long value is dropped rather than truncated, they are per-occasion values on a fingerprint-keyed fact, and the live conditional-GET trap where a matching validator earns a body-less 304 that Replicator still closes as a terminal fetch_failed (#17). The full failure taxonomy table — every condition, Replicator's action, and what the issuer sees — with the three classes of silent row analysed (no usable command_id, silent only while in flight, and the duplicate-command_id row that is silent because the command already succeeded). The four silent conditions in full. How to read the DLQ with a plain XREAD and no consumer group, and the synthesized-record anomaly that carries no key. The provenance and trust posture: content.fetch is an unauthenticated capability resting entirely on Archiver's localhost bus ACL, which headers widens, and the two properties an issuer can rely on (refused before any request goes out; header values never reach the journal). Plus the co-core version history, what the envelope key is operationally for, and how pacing behaves at the deployed defaults." }, { "name": "boundaries-charter", diff --git a/AGENTS.md b/AGENTS.md index 992bae4..5c151d2 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -144,9 +144,10 @@ Replicator is a **consumer** first. Follow the conventions co-core and the archi requires `--production` for the one combination the live worker consumes — a frame there is fetched for real. - **Two normative contracts bound the wire and the roadmap**, both under - `docs/contracts/` and linked from sibling repos. `tests/test_boundaries.py` - enforces eight charter invariants in CI; change a charter and its tests - together. + `docs/contracts/` and linked from sibling repos. The issuer contract splits + into a read-through half and a lookup half (`-contract` / `-reference`, #24); + both are normative. `tests/test_boundaries.py` enforces eight charter + invariants in CI; change a charter and its tests together. Blob paths, modes, and the retention sweep: [docs/STORAGE.md](docs/STORAGE.md). Fakeredis's divergences from the live broker, and the keys an integration run may @@ -229,4 +230,5 @@ are deliberately not JSON: [docs/STYLE.md](docs/STYLE.md). - [docs/COMMANDS.md](docs/COMMANDS.md) — every runnable command, with flags - [docs/SKILLS.md](docs/SKILLS.md) — vendored skill inventory and refresh procedure - [docs/contracts/content-fetch-issuer-contract.md](docs/contracts/content-fetch-issuer-contract.md) — what a `content.fetch` producer must do; normative, linked from issuer repos +- [docs/contracts/content-fetch-issuer-reference.md](docs/contracts/content-fetch-issuer-reference.md) — its lookup half: the refusal list, the failure taxonomy, the silent conditions, trust posture - [docs/contracts/replicator-boundaries.md](docs/contracts/replicator-boundaries.md) — what Replicator may become; run its three tests against any proposed capability diff --git a/README.md b/README.md index 699cfca..433b958 100644 --- a/README.md +++ b/README.md @@ -27,8 +27,10 @@ The founding design lives in **Issuing `content.fetch` commands?** Read [`docs/contracts/content-fetch-issuer-contract.md`](docs/contracts/content-fetch-issuer-contract.md) -first — it is the normative issuer contract and its permanent home. Publish through co-core's -`to_wire`, never hand-rolled fields; and because the wire carries no domain identity, correlation is +first — it is the normative issuer contract and its permanent home, with the refusal list, failure +taxonomy and trust posture in its companion +[`content-fetch-issuer-reference.md`](docs/contracts/content-fetch-issuer-reference.md). +Publish through co-core's `to_wire`, never hand-rolled fields; and because the wire carries no domain identity, correlation is entirely the issuer's job. Most ways of getting either wrong fail silently. ## Shape diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a4ca0f2..df78698 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -41,7 +41,7 @@ deploy/ — Systemd unit + deployment config Each rule `AGENTS.md` states in one line, with the reasoning that makes it non-negotiable: -- **The issuer contract is written down and lives here.** `docs/contracts/content-fetch-issuer-contract.md` is the normative statement of what a `content.fetch` producer must do — per-occasion `command_id`, `url` is not a correlation key, persist the `command_id → domain` map before publishing, correlate idempotently, and keep a reaper as a backstop for the outcomes no fact can carry. Issuer-side repos (Watcher, Phase 4) link to it rather than copying it. Anything asserted there is asserted about this repo's code; change one, change both (#8). +- **The issuer contract is written down and lives here.** `docs/contracts/content-fetch-issuer-contract.md` is the normative statement of what a `content.fetch` producer must do — per-occasion `command_id`, `url` is not a correlation key, persist the `command_id → domain` map before publishing, correlate idempotently, and keep a reaper as a backstop for the outcomes no fact can carry. Its companion `docs/contracts/content-fetch-issuer-reference.md` (#24) carries the lookup half — the request-options refusal list, the failure taxonomy, the four silent conditions, and the trust posture — and is equally normative. Issuer-side repos (Watcher, Phase 4) link to the contract rather than copying it. Anything asserted in either is asserted about this repo's code; change one, change both (#8). - **What Replicator is allowed to become is also written down.** `docs/contracts/replicator-boundaries.md` is the sibling charter: mechanism to Replicator, policy to the issuer, config over the bus, and an inbound admin HTTP API rejected by name. Run its three tests against any proposed capability, field, or setting before writing code — a database, domain vocabulary, or a write route is reached one defensible step at a time, not in one commit. `tests/test_boundaries.py` enforces eight invariants in CI, including the one that catches the regression review misses: an AST scan of `src/` for domain nouns in identifiers *and* string literals. Known violation recorded and pinned rather than omitted — `blob_uri` is host-local `file://` (#7). Change the charter and the tests together (#12). - **`content.blobs` carries both outcomes.** `blob_available` on success, `fetch_failed` on a command closed without bytes (#9, co-core cannobserv#270 — v0.7.2). One stream so an issuer's single consumer group sees either. The reason is named at the *raise site* (`PermanentFetchError.reason`), never recovered from a message string, because three unrelated permanent conditions share one exception type. Three rows stay DLQ-only and permanently silent, and only one of them for want of an id: a frame that did not decode, a command whose `command_id` is blank (refused before the fetch — an empty id would otherwise take the dedupe key `replicator:cmd:` and make every later blank-id command a silent no-op, CR #6), and a frame that decoded to a **non-command payload** — the last is unreportable not because it lacks a `command_id` but because any it carries is *another command's* (`BlobAvailableEvent`'s names one that succeeded), so a terminal fact keyed on it would contradict a fact the issuer already applied (CR #1). `_close` refuses a correlator-less report at the one choke point rather than at each call site. Non-terminal facts are deferred (#9 §3): the stream is broadcast and nothing trims it, so a fact per reclaim during an origin outage is unbounded growth. `src/worker/reporter.py`. - **`blob_available` carries the fetch, not just the bytes.** Six optional fields beyond the blob itself (#10, cannobserv#271 + #279 — v0.7.5): `final_url`, `status_code`, `fetched_at`, `content_type_raw`, `etag`, `last_modified`. Each is what Replicator holds at publish time and a broadcast consumer cannot recover now that fetching lives here rather than in Watcher. **`None` means nobody said it** — never a stand-in: `final_url` is never backfilled from `command.url` (an issuer could no longer tell "landed where I asked" from "nobody knows"), and `content_type_raw` is never backfilled with `DEFAULT_MEDIA_TYPE` (which is the value a consumer reads as "unknown, guess from the URL"). `media_type` keeps its normalized semantics beside the raw channel; the two are not interchangeable. `fetched_at` is stamped where the fetch *returns*, not at publish — `occurred_at` under a reclaim is minutes late. `status_code` is always 2xx on this fact, so it distinguishes 200 from 203/206 and is not a success branch. The passthroughs are **dropped over `MAX_HEADER_VALUE_LENGTH`, never truncated**: these are origin-controlled strings on a stream nothing trims, and a truncated ETag replayed in an `If-None-Match` is a validator that can never match. `src/worker/handler.py::_passthrough`. diff --git a/docs/COMMANDS.md b/docs/COMMANDS.md index abb58dc..885f060 100644 --- a/docs/COMMANDS.md +++ b/docs/COMMANDS.md @@ -86,7 +86,7 @@ uv run python -m scripts.seed_fetch \ The script rejects a malformed `--header` and a repeated name (exit 2) but deliberately does **not** pre-empt the worker's refusal list — sending a refused header is how the refusal is exercised against a live worker. The full list is in -[`docs/contracts/content-fetch-issuer-contract.md`](contracts/content-fetch-issuer-contract.md). +[`docs/contracts/content-fetch-issuer-reference.md`](contracts/content-fetch-issuer-reference.md). Watch the other side with `sudo journalctl -u replicator -f`. diff --git a/docs/contracts/content-fetch-issuer-contract.md b/docs/contracts/content-fetch-issuer-contract.md index b37aa46..887b451 100644 --- a/docs/contracts/content-fetch-issuer-contract.md +++ b/docs/contracts/content-fetch-issuer-contract.md @@ -8,6 +8,11 @@ from the code the day it was written. **Audience:** any service that publishes a `ContentFetchCommand`. Today that is [`scripts/seed_fetch.py`](../../scripts/seed_fetch.py). From Phase 4 it is Watcher. +**Companion.** [`content-fetch-issuer-reference.md`](content-fetch-issuer-reference.md) carries the +parts an issuer *looks up* rather than reads through — the request-options refusal list, the +reasoning behind the enriched `blob_available` fields, the failure taxonomy, and the trust posture. +Equally normative; split out in #24 so this file stays readable start to finish. Index at the end. + **Sibling document.** This settles the *wire*. [`replicator-boundaries.md`](replicator-boundaries.md) settles the *service* — what Replicator is allowed to become, and therefore which proposed fields this contract will never grow. Read that one before proposing a payload addition (#12). @@ -28,11 +33,9 @@ keeps Replicator clean, and it pushes the whole of correlation onto the issuer. follows fails *silently* when it is got wrong: no error, no dead letter, no log on Replicator's side — just a fact nobody can match, or a command that was never run. -Contracts settled in cannobserv#266 (co-core v0.7.0); the failure fact added in cannobserv#270 and -the tz-aware `occurred_at` in cannobserv#273, both shipped in **co-core v0.7.2**; the enriched -`blob_available` metadata in cannobserv#271/#279 and the command's request options in -cannobserv#272, shipped in **v0.7.3** and **v0.7.5**. Replicator requires **co-core ≥ 0.7.5**. -Founding rationale: +Version history and the co-core floor: +[the reference](content-fetch-issuer-reference.md#version-history). Replicator requires +**co-core ≥ 0.7.5**. Founding rationale: [`docs/plans/2026-06-25-replicator-mvp-design.md`](../plans/2026-06-25-replicator-mvp-design.md). --- @@ -70,10 +73,9 @@ cannot decode. It raises `BusMessageAnomaly` from inside the consumer's `read`, routes it to `content.fetch.dlq` — **silently, by the terms of MUST-6**. This is the single likeliest way to get the contract wrong, so it is stated before the field tables rather than after. -`key` is **not** load-bearing on the consume path: Replicator decodes `payload` and dedupes on -`payload.command_id`, never on the envelope key. Its value is operational — it is what makes a DLQ -entry correlatable without parsing JSON (see MUST-6), and it is what a future partitioned consumer -would shard on. +`key` is **not** load-bearing on the consume path: Replicator dedupes on `payload.command_id`, +never on the envelope key. Its value is operational — see +[the reference](content-fetch-issuer-reference.md#what-the-envelope-key-is-for). ## The payload (inside `payload`) @@ -83,7 +85,7 @@ would shard on. | Field | Type | Notes | |---|---|---| -| `schema_version` | `int` = 1 | Replicator supports **1 only**; anything else dead-letters (see the taxonomy) | +| `schema_version` | `int` = 1 | Replicator supports **1 only**; anything else dead-letters (see the [failure taxonomy](content-fetch-issuer-reference.md#failure-taxonomy-what-happens-and-what-the-issuer-sees)) | | `event_type` | `"content_fetch"` | | | `occurred_at` | `datetime` | **tz-aware UTC, enforced.** Not used for ordering or expiry by Replicator | | `command_id` | `str` | **The idempotency key and the sole correlator.** See MUST-1 | @@ -101,44 +103,23 @@ would shard on. ### Request options: what Replicator will send, and what it refuses (#11) -`headers` and `timeout_seconds` shape the individual fetch. Both are optional and both default to -`None`, which means **exactly** the pre-#11 behaviour: the fetcher's own `user-agent` and its 30 s -timeout, byte for byte. An issuer that sends neither is unaffected by any of the following. - -**Header names are lower-cased before the merge, and the issuer wins.** The fetch driver merges -`{"user-agent": , **your headers}` as a plain, case-*sensitive* dict. Without the fold a -capitalized `User-Agent` leaves both keys in the mapping and httpx sends **two** `User-Agent` field -lines — the default first, yours second — leaving the origin to decide which applies. Folding first -is what makes "issuer wins" a rule rather than a coincidence. Send `User-Agent` or `user-agent`; -either way exactly one line goes on the wire and it carries your value. - -**Surrounding whitespace is dropped from a value** and nothing else is: RFC 9110 excludes OWS from -a field value in the first place, so `" text/html "` is sent as `text/html`. Nothing *inside* a -value is touched. - -**Everything below is refused, not adjusted.** A refusal is a terminal -`fetch_failed` · `invalid_request_options` plus the DLQ, arriving before any request goes out — so -a refused command never reaches the origin at all. The reject-rather-than-fix posture is the same -one the `blob_available` passthroughs take: a header Replicator silently dropped, or a timeout it -silently shortened, is a change to your fetch that you cannot see and cannot account for in your -own fingerprints. - -| Refused | Why | -|---|---| -| `connection`, `keep-alive`, `proxy-connection`, `te`, `trailer`, `transfer-encoding`, `upgrade` | Hop-by-hop (RFC 9110 §7.6.1) — they describe one connection, which httpx and h11 own | -| `host`, `content-length` | Not hop-by-hop: httpx derives both. Overriding `host` addresses one origin while contacting another | -| Any `proxy-*` header | Configures the hop rather than the request | -| A name that is not an RFC 9110 token | `user agent`, `user:agent`, an empty name, anything non-ASCII — and a name with surrounding whitespace, which RFC 9110 forbids before the colon and which is therefore malformed rather than trimmable | -| A value with any byte outside `\x20`–`\x7e` | Printable US-ASCII and SP only. Excludes CR, LF, NUL, every other control character, HTAB, and all of `obs-text` (`\x80`–`\xff`) — narrower than RFC 9110 permits, deliberately. A CRLF here is request splitting | -| Two names differing only in case | Folding them would silently discard one. Refused even when the values agree — the rule is about the shape, not the values | -| More than **32** headers, or more than **8192 bytes** of them | 8 KiB is the common origin-side limit (nginx, Apache), so past it the far end answers an opaque 400. The constants in [`src/worker/handler.py`](../../src/worker/handler.py) are authoritative | -| `timeout_seconds` that is zero, negative, NaN, or infinite | Not a duration | -| `timeout_seconds` over `REPLICATOR_MAX_FETCH_TIMEOUT_SECONDS` (default **120**) | Replicator's consume path is serial, so your timeout is a lien on every *other* issuer's commands too. Ask an operator if 120 s is genuinely too short for a target | - -**Neither field touches identity.** They ride inside `payload`, not the envelope: `command_id` -remains the sole dedupe key and the sole correlator. Two commands differing only in options are two -fetch occasions (MUST-1 unchanged); a *redelivery* carrying different options is still the same -command and is still deduped. +`headers` and `timeout_seconds` shape the individual fetch. Both are optional; `None` on both means +**exactly** the pre-#11 behaviour — the fetcher's own `user-agent` and its 30 s timeout, byte for +byte. An issuer that sends neither is unaffected by any of this. + +Two rules bind an issuer that does send them: + +- **Everything on the refusal list is refused, not adjusted.** A refusal is a terminal + `fetch_failed` · `invalid_request_options` plus the DLQ, arriving **before any request goes + out** — a refused command never reaches the origin at all. +- **Neither field touches identity.** `command_id` remains the sole dedupe key and the sole + correlator. Two commands differing only in options are two fetch occasions (MUST-1 unchanged); + a *redelivery* carrying different options is still the same command and is still deduped. + +The refusal list — hop-by-hop and derived headers, the token and byte-range rules on names and +values, the count and size ceilings, the timeout bounds — plus the header-name folding rule and the +reasoning behind each, is in +[the reference](content-fetch-issuer-reference.md#request-options-what-replicator-will-send-and-what-it-refuses-11). ### The success fact @@ -164,54 +145,17 @@ command and is still deduped. The six enriched fields (cannobserv#271, `final_url` sourced by cannobserv#279, produced by #10) carry what Replicator holds at publish time and a broadcast consumer cannot recover once fetching -lives here rather than in Watcher. Three details are the whole value of them: - -- **`None` means nobody said, never "the default".** `final_url` is `None` when the *driver* did - not report a landing URL — **not** "no redirect occurred", and Replicator never substitutes the - requested `url` to fill the gap. `content_type_raw` is `None` when the *origin* sent no - `Content-Type` — deliberately **not** `application/octet-stream`, which is a value some - consumers read as "unknown, guess from the URL" and which `media_type` (normalized, required) - substitutes on its own channel. An issuer that collapses these to a default destroys the - distinction it is being handed. -- **`status_code` is always 2xx here.** Every other status closes the command as a `fetch_failed` - instead, so this field distinguishes 200 from 203 or 206 — it is not a success/failure branch, - and a branch written as `if status_code == 200` will silently drop a 203. -- **"Verbatim" excludes surrounding whitespace, and an absurd value is dropped rather than - truncated.** On the three header passthroughs — `content_type_raw`, `etag`, `last_modified` — - nothing *inside* the value is touched: no case folding, no quote stripping, no date parsing. - What is stripped is the whitespace around it, which RFC 9110 excludes from a field value in the - first place. Three cases report `None`: an absent header, a blank or whitespace-only one - ("present but empty" is not a distinction an issuer can act on), and a value longer than - Replicator's `MAX_HEADER_VALUE_LENGTH` - ([`src/worker/handler.py`](../../src/worker/handler.py); currently 1024 characters, and the - constant is authoritative). The last case is dropped rather - than truncated because these are origin-controlled strings on a broadcast stream nothing trims, - and a *truncated* ETag replayed in an `If-None-Match` is a validator that can never match, - which is worse than none. - -> **These are per-*occasion* values on a fingerprint-keyed fact.** They describe the fetch that -> produced this fact, not the bytes — which is why MUST-5 matters more now than it did. Two -> commands returning identical bytes emit two facts with the same `content_fingerprint` and -> possibly *different* `final_url`, `etag`, `last_modified`, and `fetched_at`. A consumer deduping -> its inbox on the fingerprint — already forbidden — now also pins its stored validators to the -> first emission for those bytes, and will replay a stale `If-None-Match` for as long as that -> content is unchanged. - -> **Do not attempt conditional GET yet — this is now the *only* thing standing in the way.** -> `etag` and `last_modified` are the *read* half of the seam, and since #11 the write half exists: -> Replicator honours the command's `headers`, so an `If-None-Match` you send **will** be sent to -> the origin. What has not landed is the outcome. A validator that *matches* earns a **body-less -> 304**, which Replicator still closes as `fetch_failed` / `http_status` — a terminal failure fact -> for what is in fact the most useful possible answer, and one that would tell you the content is -> gone when it is merely unchanged. -> -> So the trap is live rather than theoretical: nothing stops you replaying a validator today, and -> doing so converts every *successful* no-change check into a closed command. The missing piece is -> tracked as **#17** — a 304 needs an outcome of its own, since there is no blob to announce and -> `fetch_failed` means "no blob will ever arrive". Until #17 lands, keep these two fields in your -> own records and send the request unconditionally. (#17 recommends `reason="not_modified"` on the -> existing fact rather than a new event type, so an issuer branching on `terminal` first is already -> forward-compatible with it.) +lives here rather than in Watcher. Three rules govern reading them, and one warning governs acting +on them — all four in +[`content-fetch-issuer-reference.md`](content-fetch-issuer-reference.md#the-enriched-blob_available-fields): + +- **`None` means nobody said, never "the default"**, on `final_url` and `content_type_raw` alike. +- **`status_code` is always 2xx here** — it distinguishes 200 from 203, it is not a success branch. +- **"Verbatim" excludes surrounding whitespace**, and an over-long value is dropped, not truncated. +- **Do not attempt conditional GET yet.** Replicator honours the `headers` you send, so a validator + *will* reach the origin — but a matching one earns a body-less 304, which Replicator still closes + as a terminal `fetch_failed` · `http_status`. Tracked as **#17**. Until it lands, keep `etag` and + `last_modified` in your own records and send the request unconditionally. ### The failure fact @@ -230,7 +174,8 @@ lives here rather than in Watcher. Three details are the whole value of them: | `attempts` | `int \| None` | Set only where the attempt count is *why* the command closed | | `detail` | `str \| None` | Free text for the journal. **Never branch on it**, and never surface it to an end user — on `handler_error` it is the text of an exception Replicator did not anticipate, so its content is unbounded | -`reason` tokens, one per row of the failure taxonomy below: +`reason` tokens, one per row of the +[failure taxonomy](content-fetch-issuer-reference.md#failure-taxonomy-what-happens-and-what-the-issuer-sees): | Token | Condition | |---|---| @@ -238,25 +183,14 @@ lives here rather than in Watcher. Three details are the whole value of them: | `not_fetchable` | bad scheme / invalid URL | | `too_large` | body over `REPLICATOR_MAX_BLOB_BYTES` | | `unsupported_schema_version` | the command decoded at a `schema_version` Replicator does not support | -| `invalid_request_options` | `headers` or `timeout_seconds` are not sendable — see the refusal table above | +| `invalid_request_options` | `headers` or `timeout_seconds` are not sendable — see the [refusal list](content-fetch-issuer-reference.md#request-options-what-replicator-will-send-and-what-it-refuses-11) | | `handler_error` | unclassified, and it exhausted the delivery ceiling (`attempts` set) | -co-core's own docstring also lists `wrong_payload_type`. **Replicator never emits it**, and an -issuer should not expect it: a frame that decoded to a non-`content.fetch` payload carries, at -most, *somebody else's* `command_id` — `BlobAvailableEvent`'s names a command that **succeeded**, -which is why a blob exists for it. A terminal failure keyed on that id would tell an issuer its -good bytes are never coming. That frame dead-letters and stays silent. - -`reason` is a plain `str` and **not** a `Literal`, deliberately: a producer adding a token must -never crash an older `extra="ignore"` consumer. So the token list is additive, and a consumer -that has branched on `terminal` first is already correct for tokens that do not exist yet. - -**Everything Replicator emits today is `terminal=True`.** Non-terminal facts — a 5xx or a 429 -announcing itself while it is still retrying — are deferred (#9 §3): `content.blobs` is -broadcast and nothing trims it, so a fact per reclaim during an origin outage is unbounded -growth on a stream nobody prunes. The cost is stated plainly in MUST-6: a retrying command is -still invisible for as long as it retries. If an issuer needs the in-flight signal, say so on -its tracker rather than inferring one from silence. +`reason` is a plain `str`, not a `Literal`, so the token list is additive and a consumer that +branches on `terminal` first is already correct for tokens that do not exist yet. Replicator never +emits co-core's `wrong_payload_type`, and everything it emits today is `terminal=True` — both +deliberate, both explained in +[the reference](content-fetch-issuer-reference.md#reading-the-failure-fact). All three models are `extra="ignore"`: additive producer fields are tolerated. Branch on `schema_version` **before** destructuring, and never use the strict `*Emit` classes on a consume @@ -377,29 +311,15 @@ So an issuer's primary mechanism is now the fact: consume `content.blobs`, branc type, and on a `fetch_failed` with `terminal=True` close the pending entry **with a reason**. Off one consumer group, since both outcomes share the stream. -**Silence has not gone away — it has narrowed.** Three conditions still produce nothing, and one -produces nothing *yet*: - -- **A frame that fails `from_wire` entirely.** It has no payload, therefore no `command_id`, - therefore nothing a fact could correlate on. DLQ-only. The commonest cause is an issuer that - `XADD`ed flat fields instead of publishing through `to_wire` — see **The frame** — and, since - co-core 0.7.2, a naive `occurred_at`. -- **A frame that decodes to a payload that is not a `content.fetch` command.** Not merely - unreportable — *unsafe* to report. Most of the payload union has no `command_id` at all, and the - members that do carry one that belongs to a different command: a misrouted `blob_available` - names a command that **succeeded**. Announcing a terminal failure against it would contradict a - fact the issuer has already applied, and MUST-4 covers duplicates, not contradictions. Silent by - design. -- **A command whose `command_id` is blank.** Refused before the fetch and dead-lettered, with no - fact — there is no correlator to key one on. Silent to the issuer, but *not* silently - processed: an empty id is not a valid `command_id` (MUST-1), and accepting it would take the - dedupe key `replicator:cmd:` under which every later blank-id command becomes a no-op. -- **A command still retrying.** Replicator emits no non-terminal fact today (#9 §3), so a 5xx, - a 429, a network error, or a blob tree over its ceiling is invisible for as long as it retries — - and there is no latency bound on that: transient failures retry indefinitely at the - `REPLICATOR_CLAIM_MIN_IDLE_MS` cadence (default 60 s), and over - `REPLICATOR_BLOB_MAX_TOTAL_BYTES` the command parks in the PEL until a retention sweep frees - space. A command can legitimately be in flight for a long time. +**Silence has not gone away — it has narrowed.** Four conditions still produce nothing: a frame +that fails `from_wire` entirely, a frame that decodes to a payload that is not a `content.fetch` +command, a command whose `command_id` is blank, and a command that is still retrying. The first +three are permanently silent — no payload, or no `command_id` that is *safe* to key a fact on. The +fourth is silent *for now* (#9 §3), and has no latency bound: transient failures retry indefinitely +at the `REPLICATOR_CLAIM_MIN_IDLE_MS` cadence, and a blob tree over +`REPLICATOR_BLOB_MAX_TOTAL_BYTES` parks in the PEL until a sweep frees space. Each condition, and +why reporting the second would be worse than silence, is in +[the reference](content-fetch-issuer-reference.md#the-four-silent-conditions). Which is why the **reaper stays**, demoted from primary mechanism to backstop: @@ -412,22 +332,13 @@ Which is why the **reaper stays**, demoted from primary mechanism to backstop: still happens, and raising there would only burn the delivery ceiling and reach the same DLQ minutes later). Rare, logged, and another reason the reaper is not optional. -See the failure taxonomy below for exactly which outcomes are visible and which are not. - -**The DLQ is still readable, and still worth reading** — for what the fact cannot carry. It is an -ordinary stream on the same broker, and every entry carries the original **command** envelope — so -`key` is the failed `command_id` — plus `dlq_reason` and `dlq_original_id`. Its value is now the -complement of the fact rather than a substitute for it: it is the only place the silent rows -above show up at all, and it preserves the offending frame itself, which no fact does. Read it with -a plain `XREAD` and no consumer group: a group left behind by a non-owner accumulates a PEL nothing -drains. +See the [failure taxonomy](content-fetch-issuer-reference.md#failure-taxonomy-what-happens-and-what-the-issuer-sees) for exactly +which outcomes are visible and which are not. -One caveat that survives unchanged: the anomaly class that dead-letters a *synthesized* record (a -frame that failed to decode at all, whose original entry has since been trimmed) carries no `key` -to match on. - -Inspection commands live in [`docs/COMMANDS.md`](../COMMANDS.md). Monitoring the DLQ on -Replicator's side remains an operator responsibility, not a bus-level one. +**The DLQ is still readable, and still worth reading** — it is the only place the silent rows show +up at all, and it preserves the offending frame, which no fact does. How to read it, and the one +anomaly class that carries no `key` to match on, are in +[the reference](content-fetch-issuer-reference.md#reading-the-dlq). ### 7. Copy the bytes before the blob expires @@ -480,13 +391,9 @@ A consumer on another host cannot open it, and nothing on the wire says so. for one host means at least 100 s of fetching. Size a reaper's timeout (MUST-6) against the depth of your own burst, not against one fetch. Commands for different hosts are unaffected by each other. - **At the shipped defaults every wait is slept through inside the handler**, so the cost is - seconds of added turnaround and nothing else. Only when an operator configures the interval - *above* `REPLICATOR_READ_BLOCK_MS` (5 s) does a paced command instead stay pending for the - next reclaim, which moves the cadence from seconds to a minute. That is a deployment - decision, not a default — but it is the one that changes what a reaper should expect, so it - is stated here rather than left to be discovered. -- **No ordering.** Two commands issued in sequence may produce facts in either order. + Whether a paced command is slept through or parked for the next reclaim depends on the deployed + interval — see [the reference](content-fetch-issuer-reference.md#pacing-at-the-deployed-defaults), + since it changes what a reaper should expect. - **No cross-command dedupe.** Two `command_id`s for one URL are two fetches and two facts, by design — that is what makes MUST-1 work. - **No retention guarantee on `content.blobs`.** Replicator never trims it — `BusPublish` takes no @@ -496,91 +403,14 @@ A consumer on another host cannot open it, and nothing on the wire says so. --- -## Failure taxonomy: what happens, and what the issuer sees - -| Condition | Replicator's action | Issuer-visible | -|---|---|---| -| Duplicate `command_id` inside the dedupe window (default 24 h) | ack, no fetch | **nothing** — silent drop | -| `schema_version` ≠ 1 | fact, then `content.fetch.dlq` | `fetch_failed` · `unsupported_schema_version` | -| Frame decodes to a non-`content_fetch` payload | `content.fetch.dlq` | **nothing** — any `command_id` in it is another command's | -| Command with a blank `command_id` | `content.fetch.dlq`, before the fetch | **nothing** — no correlator to key a fact on | -| Malformed frame (fails `from_wire`; includes a naive `occurred_at`) | `content.fetch.dlq`, synthesized record | **nothing** — no payload at all | -| HTTP 4xx, or a body-less 304 | fact, then `content.fetch.dlq` | `fetch_failed` · `http_status` (+ `status_code`) | -| URL not fetchable (bad scheme / invalid URL) | fact, then `content.fetch.dlq` | `fetch_failed` · `not_fetchable` | -| Body over `REPLICATOR_MAX_BLOB_BYTES` (default 64 MiB) | fact, then `content.fetch.dlq` | `fetch_failed` · `too_large` | -| Unsendable `headers` / `timeout_seconds` | fact, then `content.fetch.dlq`, **before the fetch** | `fetch_failed` · `invalid_request_options` | -| HTTP 5xx / 408 / 429, or a network error | retry indefinitely, default ~60 s cadence | delayed fact, or **nothing while it retries** | -| Blob tree over `REPLICATOR_BLOB_MAX_TOTAL_BYTES` | parked in the PEL until a sweep frees space | delayed fact, or **nothing while it waits** | -| Unclassified handler error | retried to the delivery ceiling (~4 reclaims / ~4 min at default settings), then fact + DLQ | `fetch_failed` · `handler_error` (+ `attempts`) | -| Success | `blob_available` on `content.blobs` | the fact | - -Every `fetch_failed` row carries `terminal=True` — the command is closed and no blob will arrive. - -The "nothing" rows are not one problem, and the reaper is not the answer to all of them: - -- **Three rows have no usable `command_id`** — a non-`content_fetch` payload, a blank - `command_id`, and a frame that failed to decode. Permanently silent, and the reaper is the - only recourse. This is what MUST-6 keeps it for. -- **Two rows are silent only while the command is in flight** — a retrying transient failure and - a blob tree over its ceiling. Silent **for now** (#9 §3); an issuer that would use an - in-flight signal should say so on its tracker rather than lengthening its timeout to - compensate. -- **The duplicate-`command_id` row is neither.** It is silent because the command was *already - handled* — the first delivery ran and published its fact, so the issuer's entry is already - closed. Reaping and re-issuing there sends the origin a second request for work that - succeeded. The fix is MUST-1: mint a fresh id per fetch occasion and the row cannot occur. - ---- - -## Provenance and trust - -`content.fetch` is an **unauthenticated capability**: any writer to the bus can make Replicator -issue an arbitrary outbound HTTP request from the cluster VM and store the response. There is no -signing, no allowlist, and no issuer identity on the frame. +## Where the rest of the contract lives -Integrity rests entirely on **bus access control** — the broker is Archiver-operated and bound to -localhost on a single trusted VM. That is proportionate today. It stops being proportionate the -moment the bus spans hosts or tenants, at which point message signing or a URL allowlist becomes -the conversation. Not before. - -**`headers` widens that capability, and the widening is bounded here rather than by the broker.** -A bus writer can now attach an arbitrary header — an `Authorization` among them — to a host of its -own choosing. The trust model is unchanged (same localhost broker, same single VM), so the guards -in the refusal table above are not a substitute for it; they are the cheap part, taken because it -is cheap. Concretely they stop three things the broker's boundary says nothing about: a `Host` -override that contacts one origin while addressing another, a CRLF in a value that splits the -request into two, and a `timeout_seconds` large enough to park the serial consume path — that last -one a denial of service against every *other* issuer, not against the origin. - -Two properties an issuer can rely on: a refused command is refused **before** any request goes out, -and header **values never reach the journal** — only names are logged, so an `Authorization` an -issuer attaches is not re-exposed one layer down. - -Relatedly: nothing but the seed script writes to `content.fetch` today, and `seed_fetch.py` -requires `--production` for the one target the live worker consumes. A frame on that stream is -fetched for real. - ---- +Split out in #24 so this document stays readable start to finish. Equally normative: -## Settled: the `fetch_failed` fact - -**Resolved.** This document previously carried silence-as-failure as an open question, arguing the -fix needed "a consumer that wants it — Watcher, once Phase 4 has a real pending map to close out." -That consumer arrived ([CannObserv/watcher#241](https://github.com/CannObserv/watcher/issues/241)), -co-core shipped `FetchFailedEvent` in **v0.7.2** -([cannobserv#270](https://github.com/CannObserv/cannobserv/issues/270)), and Replicator publishes -it (#9). MUST-6 above is the current contract; the tables are the current shape. - -Two things the resolution deliberately did **not** do, recorded so they are not re-litigated as -oversights: - -- **Non-terminal facts are deferred** (#9 §3). `content.blobs` is broadcast and nothing trims it, - so emitting a fact per reclaim while an origin is down is unbounded growth on a stream nobody - prunes. The cost is real and named in MUST-6: a 429 backing off is invisible while it retries, - which is one of the four Watcher behaviours cannobserv#270 set out to enable. Reopen it on the - consumer's tracker if the reaper turns out to re-issue under live retries in practice. -- **The rows with no usable `command_id` stay DLQ-only.** Not a scope cut. `FetchFailedEvent`'s - `command_id` is required, so a fact naming no command closes nothing; and for a frame that - decoded to a *foreign* payload, the `command_id` it carries belongs to a different command — - reporting it would be worse than silence, not better. The reaper is the mechanism there, - permanently. +- [`content-fetch-issuer-reference.md`](content-fetch-issuer-reference.md) — the refusal list, the + enriched `blob_available` fields, the **failure taxonomy**, the four silent conditions, reading + the DLQ, and the **provenance and trust** posture. +- [`replicator-boundaries.md`](replicator-boundaries.md) — what Replicator may become, and therefore + which payload fields this contract will never grow (#12). +- [`2026-07-31-fetch-failed-fact-settled.md`](../plans/2026-07-31-fetch-failed-fact-settled.md) — + historical: how the silence-as-failure question was closed. MUST-6 above is the current contract. diff --git a/docs/contracts/content-fetch-issuer-reference.md b/docs/contracts/content-fetch-issuer-reference.md new file mode 100644 index 0000000..528d995 --- /dev/null +++ b/docs/contracts/content-fetch-issuer-reference.md @@ -0,0 +1,300 @@ +# The `content.fetch` issuer reference + +**Status:** normative, and a companion to +[`content-fetch-issuer-contract.md`](content-fetch-issuer-contract.md). Everything here binds +Replicator's behaviour exactly as the contract does — this file is not commentary. + +**What is here rather than there.** The contract carries what an issuer must *do*: the frame, +the payload shapes, the seven MUSTs, and the guarantee/non-guarantee pair. This file carries +what an issuer *looks up* — the header and timeout rules it consults when a command is refused, +the reasoning behind the enriched `blob_available` fields, the condition-by-condition failure +taxonomy, and the trust posture the whole capability rests on. The split exists so the contract +stays short enough to read start to finish; both files are normative, and a rule does not become +advisory by living here. + +**Changing this document.** The same rule the contract states applies: a change to the failure +taxonomy or to the refusal rules is announced on the open issuer-side trackers — currently +[CannObserv/watcher#241](https://github.com/CannObserv/watcher/issues/241) — in the same change +that edits this file. + +--- + +## Request options: what Replicator will send, and what it refuses (#11) + +Set on the command, in [`ContentFetchCommand`](content-fetch-issuer-contract.md#the-command). + +`headers` and `timeout_seconds` shape the individual fetch. Both are optional and both default to +`None`, which means **exactly** the pre-#11 behaviour: the fetcher's own `user-agent` and its 30 s +timeout, byte for byte. An issuer that sends neither is unaffected by any of the following. + +**Header names are lower-cased before the merge, and the issuer wins.** The fetch driver merges +`{"user-agent": , **your headers}` as a plain, case-*sensitive* dict. Without the fold a +capitalized `User-Agent` leaves both keys in the mapping and httpx sends **two** `User-Agent` field +lines — the default first, yours second — leaving the origin to decide which applies. Folding first +is what makes "issuer wins" a rule rather than a coincidence. Send `User-Agent` or `user-agent`; +either way exactly one line goes on the wire and it carries your value. + +**Surrounding whitespace is dropped from a value** and nothing else is: RFC 9110 excludes OWS from +a field value in the first place, so `" text/html "` is sent as `text/html`. Nothing *inside* a +value is touched. + +**Everything below is refused, not adjusted.** A refusal is a terminal +`fetch_failed` · `invalid_request_options` plus the DLQ, arriving before any request goes out — so +a refused command never reaches the origin at all. The reject-rather-than-fix posture is the same +one the `blob_available` passthroughs take: a header Replicator silently dropped, or a timeout it +silently shortened, is a change to your fetch that you cannot see and cannot account for in your +own fingerprints. + +| Refused | Why | +|---|---| +| `connection`, `keep-alive`, `proxy-connection`, `te`, `trailer`, `transfer-encoding`, `upgrade` | Hop-by-hop (RFC 9110 §7.6.1) — they describe one connection, which httpx and h11 own | +| `host`, `content-length` | Not hop-by-hop: httpx derives both. Overriding `host` addresses one origin while contacting another | +| Any `proxy-*` header | Configures the hop rather than the request | +| A name that is not an RFC 9110 token | `user agent`, `user:agent`, an empty name, anything non-ASCII — and a name with surrounding whitespace, which RFC 9110 forbids before the colon and which is therefore malformed rather than trimmable | +| A value with any byte outside `\x20`–`\x7e` | Printable US-ASCII and SP only. Excludes CR, LF, NUL, every other control character, HTAB, and all of `obs-text` (`\x80`–`\xff`) — narrower than RFC 9110 permits, deliberately. A CRLF here is request splitting | +| Two names differing only in case | Folding them would silently discard one. Refused even when the values agree — the rule is about the shape, not the values | +| More than **32** headers, or more than **8192 bytes** of them | 8 KiB is the common origin-side limit (nginx, Apache), so past it the far end answers an opaque 400. The constants in [`src/worker/handler.py`](../../src/worker/handler.py) are authoritative | +| `timeout_seconds` that is zero, negative, NaN, or infinite | Not a duration | +| `timeout_seconds` over `REPLICATOR_MAX_FETCH_TIMEOUT_SECONDS` (default **120**) | Replicator's consume path is serial, so your timeout is a lien on every *other* issuer's commands too. Ask an operator if 120 s is genuinely too short for a target | + +**Neither field touches identity.** They ride inside `payload`, not the envelope: `command_id` +remains the sole dedupe key and the sole correlator. Two commands differing only in options are two +fetch occasions (MUST-1 unchanged); a *redelivery* carrying different options is still the same +command and is still deduped. + +--- + +## The enriched `blob_available` fields + +Field types and defaults are in +[the success fact](content-fetch-issuer-contract.md#the-success-fact). This is why they are +shaped the way they are, and the one thing they do not yet let an issuer do. + +The six enriched fields (cannobserv#271, `final_url` sourced by cannobserv#279, produced by #10) +carry what Replicator holds at publish time and a broadcast consumer cannot recover once fetching +lives here rather than in Watcher. Three details are the whole value of them: + +- **`None` means nobody said, never "the default".** `final_url` is `None` when the *driver* did + not report a landing URL — **not** "no redirect occurred", and Replicator never substitutes the + requested `url` to fill the gap. `content_type_raw` is `None` when the *origin* sent no + `Content-Type` — deliberately **not** `application/octet-stream`, which is a value some + consumers read as "unknown, guess from the URL" and which `media_type` (normalized, required) + substitutes on its own channel. An issuer that collapses these to a default destroys the + distinction it is being handed. +- **`status_code` is always 2xx here.** Every other status closes the command as a `fetch_failed` + instead, so this field distinguishes 200 from 203 or 206 — it is not a success/failure branch, + and a branch written as `if status_code == 200` will silently drop a 203. +- **"Verbatim" excludes surrounding whitespace, and an absurd value is dropped rather than + truncated.** On the three header passthroughs — `content_type_raw`, `etag`, `last_modified` — + nothing *inside* the value is touched: no case folding, no quote stripping, no date parsing. + What is stripped is the whitespace around it, which RFC 9110 excludes from a field value in the + first place. Three cases report `None`: an absent header, a blank or whitespace-only one + ("present but empty" is not a distinction an issuer can act on), and a value longer than + Replicator's `MAX_HEADER_VALUE_LENGTH` + ([`src/worker/handler.py`](../../src/worker/handler.py); currently 1024 characters, and the + constant is authoritative). The last case is dropped rather + than truncated because these are origin-controlled strings on a broadcast stream nothing trims, + and a *truncated* ETag replayed in an `If-None-Match` is a validator that can never match, + which is worse than none. + +> **These are per-*occasion* values on a fingerprint-keyed fact.** They describe the fetch that +> produced this fact, not the bytes — which is why MUST-5 matters more now than it did. Two +> commands returning identical bytes emit two facts with the same `content_fingerprint` and +> possibly *different* `final_url`, `etag`, `last_modified`, and `fetched_at`. A consumer deduping +> its inbox on the fingerprint — already forbidden — now also pins its stored validators to the +> first emission for those bytes, and will replay a stale `If-None-Match` for as long as that +> content is unchanged. + +> **Do not attempt conditional GET yet — this is now the *only* thing standing in the way.** +> `etag` and `last_modified` are the *read* half of the seam, and since #11 the write half exists: +> Replicator honours the command's `headers`, so an `If-None-Match` you send **will** be sent to +> the origin. What has not landed is the outcome. A validator that *matches* earns a **body-less +> 304**, which Replicator still closes as `fetch_failed` / `http_status` — a terminal failure fact +> for what is in fact the most useful possible answer, and one that would tell you the content is +> gone when it is merely unchanged. +> +> So the trap is live rather than theoretical: nothing stops you replaying a validator today, and +> doing so converts every *successful* no-change check into a closed command. The missing piece is +> tracked as **#17** — a 304 needs an outcome of its own, since there is no blob to announce and +> `fetch_failed` means "no blob will ever arrive". Until #17 lands, keep these two fields in your +> own records and send the request unconditionally. (#17 recommends `reason="not_modified"` on the +> existing fact rather than a new event type, so an issuer branching on `terminal` first is already +> forward-compatible with it.) + +--- + +## Failure taxonomy: what happens, and what the issuer sees + +| Condition | Replicator's action | Issuer-visible | +|---|---|---| +| Duplicate `command_id` inside the dedupe window (default 24 h) | ack, no fetch | **nothing** — silent drop | +| `schema_version` ≠ 1 | fact, then `content.fetch.dlq` | `fetch_failed` · `unsupported_schema_version` | +| Frame decodes to a non-`content_fetch` payload | `content.fetch.dlq` | **nothing** — any `command_id` in it is another command's | +| Command with a blank `command_id` | `content.fetch.dlq`, before the fetch | **nothing** — no correlator to key a fact on | +| Malformed frame (fails `from_wire`; includes a naive `occurred_at`) | `content.fetch.dlq`, synthesized record | **nothing** — no payload at all | +| HTTP 4xx, or a body-less 304 | fact, then `content.fetch.dlq` | `fetch_failed` · `http_status` (+ `status_code`) | +| URL not fetchable (bad scheme / invalid URL) | fact, then `content.fetch.dlq` | `fetch_failed` · `not_fetchable` | +| Body over `REPLICATOR_MAX_BLOB_BYTES` (default 64 MiB) | fact, then `content.fetch.dlq` | `fetch_failed` · `too_large` | +| Unsendable `headers` / `timeout_seconds` | fact, then `content.fetch.dlq`, **before the fetch** | `fetch_failed` · `invalid_request_options` | +| HTTP 5xx / 408 / 429, or a network error | retry indefinitely, default ~60 s cadence | delayed fact, or **nothing while it retries** | +| Blob tree over `REPLICATOR_BLOB_MAX_TOTAL_BYTES` | parked in the PEL until a sweep frees space | delayed fact, or **nothing while it waits** | +| Unclassified handler error | retried to the delivery ceiling (~4 reclaims / ~4 min at default settings), then fact + DLQ | `fetch_failed` · `handler_error` (+ `attempts`) | +| Success | `blob_available` on `content.blobs` | the fact | + +Every `fetch_failed` row carries `terminal=True` — the command is closed and no blob will arrive. + +The "nothing" rows are not one problem, and the reaper is not the answer to all of them: + +- **Three rows have no usable `command_id`** — a non-`content_fetch` payload, a blank + `command_id`, and a frame that failed to decode. Permanently silent, and the reaper is the + only recourse. This is what MUST-6 keeps it for. +- **Two rows are silent only while the command is in flight** — a retrying transient failure and + a blob tree over its ceiling. Silent **for now** (#9 §3); an issuer that would use an + in-flight signal should say so on its tracker rather than lengthening its timeout to + compensate. +- **The duplicate-`command_id` row is neither.** It is silent because the command was *already + handled* — the first delivery ran and published its fact, so the issuer's entry is already + closed. Reaping and re-issuing there sends the origin a second request for work that + succeeded. The fix is MUST-1: mint a fresh id per fetch occasion and the row cannot occur. + +--- + +## Provenance and trust + +`content.fetch` is an **unauthenticated capability**: any writer to the bus can make Replicator +issue an arbitrary outbound HTTP request from the cluster VM and store the response. There is no +signing, no allowlist, and no issuer identity on the frame. + +Integrity rests entirely on **bus access control** — the broker is Archiver-operated and bound to +localhost on a single trusted VM. That is proportionate today. It stops being proportionate the +moment the bus spans hosts or tenants, at which point message signing or a URL allowlist becomes +the conversation. Not before. + +**`headers` widens that capability, and the widening is bounded here rather than by the broker.** +A bus writer can now attach an arbitrary header — an `Authorization` among them — to a host of its +own choosing. The trust model is unchanged (same localhost broker, same single VM), so the guards +in the refusal table above are not a substitute for it; they are the cheap part, taken because it +is cheap. Concretely they stop three things the broker's boundary says nothing about: a `Host` +override that contacts one origin while addressing another, a CRLF in a value that splits the +request into two, and a `timeout_seconds` large enough to park the serial consume path — that last +one a denial of service against every *other* issuer, not against the origin. + +Two properties an issuer can rely on: a refused command is refused **before** any request goes out, +and header **values never reach the journal** — only names are logged, so an `Authorization` an +issuer attaches is not re-exposed one layer down. + +Relatedly: nothing but the seed script writes to `content.fetch` today, and `seed_fetch.py` +requires `--production` for the one target the live worker consumes. A frame on that stream is +fetched for real. + +--- + +## Reading the failure fact + +Field types are in +[the failure fact](content-fetch-issuer-contract.md#the-failure-fact). Two tokens' worth of +behaviour is not visible from the table. + +co-core's own docstring also lists `wrong_payload_type`. **Replicator never emits it**, and an +issuer should not expect it: a frame that decoded to a non-`content.fetch` payload carries, at +most, *somebody else's* `command_id` — `BlobAvailableEvent`'s names a command that **succeeded**, +which is why a blob exists for it. A terminal failure keyed on that id would tell an issuer its +good bytes are never coming. That frame dead-letters and stays silent. + +`reason` is a plain `str` and **not** a `Literal`, deliberately: a producer adding a token must +never crash an older `extra="ignore"` consumer. So the token list is additive, and a consumer +that has branched on `terminal` first is already correct for tokens that do not exist yet. + +**Everything Replicator emits today is `terminal=True`.** Non-terminal facts — a 5xx or a 429 +announcing itself while it is still retrying — are deferred (#9 §3): `content.blobs` is +broadcast and nothing trims it, so a fact per reclaim during an origin outage is unbounded +growth on a stream nobody prunes. The cost is stated plainly in MUST-6: a retrying command is +still invisible for as long as it retries. If an issuer needs the in-flight signal, say so on +its tracker rather than inferring one from silence. + +--- + +## The four silent conditions + +Named in [MUST-6](content-fetch-issuer-contract.md#6-handle-fetch_failed-and-keep-a-reaper-anyway); +this is each one and why it is silent. + +**Silence has not gone away — it has narrowed.** Three conditions still produce nothing, and one +produces nothing *yet*: + +- **A frame that fails `from_wire` entirely.** It has no payload, therefore no `command_id`, + therefore nothing a fact could correlate on. DLQ-only. The commonest cause is an issuer that + `XADD`ed flat fields instead of publishing through `to_wire` — see **The frame** — and, since + co-core 0.7.2, a naive `occurred_at`. +- **A frame that decodes to a payload that is not a `content.fetch` command.** Not merely + unreportable — *unsafe* to report. Most of the payload union has no `command_id` at all, and the + members that do carry one that belongs to a different command: a misrouted `blob_available` + names a command that **succeeded**. Announcing a terminal failure against it would contradict a + fact the issuer has already applied, and MUST-4 covers duplicates, not contradictions. Silent by + design. +- **A command whose `command_id` is blank.** Refused before the fetch and dead-lettered, with no + fact — there is no correlator to key one on. Silent to the issuer, but *not* silently + processed: an empty id is not a valid `command_id` (MUST-1), and accepting it would take the + dedupe key `replicator:cmd:` under which every later blank-id command becomes a no-op. +- **A command still retrying.** Replicator emits no non-terminal fact today (#9 §3), so a 5xx, + a 429, a network error, or a blob tree over its ceiling is invisible for as long as it retries — + and there is no latency bound on that: transient failures retry indefinitely at the + `REPLICATOR_CLAIM_MIN_IDLE_MS` cadence (default 60 s), and over + `REPLICATOR_BLOB_MAX_TOTAL_BYTES` the command parks in the PEL until a retention sweep frees + space. A command can legitimately be in flight for a long time. + +--- + +## Reading the DLQ + +The operator's surface, and the complement of the fact rather than a substitute for it. + +**The DLQ is still readable, and still worth reading** — for what the fact cannot carry. It is an +ordinary stream on the same broker, and every entry carries the original **command** envelope — so +`key` is the failed `command_id` — plus `dlq_reason` and `dlq_original_id`. Its value is now the +complement of the fact rather than a substitute for it: it is the only place the silent rows +above show up at all, and it preserves the offending frame itself, which no fact does. Read it with +a plain `XREAD` and no consumer group: a group left behind by a non-owner accumulates a PEL nothing +drains. + +One caveat that survives unchanged: the anomaly class that dead-letters a *synthesized* record (a +frame that failed to decode at all, whose original entry has since been trimmed) carries no `key` +to match on. + +Inspection commands live in [`docs/COMMANDS.md`](../COMMANDS.md). Monitoring the DLQ on +Replicator's side remains an operator responsibility, not a bus-level one. + +--- + +## Version history + +Contracts settled in cannobserv#266 (co-core v0.7.0); the failure fact added in cannobserv#270 and +the tz-aware `occurred_at` in cannobserv#273, both shipped in **co-core v0.7.2**; the enriched +`blob_available` metadata in cannobserv#271/#279 and the command's request options in +cannobserv#272, shipped in **v0.7.3** and **v0.7.5**. Replicator requires **co-core ≥ 0.7.5**. +Founding rationale: +[`docs/plans/2026-06-25-replicator-mvp-design.md`](../plans/2026-06-25-replicator-mvp-design.md). + +--- + +## What the envelope key is for + +`key` is **not** load-bearing on the consume path: Replicator decodes `payload` and dedupes on +`payload.command_id`, never on the envelope key. Its value is operational — it is what makes a DLQ +entry correlatable without parsing JSON (see MUST-6), and it is what a future partitioned consumer +would shard on. + +--- + +## Pacing at the deployed defaults + +Qualifies the pacing entry under +[what Replicator does not guarantee](content-fetch-issuer-contract.md#what-replicator-does-not-guarantee). + + **At the shipped defaults every wait is slept through inside the handler**, so the cost is + seconds of added turnaround and nothing else. Only when an operator configures the interval + *above* `REPLICATOR_READ_BLOCK_MS` (5 s) does a paced command instead stay pending for the + next reclaim, which moves the cadence from seconds to a minute. That is a deployment + decision, not a default — but it is the one that changes what a reaper should expect, so it + is stated here rather than left to be discovered. +- **No ordering.** Two commands issued in sequence may produce facts in either order. diff --git a/docs/contracts/replicator-boundaries.md b/docs/contracts/replicator-boundaries.md index dab1d99..4d1343f 100644 --- a/docs/contracts/replicator-boundaries.md +++ b/docs/contracts/replicator-boundaries.md @@ -366,6 +366,7 @@ than no test, because this document then cites it. ## Refs - [`content-fetch-issuer-contract.md`](content-fetch-issuer-contract.md) — the wire contract this sits beside +- [`content-fetch-issuer-reference.md`](content-fetch-issuer-reference.md) — its lookup half - #7 — object-store blob backend (the tracked violation) - #9, #10, #11 — the Phase 4 contract additions - #12 — this charter diff --git a/docs/plans/2026-07-31-fetch-failed-fact-settled.md b/docs/plans/2026-07-31-fetch-failed-fact-settled.md new file mode 100644 index 0000000..18ba380 --- /dev/null +++ b/docs/plans/2026-07-31-fetch-failed-fact-settled.md @@ -0,0 +1,33 @@ +# The `fetch_failed` fact: how the open question was closed + +**Status:** resolved 2026-07-31, recorded so the resolution is not re-litigated as an oversight. +Relocated out of [`docs/contracts/content-fetch-issuer-contract.md`](../contracts/content-fetch-issuer-contract.md) +in #24 — the contract states the shipped rules; this states what the question was and what the +answer deliberately did not cover. + +The current contract is MUST-6 and the guarantee/non-guarantee pair in that document. + +--- + +## Settled: the `fetch_failed` fact + +**Resolved.** This document previously carried silence-as-failure as an open question, arguing the +fix needed "a consumer that wants it — Watcher, once Phase 4 has a real pending map to close out." +That consumer arrived ([CannObserv/watcher#241](https://github.com/CannObserv/watcher/issues/241)), +co-core shipped `FetchFailedEvent` in **v0.7.2** +([cannobserv#270](https://github.com/CannObserv/cannobserv/issues/270)), and Replicator publishes +it (#9). MUST-6 above is the current contract; the tables are the current shape. + +Two things the resolution deliberately did **not** do, recorded so they are not re-litigated as +oversights: + +- **Non-terminal facts are deferred** (#9 §3). `content.blobs` is broadcast and nothing trims it, + so emitting a fact per reclaim while an origin is down is unbounded growth on a stream nobody + prunes. The cost is real and named in MUST-6: a 429 backing off is invisible while it retries, + which is one of the four Watcher behaviours cannobserv#270 set out to enable. Reopen it on the + consumer's tracker if the reaper turns out to re-issue under live retries in practice. +- **The rows with no usable `command_id` stay DLQ-only.** Not a scope cut. `FetchFailedEvent`'s + `command_id` is required, so a fact naming no command closes nothing; and for a frame that + decoded to a *foreign* payload, the `command_id` it carries belongs to a different command — + reporting it would be worse than silence, not better. The reaper is the mechanism there, + permanently. diff --git a/src/core/errors.py b/src/core/errors.py index 5c2d01f..999f49c 100644 --- a/src/core/errors.py +++ b/src/core/errors.py @@ -30,7 +30,7 @@ class FailureReason(StrEnum): changes what Watcher branches on. ``StrEnum`` so ``model_dump_json`` writes the token itself. Mirrors the - taxonomy in ``docs/contracts/content-fetch-issuer-contract.md``; the two are + taxonomy in ``docs/contracts/content-fetch-issuer-reference.md``; the two are edited together. """ From 301e67d41c0ab11b18efe8578f0d22b9ec01ba77 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Thu, 6 Aug 2026 15:17:50 +0000 Subject: [PATCH 3/5] #24 chore: record the curation telemetry row Co-Authored-By: Claude Opus 5 (1M context) --- .skills/context-metrics.jsonl | 1 + 1 file changed, 1 insertion(+) diff --git a/.skills/context-metrics.jsonl b/.skills/context-metrics.jsonl index 23c7af8..0f4cad5 100644 --- a/.skills/context-metrics.jsonl +++ b/.skills/context-metrics.jsonl @@ -1 +1,2 @@ {"actions": ["demote:Bus Conventions", "demote:Environment Variables", "demote:Project Layout", "demote:Server Lifecycle", "demote:Infrastructure", "demote:Retention", "demote:Conventions/logging", "index:Detail Docs", "relink:orphans"], "budget": 6000, "bytes": 14254, "delta_days": null, "delta_tokens": null, "docs_orphaned": 0, "docs_total": 9, "file": "AGENTS.md", "lines": 232, "links_dead": 0, "no_loss": "ok", "note": null, "over_budget": false, "repo": "replicator", "skill_commit": "3fc7b71", "skill_version": "1.2", "tokens": 5773, "tokens_exact": true, "tokens_live": 50759, "top_section": "Code Exploration Policy", "top_section_share": 18, "ts": "2026-08-06"} +{"actions": ["split:content-fetch-issuer-contract", "relocate:Settled->docs/plans", "restructure:payload headings", "index:Where the rest lives", "reindex:context-artifacts"], "budget": 6000, "bytes": 14582, "delta_days": 0, "delta_tokens": 123, "docs_orphaned": 0, "docs_total": 10, "file": "AGENTS.md", "lines": 234, "links_dead": 0, "no_loss": "ok", "note": "#24 doc split: issuer contract 14772->9966, new reference 7349; every live doc under the 10k per-doc budget", "over_budget": false, "repo": "replicator", "skill_commit": "3fc7b71", "skill_version": "1.2", "tokens": 5896, "tokens_exact": true, "tokens_live": 54697, "top_section": "Code Exploration Policy", "top_section_share": 18, "ts": "2026-08-06"} From 34c1ed71bea29167e61a3f61883ee3ff9ee8a495 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Thu, 6 Aug 2026 15:37:29 +0000 Subject: [PATCH 4/5] #24 fix: apply first-round code review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 1 fix Restored the "No ordering" non-guarantee to the contract. A slice-based cut took one line too many and swept it into the reference's pacing section, where it dangled under an unrelated heading and read as a claim about pacing. prove-no-loss passed because it verifies a line exists somewhere, not that it landed in the right place — the one failure mode the check is structurally blind to. 2 update MUST-6's "FetchFailedEvent note above" now links the reference section the note moved to. 3 update The moved silent-conditions bullet's "see **The frame**" now links across to the contract instead of pointing nowhere. 4 update De-indented the moved pacing paragraph, which kept its list-continuation indent under a heading with no list. 5 update Reordered the reference into reading order: request options, enriched fields, taxonomy, reading the failure fact, the four silent conditions, reading the DLQ, provenance and trust, then the three short appendices. Accretion order had provenance splitting the taxonomy from its own expansions. 6 update Contract 10,017 -> 9,481, margin 519. Restoring finding 1 had pushed it back over. Moved to the reference: MUST-4's why-a-duplicate-failure-fact- differs mechanism, MUST-7's TTL clock mechanism, MUST-6's swallowed- publish bullet, MUST-2's map-loss consequence, MUST-5's per-occasion consequence (now adjacent to the blockquote making the same argument from the other side), and the re-hash note. Each rule stays; each mechanism moves. Tail index compressed. 7 update "Three behaviours", not "two tokens' worth" — the section covers three things and one of them is a token. 8 update The relocated Settled doc's "MUST-6 above" now links to MUST-6. 9 drop Dropped its duplicate H2. 10 rewrap README:33. 11 update Linked the bare MUST-6 reference at what-the-envelope-key-is-for. prove-no-loss --base d4a5e02: 447 of 475 lines verbatim in place, 28 unaccounted — every one a pointer the move itself disproved ("see the taxonomy below", "the refusal table above", "see **The frame**"), now a cross-file link, or the dropped duplicate heading. All under approved directives. Every live doc under budget; AGENTS.md 5,896/6,000. All links and anchors resolve under GitHub's slug rule (the first-round checker collapsed whitespace runs and would have passed two broken anchors). 444 passed, ruff clean, manifest re-embedded. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 5 +- .../content-fetch-issuer-contract.md | 51 ++---- .../content-fetch-issuer-reference.md | 150 ++++++++++++------ .../2026-07-31-fetch-failed-fact-settled.md | 6 +- 4 files changed, 125 insertions(+), 87 deletions(-) diff --git a/README.md b/README.md index 433b958..f96782b 100644 --- a/README.md +++ b/README.md @@ -30,8 +30,9 @@ The founding design lives in first — it is the normative issuer contract and its permanent home, with the refusal list, failure taxonomy and trust posture in its companion [`content-fetch-issuer-reference.md`](docs/contracts/content-fetch-issuer-reference.md). -Publish through co-core's `to_wire`, never hand-rolled fields; and because the wire carries no domain identity, correlation is -entirely the issuer's job. Most ways of getting either wrong fail silently. +Publish through co-core's `to_wire`, never hand-rolled fields; and because the wire carries no +domain identity, correlation is entirely the issuer's job. Most ways of getting either wrong fail +silently. ## Shape diff --git a/docs/contracts/content-fetch-issuer-contract.md b/docs/contracts/content-fetch-issuer-contract.md index 887b451..0fb0988 100644 --- a/docs/contracts/content-fetch-issuer-contract.md +++ b/docs/contracts/content-fetch-issuer-contract.md @@ -238,10 +238,6 @@ Persist first, then publish; outbox-style, symmetric to archiver's producer. (Re no outbox and must not grow one — its durable record of intent is the consumer group's PEL. The outbox belongs to producers with a database, which is the issuer.) -Losing the map is recoverable but not free: the intent can be re-issued under a fresh -`command_id`, at the cost of another origin request. What is *not* recoverable is the in-flight -fact — it will arrive, match nothing, and have to be discarded. - ### 3. Correlate on `command_id` only — `url` is not a key Archiver's model permits **multiple InfoSources per URL** (non-unique `url`, different extraction @@ -263,14 +259,10 @@ cost of that ordering is the opposite duplicate — a crash between a successful Applying a fact must therefore be safe to do twice. Same `command_id`, same `content_fingerprint`, same `blob_uri` — an upsert, not an append. -**This covers `fetch_failed` too, and there the duplicates are not even identical.** The failure -fact is published *before* the dead-letter (`dead_letter` acks inside itself, so a fact published -after it is lost outright on a crash in between). A crash in that window redelivers the command, -re-runs the failure, and publishes a **second `fetch_failed` with the same `command_id` and a -fresh `occurred_at`** — so its envelope key differs and consumer-side dedup-on-key will not -collapse it. That is by design (see **The frame**); it is also not something Replicator can -engineer away, since a deterministic `occurred_at` would need a durable store Replicator does not -have and must not grow. +**This covers `fetch_failed` too, and there the duplicates are not even identical** — a second +failure fact under the same `command_id` with a fresh `occurred_at`, so its envelope key differs +and consumer-side dedup-on-key will not collapse it. Why, and why Replicator cannot engineer it +away, is in [the reference](content-fetch-issuer-reference.md#why-a-duplicate-failure-fact-is-not-identical). Closing a pending entry must therefore be idempotent in both directions: applying the same terminal failure twice, and applying a failure to an entry already closed. @@ -289,12 +281,6 @@ opposite direction. Dedupe on `command_id`. Treat the fingerprint as content identity. -Since the fact carries per-occasion fetch metadata (cannobserv#271), fingerprint-dedupe now also -loses **which fetch** a fact describes. Two facts sharing a fingerprint may carry different -`final_url`, `etag`, `last_modified`, and `fetched_at`; keeping only the first pins an issuer's -stored validators to whichever occasion arrived first and holds them there for as long as the -content is unchanged — precisely the period a conditional GET would have been useful. - And do not dedupe **`fetch_failed`** on `command_id` either — for the opposite reason. More than one failure fact per command is expected (MUST-4), and once non-terminal facts exist a command may legitimately emit several. Use `terminal` to decide whether the entry closes; use @@ -328,9 +314,6 @@ Which is why the **reaper stays**, demoted from primary mechanism to backstop: cadence is an operator setting on Replicator's host, and an issuer that pins 60 s starts re-issuing under a live retry the day it is tuned. Expect duplicate work rather than assuming loss. -- A `fetch_failed` publish that itself fails is swallowed on Replicator's side (the dead-letter - still happens, and raising there would only burn the delivery ceiling and reach the same DLQ - minutes later). Rare, logged, and another reason the reaper is not optional. See the [failure taxonomy](content-fetch-issuer-reference.md#failure-taxonomy-what-happens-and-what-the-issuer-sees) for exactly which outcomes are visible and which are not. @@ -349,10 +332,9 @@ host. Treat it as a floor to ask about, not a constant to schedule against: cons promptly and re-issue if the URI fails to open, rather than building a pipeline whose timing assumes seven days. -The clock runs from **last reference by a fetch**, not last read by a consumer: Replicator `utime`s -the file when a re-fetch short-circuits on existing bytes -([`src/storage/local.py::_touch`](../../src/storage/local.py)), but a consumer opening the path does -not touch mtime. Holding a `blob_uri` for a week without a re-fetch loses the bytes. +The clock runs from **last reference by a fetch**, not last read by a consumer — holding a +`blob_uri` for a week without a re-fetch loses the bytes. The mechanism is in +[the reference](content-fetch-issuer-reference.md#how-the-blob-ttl-clock-runs). Also: `blob_uri` is a **`file://` URI on Replicator's host**. The contract is VM-local today. A consumer on another host cannot open it, and nothing on the wire says so. @@ -370,8 +352,7 @@ A consumer on another host cannot open it, and nothing on the wire says so. - **`command_id` is echoed** on every fact produced from a command, success or failure. - **The fingerprint is definitional.** Replicator is the cluster's sole fetcher and sole fingerprinter, so `content_fingerprint` *is* the content identity — there is nothing to - cross-check it against. A consumer can still re-hash the bytes at `blob_uri` and compare; worth - doing once the store crosses a boundary (object store, another host). + cross-check it against. - **Identical bytes are one blob.** Content-addressed storage, so a re-fetch of unchanged content costs an origin request and nothing else. - **At-least-once delivery**, both directions. @@ -379,7 +360,8 @@ A consumer on another host cannot open it, and nothing on the wire says so. ## What Replicator does **not** guarantee - **No *non-terminal* failure fact** — a command that is retrying announces nothing until it - either succeeds or is closed. See MUST-6 and the `FetchFailedEvent` note above. + either succeeds or is closed. See MUST-6 and + [the failure-fact notes](content-fetch-issuer-reference.md#reading-the-failure-fact). - **No failure fact for a frame that is not a `content.fetch` command** — any `command_id` it carries is another command's, so there is nothing *safe* to correlate one on. MUST-6. - **No failure fact for a command whose `command_id` is blank** — nothing to correlate one on at @@ -394,6 +376,7 @@ A consumer on another host cannot open it, and nothing on the wire says so. Whether a paced command is slept through or parked for the next reclaim depends on the deployed interval — see [the reference](content-fetch-issuer-reference.md#pacing-at-the-deployed-defaults), since it changes what a reaper should expect. +- **No ordering.** Two commands issued in sequence may produce facts in either order. - **No cross-command dedupe.** Two `command_id`s for one URL are two fetches and two facts, by design — that is what makes MUST-1 work. - **No retention guarantee on `content.blobs`.** Replicator never trims it — `BusPublish` takes no @@ -407,10 +390,10 @@ A consumer on another host cannot open it, and nothing on the wire says so. Split out in #24 so this document stays readable start to finish. Equally normative: -- [`content-fetch-issuer-reference.md`](content-fetch-issuer-reference.md) — the refusal list, the - enriched `blob_available` fields, the **failure taxonomy**, the four silent conditions, reading - the DLQ, and the **provenance and trust** posture. -- [`replicator-boundaries.md`](replicator-boundaries.md) — what Replicator may become, and therefore - which payload fields this contract will never grow (#12). +- [`content-fetch-issuer-reference.md`](content-fetch-issuer-reference.md) — everything this file + points at: the refusal list, the enriched fields, the **failure taxonomy**, the silent + conditions, the DLQ, and **provenance and trust**. +- [`replicator-boundaries.md`](replicator-boundaries.md) — which payload fields this contract will + never grow (#12). - [`2026-07-31-fetch-failed-fact-settled.md`](../plans/2026-07-31-fetch-failed-fact-settled.md) — - historical: how the silence-as-failure question was closed. MUST-6 above is the current contract. + historical: how the silence-as-failure question was closed. diff --git a/docs/contracts/content-fetch-issuer-reference.md b/docs/contracts/content-fetch-issuer-reference.md index 528d995..c824bbf 100644 --- a/docs/contracts/content-fetch-issuer-reference.md +++ b/docs/contracts/content-fetch-issuer-reference.md @@ -61,7 +61,6 @@ own fingerprints. remains the sole dedupe key and the sole correlator. Two commands differing only in options are two fetch occasions (MUST-1 unchanged); a *redelivery* carrying different options is still the same command and is still deduped. - --- ## The enriched `blob_available` fields @@ -120,7 +119,6 @@ lives here rather than in Watcher. Three details are the whole value of them: > own records and send the request unconditionally. (#17 recommends `reason="not_modified"` on the > existing fact rather than a new event type, so an issuer branching on `terminal` first is already > forward-compatible with it.) - --- ## Failure taxonomy: what happens, and what the issuer sees @@ -156,44 +154,13 @@ The "nothing" rows are not one problem, and the reaper is not the answer to all handled* — the first delivery ran and published its fact, so the issuer's entry is already closed. Reaping and re-issuing there sends the origin a second request for work that succeeded. The fix is MUST-1: mint a fresh id per fetch occasion and the row cannot occur. - ---- - -## Provenance and trust - -`content.fetch` is an **unauthenticated capability**: any writer to the bus can make Replicator -issue an arbitrary outbound HTTP request from the cluster VM and store the response. There is no -signing, no allowlist, and no issuer identity on the frame. - -Integrity rests entirely on **bus access control** — the broker is Archiver-operated and bound to -localhost on a single trusted VM. That is proportionate today. It stops being proportionate the -moment the bus spans hosts or tenants, at which point message signing or a URL allowlist becomes -the conversation. Not before. - -**`headers` widens that capability, and the widening is bounded here rather than by the broker.** -A bus writer can now attach an arbitrary header — an `Authorization` among them — to a host of its -own choosing. The trust model is unchanged (same localhost broker, same single VM), so the guards -in the refusal table above are not a substitute for it; they are the cheap part, taken because it -is cheap. Concretely they stop three things the broker's boundary says nothing about: a `Host` -override that contacts one origin while addressing another, a CRLF in a value that splits the -request into two, and a `timeout_seconds` large enough to park the serial consume path — that last -one a denial of service against every *other* issuer, not against the origin. - -Two properties an issuer can rely on: a refused command is refused **before** any request goes out, -and header **values never reach the journal** — only names are logged, so an `Authorization` an -issuer attaches is not re-exposed one layer down. - -Relatedly: nothing but the seed script writes to `content.fetch` today, and `seed_fetch.py` -requires `--production` for the one target the live worker consumes. A frame on that stream is -fetched for real. - --- ## Reading the failure fact Field types are in -[the failure fact](content-fetch-issuer-contract.md#the-failure-fact). Two tokens' worth of -behaviour is not visible from the table. +[the failure fact](content-fetch-issuer-contract.md#the-failure-fact). Three behaviours are not +visible from the table. co-core's own docstring also lists `wrong_payload_type`. **Replicator never emits it**, and an issuer should not expect it: a frame that decoded to a non-`content.fetch` payload carries, at @@ -211,7 +178,6 @@ broadcast and nothing trims it, so a fact per reclaim during an origin outage is growth on a stream nobody prunes. The cost is stated plainly in MUST-6: a retrying command is still invisible for as long as it retries. If an issuer needs the in-flight signal, say so on its tracker rather than inferring one from silence. - --- ## The four silent conditions @@ -224,7 +190,8 @@ produces nothing *yet*: - **A frame that fails `from_wire` entirely.** It has no payload, therefore no `command_id`, therefore nothing a fact could correlate on. DLQ-only. The commonest cause is an issuer that - `XADD`ed flat fields instead of publishing through `to_wire` — see **The frame** — and, since + `XADD`ed flat fields instead of publishing through `to_wire` — see + [The frame](content-fetch-issuer-contract.md#the-frame-envelope) — and, since co-core 0.7.2, a naive `occurred_at`. - **A frame that decodes to a payload that is not a `content.fetch` command.** Not merely unreportable — *unsafe* to report. Most of the payload union has no `command_id` at all, and the @@ -242,7 +209,6 @@ produces nothing *yet*: `REPLICATOR_CLAIM_MIN_IDLE_MS` cadence (default 60 s), and over `REPLICATOR_BLOB_MAX_TOTAL_BYTES` the command parks in the PEL until a retention sweep frees space. A command can legitimately be in flight for a long time. - --- ## Reading the DLQ @@ -263,7 +229,35 @@ to match on. Inspection commands live in [`docs/COMMANDS.md`](../COMMANDS.md). Monitoring the DLQ on Replicator's side remains an operator responsibility, not a bus-level one. +--- + +## Provenance and trust + +`content.fetch` is an **unauthenticated capability**: any writer to the bus can make Replicator +issue an arbitrary outbound HTTP request from the cluster VM and store the response. There is no +signing, no allowlist, and no issuer identity on the frame. + +Integrity rests entirely on **bus access control** — the broker is Archiver-operated and bound to +localhost on a single trusted VM. That is proportionate today. It stops being proportionate the +moment the bus spans hosts or tenants, at which point message signing or a URL allowlist becomes +the conversation. Not before. + +**`headers` widens that capability, and the widening is bounded here rather than by the broker.** +A bus writer can now attach an arbitrary header — an `Authorization` among them — to a host of its +own choosing. The trust model is unchanged (same localhost broker, same single VM), so the guards +in the refusal table above are not a substitute for it; they are the cheap part, taken because it +is cheap. Concretely they stop three things the broker's boundary says nothing about: a `Host` +override that contacts one origin while addressing another, a CRLF in a value that splits the +request into two, and a `timeout_seconds` large enough to park the serial consume path — that last +one a denial of service against every *other* issuer, not against the origin. + +Two properties an issuer can rely on: a refused command is refused **before** any request goes out, +and header **values never reach the journal** — only names are logged, so an `Authorization` an +issuer attaches is not re-exposed one layer down. +Relatedly: nothing but the seed script writes to `content.fetch` today, and `seed_fetch.py` +requires `--production` for the one target the live worker consumes. A frame on that stream is +fetched for real. --- ## Version history @@ -274,16 +268,16 @@ the tz-aware `occurred_at` in cannobserv#273, both shipped in **co-core v0.7.2** cannobserv#272, shipped in **v0.7.3** and **v0.7.5**. Replicator requires **co-core ≥ 0.7.5**. Founding rationale: [`docs/plans/2026-06-25-replicator-mvp-design.md`](../plans/2026-06-25-replicator-mvp-design.md). - --- ## What the envelope key is for `key` is **not** load-bearing on the consume path: Replicator decodes `payload` and dedupes on `payload.command_id`, never on the envelope key. Its value is operational — it is what makes a DLQ -entry correlatable without parsing JSON (see MUST-6), and it is what a future partitioned consumer +entry correlatable without parsing JSON (see +[MUST-6](content-fetch-issuer-contract.md#6-handle-fetch_failed-and-keep-a-reaper-anyway)), and it +is what a future partitioned consumer would shard on. - --- ## Pacing at the deployed defaults @@ -291,10 +285,70 @@ would shard on. Qualifies the pacing entry under [what Replicator does not guarantee](content-fetch-issuer-contract.md#what-replicator-does-not-guarantee). - **At the shipped defaults every wait is slept through inside the handler**, so the cost is - seconds of added turnaround and nothing else. Only when an operator configures the interval - *above* `REPLICATOR_READ_BLOCK_MS` (5 s) does a paced command instead stay pending for the - next reclaim, which moves the cadence from seconds to a minute. That is a deployment - decision, not a default — but it is the one that changes what a reaper should expect, so it - is stated here rather than left to be discovered. -- **No ordering.** Two commands issued in sequence may produce facts in either order. +**At the shipped defaults every wait is slept through inside the handler**, so the cost is +seconds of added turnaround and nothing else. Only when an operator configures the interval +*above* `REPLICATOR_READ_BLOCK_MS` (5 s) does a paced command instead stay pending for the +next reclaim, which moves the cadence from seconds to a minute. That is a deployment +decision, not a default — but it is the one that changes what a reaper should expect, so it +is stated here rather than left to be discovered. + +--- + +## Why a duplicate failure fact is not identical + +Expands [MUST-4](content-fetch-issuer-contract.md#4-make-correlation-idempotent--one-command-can-yield-more-than-one-fact). + +**This covers `fetch_failed` too, and there the duplicates are not even identical.** The failure +fact is published *before* the dead-letter (`dead_letter` acks inside itself, so a fact published +after it is lost outright on a crash in between). A crash in that window redelivers the command, +re-runs the failure, and publishes a **second `fetch_failed` with the same `command_id` and a +fresh `occurred_at`** — so its envelope key differs and consumer-side dedup-on-key will not +collapse it. That is by design (see [The frame](content-fetch-issuer-contract.md#the-frame-envelope)); it is also not something Replicator can +engineer away, since a deterministic `occurred_at` would need a durable store Replicator does not +have and must not grow. + +--- + +## How the blob TTL clock runs + +Expands [MUST-7](content-fetch-issuer-contract.md#7-copy-the-bytes-before-the-blob-expires). + +The clock runs from **last reference by a fetch**, not last read by a consumer: Replicator `utime`s +the file when a re-fetch short-circuits on existing bytes +([`src/storage/local.py::_touch`](../../src/storage/local.py)), but a consumer opening the path does +not touch mtime. Holding a `blob_uri` for a week without a re-fetch loses the bytes. + +Also, on the reaper [MUST-6](content-fetch-issuer-contract.md#6-handle-fetch_failed-and-keep-a-reaper-anyway) +keeps: + +- A `fetch_failed` publish that itself fails is swallowed on Replicator's side (the dead-letter + still happens, and raising there would only burn the delivery ceiling and reach the same DLQ + minutes later). Rare, logged, and another reason the reaper is not optional. + +--- + +## Consequences an issuer inherits from the MUSTs + +Three follow-ons, each expanding a rule rather than adding one. + +**Losing the `command_id` -> domain map** +([MUST-2](content-fetch-issuer-contract.md#2-persist-command_id--domain-durably-before-publishing)): + +Losing the map is recoverable but not free: the intent can be re-issued under a fresh +`command_id`, at the cost of another origin request. What is *not* recoverable is the in-flight +fact — it will arrive, match nothing, and have to be discarded. + +**Fingerprint-dedupe now loses more than a correlation** +([MUST-5](content-fetch-issuer-contract.md#5-do-not-dedupe-facts-on-content_fingerprint)) — the +same argument as the per-occasion blockquote above, from the consumer's side: + +Since the fact carries per-occasion fetch metadata (cannobserv#271), fingerprint-dedupe now also +loses **which fetch** a fact describes. Two facts sharing a fingerprint may carry different +`final_url`, `etag`, `last_modified`, and `fetched_at`; keeping only the first pins an issuer's +stored validators to whichever occasion arrived first and holds them there for as long as the +content is unchanged — precisely the period a conditional GET would have been useful. + +**Re-hashing the bytes** (against *the fingerprint is definitional*, under +[what Replicator guarantees](content-fetch-issuer-contract.md#what-replicator-guarantees)): + A consumer can still re-hash the bytes at `blob_uri` and compare; worth + doing once the store crosses a boundary (object store, another host). diff --git a/docs/plans/2026-07-31-fetch-failed-fact-settled.md b/docs/plans/2026-07-31-fetch-failed-fact-settled.md index 18ba380..edaf6db 100644 --- a/docs/plans/2026-07-31-fetch-failed-fact-settled.md +++ b/docs/plans/2026-07-31-fetch-failed-fact-settled.md @@ -9,14 +9,14 @@ The current contract is MUST-6 and the guarantee/non-guarantee pair in that docu --- -## Settled: the `fetch_failed` fact - **Resolved.** This document previously carried silence-as-failure as an open question, arguing the fix needed "a consumer that wants it — Watcher, once Phase 4 has a real pending map to close out." That consumer arrived ([CannObserv/watcher#241](https://github.com/CannObserv/watcher/issues/241)), co-core shipped `FetchFailedEvent` in **v0.7.2** ([cannobserv#270](https://github.com/CannObserv/cannobserv/issues/270)), and Replicator publishes -it (#9). MUST-6 above is the current contract; the tables are the current shape. +it (#9). +[MUST-6](../contracts/content-fetch-issuer-contract.md#6-handle-fetch_failed-and-keep-a-reaper-anyway) +is the current contract; the tables are the current shape. Two things the resolution deliberately did **not** do, recorded so they are not re-litigated as oversights: From e65b866269e9357e0bf0288559d0c6e69390ee95 Mon Sep 17 00:00:00 2001 From: gregoryfoster Date: Thu, 6 Aug 2026 15:42:11 +0000 Subject: [PATCH 5/5] #24 docs: widen the reference's self-description to the MUST mechanisms MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second-round review finding 13. The header described four things; after finding 6 moved five mechanism blocks in, the file also carries the mechanisms behind several MUSTs — each rule stated in the contract, each explanation here. Co-Authored-By: Claude Opus 5 (1M context) --- docs/contracts/content-fetch-issuer-reference.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/docs/contracts/content-fetch-issuer-reference.md b/docs/contracts/content-fetch-issuer-reference.md index c824bbf..3404484 100644 --- a/docs/contracts/content-fetch-issuer-reference.md +++ b/docs/contracts/content-fetch-issuer-reference.md @@ -8,9 +8,10 @@ Replicator's behaviour exactly as the contract does — this file is not comment the payload shapes, the seven MUSTs, and the guarantee/non-guarantee pair. This file carries what an issuer *looks up* — the header and timeout rules it consults when a command is refused, the reasoning behind the enriched `blob_available` fields, the condition-by-condition failure -taxonomy, and the trust posture the whole capability rests on. The split exists so the contract -stays short enough to read start to finish; both files are normative, and a rule does not become -advisory by living here. +taxonomy, the trust posture the whole capability rests on, and the mechanisms behind several of +the MUSTs, each of which states its rule there and explains itself here. The split exists so the +contract stays short enough to read start to finish; both files are normative, and a rule does not +become advisory by living here. **Changing this document.** The same rule the contract states applies: a change to the failure taxonomy or to the refusal rules is announced on the open issuer-side trackers — currently