Why
docs/contracts/content-fetch-issuer-contract.md settles the wire: what an issuer must
do so a fact is never lost. It says nothing about the service — what Replicator is
allowed to become. Those are different documents and only one of them exists.
The gap matters now because Phase 4 (CannObserv/watcher#241) is the first time capability
gets reallocated across the boundary, and the reallocation is not one-directional:
politeness enforcement wants to move in, alternate fetch drivers want to move in, and
per-URL validator state must be kept out. Every one of those arrives as a
reasonable-looking field or a small settings table, and the cumulative result — a
Replicator with a database, domain vocabulary, and an admin API — is reached one
defensible step at a time. Nothing in the repo today would stop it or even flag it.
The invariants are all currently true: 8 dependencies, none of them a database; one
route, /health; every wire model imported from co-core. This is drift prevention from a
good position, not remediation.
Sibling to the issuer contract, same posture: normative, lives with the code it describes,
linked rather than copied.
Draft: docs/contracts/replicator-boundaries.md
The rule
Replicator owns the mechanics of acquiring bytes from a network and holding them
briefly. It never owns why, when, or what they mean.
The three tests
Any proposed capability, field, or setting runs these in order:
- Does it need durable per-resource history? → issuer. Replicator's state must be
exactly one of: content-addressed on disk (rebuildable by re-fetch), in-memory derived
(rebuildable by replay), or in the broker (PEL, dedupe keys). State outside those three
is a database, whatever it is called.
- Does it need cross-command coordination over a resource only the fetcher can see?
(a host's tolerance, the disk, a connection pool, a browser pool) → Replicator.
Nobody else can see it, and N issuers being polite independently is a fiction.
- Can it be expressed without domain vocabulary? If it needs the words InfoSource,
InfoItem, WatchedItem, aspect, tenant → issuer, always.
Tests 1 and 2 can both fire. Politeness is the case: cross-command state (→ Replicator)
plus per-host operator config (→ issuer). The resolution generalizes:
Mechanism to Replicator. Policy to the issuer. Config travels over the bus.
Worked example — conditional GET
ETag/Last-Modified look like fetcher state. Test 1 sends them to the issuer: they are
durable per-resource history. So the fact returns them (CannObserv/cannobserv#271) and the
next command replays them as request headers (CannObserv/cannobserv#272). Replicator gains
conditional GET while holding zero bytes of per-URL history. Cite this one when a proposal
argues that some small table would be simpler.
Where things live
Replicator (mechanism): fetch execution, redirect following, retry cadence,
fingerprinting, temp storage and its lifetime, disk ceiling, per-host pacing
enforcement, and — future — alternate fetch drivers (browser, archival capture) and
escalation between them.
Issuer (policy/domain): scheduling and cadence, extraction specs, change semantics,
notification, correlation, pending maps and reapers, per-URL validator state, and the
politeness numbers.
Never Replicator's: domain identity, extraction specs, change decisions, notification,
InfoSource correlation, scheduling. Named explicitly so a proposal has to argue against a
written line rather than into a vacuum.
Config taxonomy
| Channel |
Carries |
Examples |
| env |
facts about this host |
blob TTL, disk ceiling, claim cadence, dedupe TTL |
| command |
this occasion |
headers, timeout_seconds, later a fetch strategy |
| config stream |
cluster policy needing cross-command state |
per-host politeness |
The fourth channel is rejected by name: an inbound admin HTTP API. It is the easy path
for every future config need and it ends the property that makes this service testable and
relocatable. Ingress stays health-only, as an enforced invariant rather than a habit.
The policy stream (agreed, not yet built)
content.fetch.policy — last-write-wins per host key. Replicator replays it from 0-0 at
boot into memory and tails it thereafter. No DB, rebuildable, no inbound calls, and the
data stays owned by its producer.
Two properties that must hold or the design fails quietly:
- the owner republishes the full set periodically and on change, so boot replay never
depends on broker retention
- an unknown host resolves to a conservative default, never to unlimited
Producer: Watcher for Phase 4 (the numbers live in its Domain table today). Because it is
bus-delivered, the producer can later move to Archiver — the natural home if a second
issuer ever exists — with no Replicator change at all. That portability is the reason for
the indirection.
Enforcement mechanism: when a host's bucket is dry, leave the message in the PEL and let
the reclaim bring it back. That is already the idiom for the disk ceiling — a policy check
in the handler, not new machinery.
Reviewing a proposed payload field
One question: does this name a domain concept? politeness_key: str passes — opaque to
Replicator. info_source_id fails. The wire's domain-agnosticism is the property the whole
issuer contract is built on; it erodes one plausible field at a time.
Known violation, tracked
blob_uri is a host-local file:// path and nothing on the wire says so. Any consumer
must live on Replicator's VM — a shared-filesystem data-plane coupling in a service
otherwise reached only through the broker, and a constraint on the issuer's deployment
topology that the issuer never agreed to. Tracked in #7 (object-store blob backend).
Recorded here as a violation rather than omitted: a charter that asserts an isolation the
code does not have teaches its readers that the document is decorative.
Cluster-side corollary
The single-fetcher invariant is only true if issuers hold up their end. Watcher's
create-time probe_url currently fetches origins outside Replicator's politeness envelope;
Phase 4 resolves it by making create asynchronous (item enters a probing state, a normal
content.fetch is issued, final_url on the fact fills in the resolved URL). After that,
no service but Replicator fetches watched content. Worth stating here because the
invariant is the cluster's, not this repo's alone — and worth an issuer-side test, in the
issuer's repo.
Enforcement — tests/test_boundaries.py
Six assertions, ~120 lines, all currently passing. The point is failing a PR, not
documenting an intention.
| Invariant |
Test |
| No database |
no persistence dep resolves in the lock (sqlalchemy, asyncpg, psycopg, alembic); no sqlite3 import |
| No domain vocabulary |
grep/AST over src/: info_source, info_item, watched_item, watch, tenant, aspect never appear |
| Ingress is health-only |
app.routes equals an allowlist |
| No locally-defined wire models |
no Pydantic model in src/ declares an event_type field — every wire payload comes from co-core |
| No issuer SDK |
no dependency on archiver-client, notifier-client, or any sibling repo |
| Config surface |
every Settings field is REPLICATOR_*-prefixed; no file or network reads at import time |
The domain-vocabulary grep is the load-bearing one. It is crude, and it catches the exact
regression no type checker and no reviewer reliably catches, because that regression always
arrives as one reasonable-looking field.
Link the charter from the test module's docstring, so whoever trips an assertion reads the
reasoning before deciding whether to delete it.
Follow-on, not filed yet
The policy stream needs a co-core model (cannobserv), a consumer here, and a producer in
Watcher. Held until CannObserv/cannobserv#270–272 land, to keep one co-core release in
flight at a time.
Refs
Why
docs/contracts/content-fetch-issuer-contract.mdsettles the wire: what an issuer mustdo so a fact is never lost. It says nothing about the service — what Replicator is
allowed to become. Those are different documents and only one of them exists.
The gap matters now because Phase 4 (CannObserv/watcher#241) is the first time capability
gets reallocated across the boundary, and the reallocation is not one-directional:
politeness enforcement wants to move in, alternate fetch drivers want to move in, and
per-URL validator state must be kept out. Every one of those arrives as a
reasonable-looking field or a small settings table, and the cumulative result — a
Replicator with a database, domain vocabulary, and an admin API — is reached one
defensible step at a time. Nothing in the repo today would stop it or even flag it.
The invariants are all currently true: 8 dependencies, none of them a database; one
route,
/health; every wire model imported from co-core. This is drift prevention from agood position, not remediation.
Sibling to the issuer contract, same posture: normative, lives with the code it describes,
linked rather than copied.
Draft:
docs/contracts/replicator-boundaries.mdThe rule
The three tests
Any proposed capability, field, or setting runs these in order:
exactly one of: content-addressed on disk (rebuildable by re-fetch), in-memory derived
(rebuildable by replay), or in the broker (PEL, dedupe keys). State outside those three
is a database, whatever it is called.
(a host's tolerance, the disk, a connection pool, a browser pool) → Replicator.
Nobody else can see it, and N issuers being polite independently is a fiction.
InfoItem, WatchedItem, aspect, tenant → issuer, always.
Tests 1 and 2 can both fire. Politeness is the case: cross-command state (→ Replicator)
plus per-host operator config (→ issuer). The resolution generalizes:
Worked example — conditional GET
ETag/Last-Modified look like fetcher state. Test 1 sends them to the issuer: they are
durable per-resource history. So the fact returns them (CannObserv/cannobserv#271) and the
next command replays them as request headers (CannObserv/cannobserv#272). Replicator gains
conditional GET while holding zero bytes of per-URL history. Cite this one when a proposal
argues that some small table would be simpler.
Where things live
Replicator (mechanism): fetch execution, redirect following, retry cadence,
fingerprinting, temp storage and its lifetime, disk ceiling, per-host pacing
enforcement, and — future — alternate fetch drivers (browser, archival capture) and
escalation between them.
Issuer (policy/domain): scheduling and cadence, extraction specs, change semantics,
notification, correlation, pending maps and reapers, per-URL validator state, and the
politeness numbers.
Never Replicator's: domain identity, extraction specs, change decisions, notification,
InfoSource correlation, scheduling. Named explicitly so a proposal has to argue against a
written line rather than into a vacuum.
Config taxonomy
headers,timeout_seconds, later a fetchstrategyThe fourth channel is rejected by name: an inbound admin HTTP API. It is the easy path
for every future config need and it ends the property that makes this service testable and
relocatable. Ingress stays health-only, as an enforced invariant rather than a habit.
The policy stream (agreed, not yet built)
content.fetch.policy— last-write-wins per host key. Replicator replays it from0-0atboot into memory and tails it thereafter. No DB, rebuildable, no inbound calls, and the
data stays owned by its producer.
Two properties that must hold or the design fails quietly:
depends on broker retention
Producer: Watcher for Phase 4 (the numbers live in its
Domaintable today). Because it isbus-delivered, the producer can later move to Archiver — the natural home if a second
issuer ever exists — with no Replicator change at all. That portability is the reason for
the indirection.
Enforcement mechanism: when a host's bucket is dry, leave the message in the PEL and let
the reclaim bring it back. That is already the idiom for the disk ceiling — a policy check
in the handler, not new machinery.
Reviewing a proposed payload field
One question: does this name a domain concept?
politeness_key: strpasses — opaque toReplicator.
info_source_idfails. The wire's domain-agnosticism is the property the wholeissuer contract is built on; it erodes one plausible field at a time.
Known violation, tracked
blob_uriis a host-localfile://path and nothing on the wire says so. Any consumermust live on Replicator's VM — a shared-filesystem data-plane coupling in a service
otherwise reached only through the broker, and a constraint on the issuer's deployment
topology that the issuer never agreed to. Tracked in #7 (object-store blob backend).
Recorded here as a violation rather than omitted: a charter that asserts an isolation the
code does not have teaches its readers that the document is decorative.
Cluster-side corollary
The single-fetcher invariant is only true if issuers hold up their end. Watcher's
create-time
probe_urlcurrently fetches origins outside Replicator's politeness envelope;Phase 4 resolves it by making create asynchronous (item enters a probing state, a normal
content.fetchis issued,final_urlon the fact fills in the resolved URL). After that,no service but Replicator fetches watched content. Worth stating here because the
invariant is the cluster's, not this repo's alone — and worth an issuer-side test, in the
issuer's repo.
Enforcement —
tests/test_boundaries.pySix assertions, ~120 lines, all currently passing. The point is failing a PR, not
documenting an intention.
sqlite3importsrc/:info_source,info_item,watched_item,watch,tenant,aspectnever appearapp.routesequals an allowlistsrc/declares anevent_typefield — every wire payload comes from co-corearchiver-client,notifier-client, or any sibling repoSettingsfield isREPLICATOR_*-prefixed; no file or network reads at import timeThe domain-vocabulary grep is the load-bearing one. It is crude, and it catches the exact
regression no type checker and no reviewer reliably catches, because that regression always
arrives as one reasonable-looking field.
Link the charter from the test module's docstring, so whoever trips an assertion reads the
reasoning before deciding whether to delete it.
Follow-on, not filed yet
The policy stream needs a co-core model (cannobserv), a consumer here, and a producer in
Watcher. Held until CannObserv/cannobserv#270–272 land, to keep one co-core release in
flight at a time.
Refs
docs/contracts/content-fetch-issuer-contract.md(the wire contract this sits beside)