#12 docs: boundaries charter + CI-enforced invariants - #18
Conversation
Sibling to the issuer contract: that settles the wire, this settles the service. Mechanism to Replicator, policy to the issuer, config over the bus, and an inbound admin HTTP API rejected by name. tests/test_boundaries.py enforces eight invariants. Four departures from the spec in #12, each because the assertion as drafted would not have held or would not have held for long: - The vocabulary scan is AST over identifiers *and* string literals, not a grep. The grep matched "both tasks watch one stop event" in a docstring; a test whose first tripper is an English sentence gets deleted rather than heeded. Bare "watch" dropped, "watched_item" kept. String literals are in scope because domain leakage arrives as a dict key as often as an attribute. - Ingress is asserted as read-only (allowlisted paths, GET/HEAD only) rather than health-only. FastAPI's three introspection routes are not the admin API the charter rejects, and disabling them behind a dev flag would make the app under test differ from the app that ships. - Added an eighth assertion for the process that is actually deployed: the worker imports no server framework and the unit binds no port. The drafted ingress test covered src/api, which nothing deploys. - BUILD_ID is exempted by name, not silently; a second test pins the exemption list at one entry. Charter changes beyond the draft: politeness promoted to the worked example (it is the case the three tests do not resolve, so it teaches the mechanism/policy split); the policy stream stated as a Phase 4 cutover precondition with the conservative-default interim; MAXLEN trimming and the rejection of a Redis hash written down; and the park-and-reclaim rate floor quantified — 60s granularity against Watcher's 1.0s baseline is a 60x throttle that fails silently in the safe direction. blob_uri's host-local file:// coupling recorded as a tracked violation and pinned by a characterization test, so #7 flips a written line. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 of the sequencing on #12. Enforcement is mechanism and belongs here; the numbers are the issuer's and do not yet travel over the bus, so this carries one conservative default from env until content.fetch.policy exists. REPLICATOR_MIN_HOST_INTERVAL_SECONDS defaults to 1.0 — Watcher's own DEFAULT_MIN_INTERVAL, chosen because it invents nothing. At the Phase 4 cutover Watcher's limiter starts pacing command publication rather than origin requests: it does not fail, it silently becomes decorative (watcher#245). This closes that window. The wait is split by duration, and that split is the whole design: - <= REPLICATOR_READ_BLOCK_MS is slept through in the handler - longer raises TransientFetchError and parks for claim_stale Park-only is the obvious implementation and is wrong by 60x. A parked wait cannot be shorter than REPLICATOR_CLAIM_MIN_IDLE_MS (60s) while the normal interval is 1s, so every host would have been paced at 1/60th of today's rate — silently, and in the safe direction, which is what would have made it easy to ship. Sleep-only holds every other host's commands, and a SIGTERM, behind one origin's politeness. The park bound is derived from the poll window rather than given its own knob: a wait shorter than a poll the loop already performs adds nothing to the shutdown latency TimeoutStopSec is sized for, and test_deploy.py now pins that third term. Transient in both directions, so being polite can never burn the delivery ceiling. The stop event cuts a sleep short and the command parks rather than fetching unpaced on the way out — an interrupted wait is not an elapsed one. Only a request that goes out calls record(): stamping a parked attempt would space the origin from requests it never received. The pacer is built from settings when not injected. The seam fails open, and a byte path that quietly stopped pacing is indistinguishable from one that is working — the BlobUsage argument one seam over. Issuer-visible, so the contract says so: a burst of N commands for one host now takes at least N seconds, and a reaper's timeout must be sized against the depth of the burst rather than against one fetch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 added: per-host pacing (
|
| Wait | What happens |
|---|---|
<= REPLICATOR_READ_BLOCK_MS (5 s) |
slept through in the handler |
| longer | TransientFetchError → parks for claim_stale |
Park-only is the obvious implementation and is wrong by 60×. A parked wait cannot be shorter than REPLICATOR_CLAIM_MIN_IDLE_MS (60 s) while the normal interval is 1 s, so every host would have been paced at 1/60th of today's rate — silently, and in the safe direction, which is exactly what would have made it easy to ship. Sleep-only holds every other host's commands, and a SIGTERM, behind one origin's politeness.
The park bound is derived from the poll window rather than given its own knob: a wait shorter than a poll the loop already performs adds nothing to the shutdown latency TimeoutStopSec is sized for. tests/test_deploy.py now pins that third term (5 + 5 + 120 < 180).
Details that are load-bearing
- Transient in both directions — being polite can never burn the delivery ceiling.
- The stop event cuts a sleep short and the command then parks, rather than fetching unpaced on the way out. An interrupted wait is not an elapsed one.
- Only a request that goes out calls
record(). Stamping a parked attempt would space the origin from requests it never received, and a run of redeliveries would push the next real fetch out indefinitely. - The pacer is built from settings when not injected — the seam fails open, and a byte path that quietly stopped pacing is indistinguishable from one that is working. Same argument as the
BlobUsageidentity pin one seam over, with a test to match. - In-memory host → last-request map, pruned. Derived, bounded, rebuildable by replay — the second of the three state shapes the charter permits. Pruning drops only hosts whose interval has elapsed, so the bound can never be enforced by forgetting a host that is still owed space.
Issuer-visible, so the contract says so
New line under What Replicator does not guarantee: a burst of N commands for one host now takes at least N seconds, and a reaper's timeout (MUST-6) must be sized against the depth of the burst rather than against one fetch. Announcing on CannObserv/watcher#241 per the contract's change rule.
Verification
389 passed, 12 deselected — coverage 99.72% (pacing.py 100%)
ruff check . / ruff format --check . — clean
CI green on dfce7b0
cannobserv#285 carries the model (CONTENT_FETCH_POLICY, FetchPolicyEvent) and the two gaps filing it surfaced: BusPublish has no maxlen, so the MAXLEN trimming this section requires cannot be expressed through co-core at all, and AsyncBusConsumer is group-only, so the replay-then-tail read a last-write-wins stream needs has no driver seam. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 — the ingress route walk failed open twice. A Mount carries its prefix on `.path`, not on an include_context, so a sub-app mounted at /admin reported its children's bare paths: GET /admin/health passed the allowlist as /health. And a WebSocketRoute is not a Route and has no methods, so it fell through to the container branch, found no children, and contributed nothing at all. Mounting a sub-app is the shape an admin API would actually arrive in — the exact thing the charter rejects by name. Both planted as tests. 2, 3 — pacing had no production signal: _pace logs at DEBUG and the root logger is INFO, so a mechanism that caps per-host throughput was absent from the journal entirely. paced_seconds and tracked_hosts now ride the success line that already exists. tracked_hosts's docstring claimed a "sweep-style log line" that did not exist; now it names the real one. 4 — documented that pacing keys on the host asked for, not the host reached: httpx follows redirects inside the driver, so URLs funnelling into one portal hit it at N times the intended rate. final_url is available where the fix would go, but recording the landing host breaks "one request, one record", so it belongs to the policy stream's design. 5 — the issuer contract implied a paced command could stay pending under shipped defaults. It cannot: 1.0s interval against a 5s poll window means every wait is slept through. Now qualified. 6 — the parked-command test compared two live wait_seconds readings, sound only via an implicit argument about elapsed intervals. Hand-advanced clock, exact remaining wait. Clock moved to the package conftest. 7 — a prune that reclaims nothing no longer retries on every record: with more than MAX_TRACKED_HOSTS hosts inside a long interval that was a full dict rebuild per message forever. 8 — min_host_interval_seconds capped at 3600. Past an hour the command parks and re-parks without ever dead-lettering while the issuer's reaper concludes loss; a fat-fingered extra zero should fail at startup. 9 — noted what the pacing-last ordering costs: the tree can cross the storage ceiling during a wait. 11 — the park bound was pinned from above only; a default of 0 would have passed while parking every paced command. Mirror case added. 10 — stet. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
12 — the premise was wrong and the test is kept. Inlining _prune's body into record fails the spy at `assert 0 == 1` rather than passing it, so the second assertion already forecloses the "passes while measuring nothing" failure the finding worried about. Verified by making that edit, not by argument. Recorded in the docstring, and the observable half added alongside: tracked_hosts stays over the bound while nothing is reclaimable, then drops. 13 — tracked_hosts is a gauge and was being repeated on every successful fetch. Moved to _pace's own line, which is promoted from DEBUG (dropped under the root INFO level) to INFO and fires only on the branch that actually waits. The success line keeps paced_seconds, which is per-fetch data correlated with duration_ms rather than a gauge. The sweep's periodic line was the other candidate and is silent on an idle tree. 14 — the cap was unasserted. Three tests: the 1.0 default (Watcher's own DEFAULT_MIN_INTERVAL, which is the whole argument for the number), disable and override, and ValidationError on both out-of-range edges. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…l in AGENTS.md The doc-check flagged AGENTS.md against a config change. Two gaps worth closing for the next reader: pacing keys on the host asked for rather than the host reached (CR #4), and where the mechanism shows up in the journal after the round-2 log move (CR #13). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three robustness gaps in the poison/replay paths, sharing one root: the reader absorbs every failure, so it can fail silently and indefinitely. CR #13: `replay_policies` now takes a required `stop` event. How long it runs is the producer's business — the charter asks it to MAXLEN and Replicator cannot enforce that — and signal handlers are installed before it is called, so a SIGTERM was set and then ignored until replay finished, leaving systemd to SIGKILL at TimeoutStopSec. Required rather than defaulted: "uninterruptible" must not be the outcome of forgetting to wire one. CR #14: both poison paths are bounded by MAX_POISON_SKIPS, mirroring loop.py. Sharper here than there, because CR #9 made an anomaly clear the outage counter — correct in itself, but it removed the only thing accidentally slowing a run of malformed frames down. Replay gives up (the tail resumes from the same cursor); the tail parks, since returning would end the worker. CR #15: `_skip_poison` checks the anomaly's id instead of trusting it, and returns whether it advanced. `BusMessageAnomaly.__init__` defaults message_id to "?" rather than requiring it, and seeking to that wedges the reader permanently — verified against the live broker: every later read raises `ResponseError: Invalid stream ID`, absorbed and backed off on forever while the map silently freezes. Not reachable through today's driver, guarded as a co-core coupling the way loop.py's transient tuple is. CR #16: AGENTS.md and the charter now record that `AsyncBusTailReader.replay()` is unusable — the fact behind CR #1, which lived only in a docstring. It is the obvious API and its name says exactly what a caller wants. CR #17: ran `codebase_context_index`. The round-1 manifest edit changed what the repo says and not what `codebase_context_search` returns; nothing re-embeds it. Verified the search now answers with the current text, and AGENTS.md says the manifest is a source rather than the artifact. CR #18: the integration test reads instead of calling the removed `replay`. `run(_stopped())` now skips the replay, which is correct — the worker is shutting down — so the boot-ordering test ends via its stubbed consume loop instead. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Closes #12 (step 1 of the sequencing agreed in the issue thread).
What ships
docs/contracts/replicator-boundaries.md— the charter, sibling to the issuer contract: that settles the wire, this settles the service.tests/test_boundaries.py— eight invariants, run by the existing CI on every PR tomain. All pass; each corpus scan was verified red against a planted violating module before commit.Departures from the draft in #12
Four, each because the assertion as drafted would not have held or would not have held for long:
watchdropped,watched_itemkeptboth tasks watch one stop eventin a docstring. A test whose first tripper is an English sentence gets deleted rather than heeded. String literals are in scope because leakage arrives as a dict key (detail={"info_source_id": ...}) as often as an attributeGET/HEADonly, recursive route walksrc/api, which nothing deploys. The worker is what binds no port, and an admin listener could have grown insidesrc/worker/with the invariant still greenSettingsfield isREPLICATOR_*BUILD_IDexempted by name, and a second test pinning the exemption list at one entryCharter content beyond the draft
MAXLENtrimming specified, because periodic full republication onto an untrimmed stream is unbounded growth — the same argument that deferred non-terminal facts in Publish fetch_failed facts so permanent failures stop being silent (cannobserv#270) #9 §3 — and here it costs boot time too.schema_version, no co-core model, couples to a key name rather than a payload contract.DEFAULT_MIN_INTERVAL = 1.0s baseline,BACKOFF_MAX_INTERVAL = 60.0s.REPLICATOR_CLAIM_MIN_IDLE_MSis 60 s, so parking matches the backoff case and misses the normal case by 60×. Silent, and in the safe direction, which is what makes it easy to ship by accident. The charter states the constraint and two candidate resolutions without adopting either.Verification
Planted-violation check (one module importing
sqlite3+uvicornand declaringevent_type/info_source_id) fires five of the eight corpus assertions; removed before commit.🤖 Generated with Claude Code