Skip to content

#12 docs: boundaries charter + CI-enforced invariants - #18

Merged
gregoryfoster merged 6 commits into
mainfrom
12-boundaries-charter
Aug 5, 2026
Merged

#12 docs: boundaries charter + CI-enforced invariants#18
gregoryfoster merged 6 commits into
mainfrom
12-boundaries-charter

Conversation

@gregoryfoster

Copy link
Copy Markdown
Contributor

Closes #12 (step 1 of the sequencing agreed in the issue thread).

What ships

  • docs/contracts/replicator-boundaries.md — the charter, sibling to the issuer contract: that settles the wire, this settles the service.
  • tests/test_boundaries.pyeight invariants, run by the existing CI on every PR to main. All pass; each corpus scan was verified red against a planted violating module before commit.
  • Cross-links from the issuer contract and AGENTS.md, plus the charter registered as a SocratiCode context artifact.

Departures from the draft in #12

Four, each because the assertion as drafted would not have held or would not have held for long:

Draft Shipped Why
grep for domain vocabulary AST over identifiers and string literals, docstrings and comments skipped; bare watch dropped, watched_item kept the grep matched both tasks watch one stop event in a docstring. A test whose first tripper is an English sentence gets deleted rather than heeded. String literals are in scope because leakage arrives as a dict key (detail={"info_source_id": ...}) as often as an attribute
ingress is health-only ingress is read-only: allowlisted paths, GET/HEAD only, recursive route walk FastAPI's three introspection routes are not the admin API the charter rejects; disabling them behind a dev flag would make the app under test differ from the app that ships, for a surface no deployment serves
six assertions eight — added the deployed process has no ingress, and the tracked violation is pinned the drafted ingress test covered src/api, which nothing deploys. The worker is what binds no port, and an admin listener could have grown inside src/worker/ with the invariant still green
every Settings field is REPLICATOR_* plus BUILD_ID exempted by name, and a second test pinning the exemption list at one entry the repo departed from the prefix rule deliberately; an unnamed exemption reads as the charter being wrong about the code, and an exemption list that grows is the convention ending quietly

Charter content beyond the draft

  • Politeness promoted to the worked example. It is the case the three tests do not resolve to one side, so it is the one that teaches mechanism-vs-policy. Conditional GET kept, shorter.
  • The policy stream is a Phase 4 cutover precondition, not a follow-on, with the conservative-default interim named as the cheap alternative. Cross-ref Phase 4 cutover blocker: per-domain rate limiting stops working the moment Watcher stops fetching watcher#245.
  • MAXLEN trimming specified, because periodic full republication onto an untrimmed stream is unbounded growth — the same argument that deferred non-terminal facts in Publish fetch_failed facts so permanent failures stop being silent (cannobserv#270) #9 §3 — and here it costs boot time too.
  • A Redis hash is rejected in writing, so it does not get re-litigated: no schema_version, no co-core model, couples to a key name rather than a payload contract.
  • The park-and-reclaim rate floor is quantified. Verified against the Watcher checkout: DEFAULT_MIN_INTERVAL = 1.0 s baseline, BACKOFF_MAX_INTERVAL = 60.0 s. REPLICATOR_CLAIM_MIN_IDLE_MS is 60 s, so parking matches the backoff case and misses the normal case by 60×. Silent, and in the safe direction, which is what makes it easy to ship by accident. The charter states the constraint and two candidate resolutions without adopting either.

Verification

369 passed, 12 deselected — coverage 99.70%
ruff check . — All checks passed

Planted-violation check (one module importing sqlite3 + uvicorn and declaring event_type / info_source_id) fires five of the eight corpus assertions; removed before commit.

🤖 Generated with Claude Code

gregoryfoster and others added 2 commits August 5, 2026 20:07
Sibling to the issuer contract: that settles the wire, this settles the
service. Mechanism to Replicator, policy to the issuer, config over the bus,
and an inbound admin HTTP API rejected by name.

tests/test_boundaries.py enforces eight invariants. Four departures from the
spec in #12, each because the assertion as drafted would not have held or
would not have held for long:

- The vocabulary scan is AST over identifiers *and* string literals, not a
  grep. The grep matched "both tasks watch one stop event" in a docstring;
  a test whose first tripper is an English sentence gets deleted rather than
  heeded. Bare "watch" dropped, "watched_item" kept. String literals are in
  scope because domain leakage arrives as a dict key as often as an attribute.
- Ingress is asserted as read-only (allowlisted paths, GET/HEAD only) rather
  than health-only. FastAPI's three introspection routes are not the admin API
  the charter rejects, and disabling them behind a dev flag would make the app
  under test differ from the app that ships.
- Added an eighth assertion for the process that is actually deployed: the
  worker imports no server framework and the unit binds no port. The drafted
  ingress test covered src/api, which nothing deploys.
- BUILD_ID is exempted by name, not silently; a second test pins the exemption
  list at one entry.

Charter changes beyond the draft: politeness promoted to the worked example
(it is the case the three tests do not resolve, so it teaches the
mechanism/policy split); the policy stream stated as a Phase 4 cutover
precondition with the conservative-default interim; MAXLEN trimming and the
rejection of a Redis hash written down; and the park-and-reclaim rate floor
quantified — 60s granularity against Watcher's 1.0s baseline is a 60x
throttle that fails silently in the safe direction.

blob_uri's host-local file:// coupling recorded as a tracked violation and
pinned by a characterization test, so #7 flips a written line.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Step 2 of the sequencing on #12. Enforcement is mechanism and belongs here;
the numbers are the issuer's and do not yet travel over the bus, so this
carries one conservative default from env until content.fetch.policy exists.

REPLICATOR_MIN_HOST_INTERVAL_SECONDS defaults to 1.0 — Watcher's own
DEFAULT_MIN_INTERVAL, chosen because it invents nothing. At the Phase 4
cutover Watcher's limiter starts pacing command publication rather than
origin requests: it does not fail, it silently becomes decorative
(watcher#245). This closes that window.

The wait is split by duration, and that split is the whole design:

- <= REPLICATOR_READ_BLOCK_MS is slept through in the handler
- longer raises TransientFetchError and parks for claim_stale

Park-only is the obvious implementation and is wrong by 60x. A parked wait
cannot be shorter than REPLICATOR_CLAIM_MIN_IDLE_MS (60s) while the normal
interval is 1s, so every host would have been paced at 1/60th of today's
rate — silently, and in the safe direction, which is what would have made it
easy to ship. Sleep-only holds every other host's commands, and a SIGTERM,
behind one origin's politeness. The park bound is derived from the poll
window rather than given its own knob: a wait shorter than a poll the loop
already performs adds nothing to the shutdown latency TimeoutStopSec is
sized for, and test_deploy.py now pins that third term.

Transient in both directions, so being polite can never burn the delivery
ceiling. The stop event cuts a sleep short and the command parks rather than
fetching unpaced on the way out — an interrupted wait is not an elapsed one.
Only a request that goes out calls record(): stamping a parked attempt would
space the origin from requests it never received.

The pacer is built from settings when not injected. The seam fails open, and
a byte path that quietly stopped pacing is indistinguishable from one that is
working — the BlobUsage argument one seam over.

Issuer-visible, so the contract says so: a burst of N commands for one host
now takes at least N seconds, and a reaper's timeout must be sized against
the depth of the burst rather than against one fetch.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gregoryfoster

Copy link
Copy Markdown
Contributor Author

Step 2 added: per-host pacing (dfce7b0)

Scope grew by design — the charter's interim was cheap enough to ship in the same PR, and it is the thing that unblocks the cutover.

REPLICATOR_MIN_HOST_INTERVAL_SECONDS, default 1.0 s = Watcher's own DEFAULT_MIN_INTERVAL. Chosen because it invents nothing: the numbers are the issuer's under the charter, so the least-wrong stand-in is the one the cluster already commits to. The cutover changes who paces, not how much.

The split is the design

Wait What happens
<= REPLICATOR_READ_BLOCK_MS (5 s) slept through in the handler
longer TransientFetchError → parks for claim_stale

Park-only is the obvious implementation and is wrong by 60×. A parked wait cannot be shorter than REPLICATOR_CLAIM_MIN_IDLE_MS (60 s) while the normal interval is 1 s, so every host would have been paced at 1/60th of today's rate — silently, and in the safe direction, which is exactly what would have made it easy to ship. Sleep-only holds every other host's commands, and a SIGTERM, behind one origin's politeness.

The park bound is derived from the poll window rather than given its own knob: a wait shorter than a poll the loop already performs adds nothing to the shutdown latency TimeoutStopSec is sized for. tests/test_deploy.py now pins that third term (5 + 5 + 120 < 180).

Details that are load-bearing

  • Transient in both directions — being polite can never burn the delivery ceiling.
  • The stop event cuts a sleep short and the command then parks, rather than fetching unpaced on the way out. An interrupted wait is not an elapsed one.
  • Only a request that goes out calls record(). Stamping a parked attempt would space the origin from requests it never received, and a run of redeliveries would push the next real fetch out indefinitely.
  • The pacer is built from settings when not injected — the seam fails open, and a byte path that quietly stopped pacing is indistinguishable from one that is working. Same argument as the BlobUsage identity pin one seam over, with a test to match.
  • In-memory host → last-request map, pruned. Derived, bounded, rebuildable by replay — the second of the three state shapes the charter permits. Pruning drops only hosts whose interval has elapsed, so the bound can never be enforced by forgetting a host that is still owed space.

Issuer-visible, so the contract says so

New line under What Replicator does not guarantee: a burst of N commands for one host now takes at least N seconds, and a reaper's timeout (MUST-6) must be sized against the depth of the burst rather than against one fetch. Announcing on CannObserv/watcher#241 per the contract's change rule.

Verification

389 passed, 12 deselected — coverage 99.72% (pacing.py 100%)
ruff check . / ruff format --check . — clean
CI green on dfce7b0

gregoryfoster and others added 4 commits August 5, 2026 20:57
cannobserv#285 carries the model (CONTENT_FETCH_POLICY, FetchPolicyEvent) and
the two gaps filing it surfaced: BusPublish has no maxlen, so the MAXLEN
trimming this section requires cannot be expressed through co-core at all,
and AsyncBusConsumer is group-only, so the replay-then-tail read a
last-write-wins stream needs has no driver seam.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 — the ingress route walk failed open twice. A Mount carries its prefix on
`.path`, not on an include_context, so a sub-app mounted at /admin reported
its children's bare paths: GET /admin/health passed the allowlist as
/health. And a WebSocketRoute is not a Route and has no methods, so it fell
through to the container branch, found no children, and contributed nothing
at all. Mounting a sub-app is the shape an admin API would actually arrive
in — the exact thing the charter rejects by name. Both planted as tests.

2, 3 — pacing had no production signal: _pace logs at DEBUG and the root
logger is INFO, so a mechanism that caps per-host throughput was absent
from the journal entirely. paced_seconds and tracked_hosts now ride the
success line that already exists. tracked_hosts's docstring claimed a
"sweep-style log line" that did not exist; now it names the real one.

4 — documented that pacing keys on the host asked for, not the host
reached: httpx follows redirects inside the driver, so URLs funnelling into
one portal hit it at N times the intended rate. final_url is available
where the fix would go, but recording the landing host breaks "one request,
one record", so it belongs to the policy stream's design.

5 — the issuer contract implied a paced command could stay pending under
shipped defaults. It cannot: 1.0s interval against a 5s poll window means
every wait is slept through. Now qualified.

6 — the parked-command test compared two live wait_seconds readings, sound
only via an implicit argument about elapsed intervals. Hand-advanced clock,
exact remaining wait. Clock moved to the package conftest.

7 — a prune that reclaims nothing no longer retries on every record: with
more than MAX_TRACKED_HOSTS hosts inside a long interval that was a full
dict rebuild per message forever.

8 — min_host_interval_seconds capped at 3600. Past an hour the command
parks and re-parks without ever dead-lettering while the issuer's reaper
concludes loss; a fat-fingered extra zero should fail at startup.

9 — noted what the pacing-last ordering costs: the tree can cross the
storage ceiling during a wait.

11 — the park bound was pinned from above only; a default of 0 would have
passed while parking every paced command. Mirror case added.

10 — stet.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
12 — the premise was wrong and the test is kept. Inlining _prune's body into
record fails the spy at `assert 0 == 1` rather than passing it, so the
second assertion already forecloses the "passes while measuring nothing"
failure the finding worried about. Verified by making that edit, not by
argument. Recorded in the docstring, and the observable half added
alongside: tracked_hosts stays over the bound while nothing is reclaimable,
then drops.

13 — tracked_hosts is a gauge and was being repeated on every successful
fetch. Moved to _pace's own line, which is promoted from DEBUG (dropped
under the root INFO level) to INFO and fires only on the branch that
actually waits. The success line keeps paced_seconds, which is per-fetch
data correlated with duration_ms rather than a gauge. The sweep's periodic
line was the other candidate and is silent on an idle tree.

14 — the cap was unasserted. Three tests: the 1.0 default (Watcher's own
DEFAULT_MIN_INTERVAL, which is the whole argument for the number), disable
and override, and ValidationError on both out-of-range edges.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…l in AGENTS.md

The doc-check flagged AGENTS.md against a config change. Two gaps worth
closing for the next reader: pacing keys on the host asked for rather than
the host reached (CR #4), and where the mechanism shows up in the journal
after the round-2 log move (CR #13).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gregoryfoster
gregoryfoster merged commit 24a9b1b into main Aug 5, 2026
2 checks passed
@gregoryfoster
gregoryfoster deleted the 12-boundaries-charter branch August 5, 2026 22:00
gregoryfoster added a commit that referenced this pull request Aug 6, 2026
Three robustness gaps in the poison/replay paths, sharing one root: the reader
absorbs every failure, so it can fail silently and indefinitely.

CR #13: `replay_policies` now takes a required `stop` event. How long it runs
is the producer's business — the charter asks it to MAXLEN and Replicator
cannot enforce that — and signal handlers are installed before it is called, so
a SIGTERM was set and then ignored until replay finished, leaving systemd to
SIGKILL at TimeoutStopSec. Required rather than defaulted: "uninterruptible"
must not be the outcome of forgetting to wire one.

CR #14: both poison paths are bounded by MAX_POISON_SKIPS, mirroring loop.py.
Sharper here than there, because CR #9 made an anomaly clear the outage counter
— correct in itself, but it removed the only thing accidentally slowing a run
of malformed frames down. Replay gives up (the tail resumes from the same
cursor); the tail parks, since returning would end the worker.

CR #15: `_skip_poison` checks the anomaly's id instead of trusting it, and
returns whether it advanced. `BusMessageAnomaly.__init__` defaults message_id
to "?" rather than requiring it, and seeking to that wedges the reader
permanently — verified against the live broker: every later read raises
`ResponseError: Invalid stream ID`, absorbed and backed off on forever while
the map silently freezes. Not reachable through today's driver, guarded as a
co-core coupling the way loop.py's transient tuple is.

CR #16: AGENTS.md and the charter now record that `AsyncBusTailReader.replay()`
is unusable — the fact behind CR #1, which lived only in a docstring. It is the
obvious API and its name says exactly what a caller wants.

CR #17: ran `codebase_context_index`. The round-1 manifest edit changed what
the repo says and not what `codebase_context_search` returns; nothing
re-embeds it. Verified the search now answers with the current text, and
AGENTS.md says the manifest is a source rather than the artifact.

CR #18: the integration test reads instead of calling the removed `replay`.

`run(_stopped())` now skips the replay, which is correct — the worker is
shutting down — so the boot-ordering test ends via its stubbed consume loop
instead.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Boundaries charter: what Replicator is allowed to become (+ CI-enforced invariants)

1 participant