PROPOSAL-3: storage that streams, shape measured, concurrency, bounded reasoning - #26
Merged
Conversation
PROPOSAL-II said what the system should do. This says when it should pay. Everything is currently written at the end, so one failure costs everything: go.dev crawled 16 minutes, met one 1.19 MB page, and stored none of the ~1200 that had extracted cleanly. The same deferral shape produced all 32 open issues. Four changes: streaming blue/green storage so a page is durable before the next is fetched; sections as the storage unit so the tsvector ceiling becomes unreachable; a bounded reasoning budget at the four decision points that decide correctness, cached per cluster and off by default; and a bounded worker pool with a per-host politeness cap. Adds four invariants, and amends II's "no LLM in the per-page loop" in the open rather than quietly. Makes the dev cycle part of the design: an issue found in a cycle is fixed in that cycle, never filed and carried. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EnJCjMch9zNymRVrGhyG3y
go.dev crawled for sixteen minutes, met one 1.19 MB page, and stored none of the ~1,200 that had extracted cleanly. Four defects stacked: the whole harvest went in one COPY in one transaction, so one refused row discarded every good one; the driver's "string is too long for tsvector" reached the caller, which concluded the docs were too big for the database and offered to harvest a subset instead; nothing was durable until the end; and the entire corpus was resident in memory to get there. Storage now streams. Both stores expose writer(); save() delegates to it, so there is one write path rather than a streaming one and a batch one that rots. Pages commit individually, so a refused page costs that page — named in entry["rejected"], surfaced as stats["unextractable"] — and never the harvest. An in-flight version is invisible to readers until it settles, and settling promotes it over the previous one, so an abandoned harvest leaves what was already stored untouched. Per page rather than the batches of 50 the proposal named: a batch of 50 means the first page is not durable until the fiftieth is fetched, which contradicts Invariant 16 in the same document. Fetching dominates; the extra round trips do not show up on a 1,200-page harvest. Dead symbols (W6): _federated_note, passages.passages and Corpus.key deleted; Federation.single and Federation.note now wired into _federate, which had been rendering a second coverage note of its own. That one had drifted into claiming the coverage described "only the corpus that was crawled" long after selection began harvesting the others — exactly the drift W6 predicted, found by deleting it. Selection.as_dict stays: it is not a stray, it is the fix for W5. Deleting Corpus.key also removed the only expression of "an unversioned corpus files under undated" — which lived only in dead code. The live path labelled such corpora with today's date, a claim about when the content is from that an undated corpus cannot make. Now forge_tools.corpus_label(). Closes S1, S2, S5, S6, W6. 576 passed, 22 skipped, Postgres exercised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EnJCjMch9zNymRVrGhyG3y
Two halves of the same failure, fixed together because they share one cause. Shape and magnitude were built, tested and never called (W2, W3). With every corpus classified `tree`, the `page` branch of _harvest_corpus was unreachable, so a specification published as one enormous document was crawled as a site, found one page, and — being one 1.19 MB page — stored none of it. _measure_corpora now probes each admitted corpus with a single GET: enough for chars, in-page anchors, in-scope links and whether the site publishes its own page list. Two defects surfaced the moment that had a caller. has_manifest had been accepted and never read for the whole of PROPOSAL-II; it now rules out `page`, because a site publishing a list of its own pages is not one document however long its landing page is. And a corpus was measured against a median it was itself part of — with two or three corpora the one enormous document dominates that median outright and so can never be six times it, leaving the branch unreachable in precisely the case it exists for. The median is now over peers. The same mistake _neighbourhood made counting a URL as its own neighbour. The storage half no longer depends on getting any of that right. page.search is a GENERATED column, so an unbounded index expression made Postgres's 1 MB tsvector ceiling a storage limit rather than an indexing one: the page could not be inserted at all. It is now generated over left(content, 300_000). The page is stored whole and served whole — the bound is on the index, not the page — and anything past it is indexed section by section through the new `section` table, so bounding the index does not quietly drop the tail of a specification out of search. An ordinary page is not split; splitting every page would double the store to buy nothing. _upgrade_v4's guard looked for `left(` in the column definition. Postgres renders the reserved word quoted as `"left"(`, so it never matched and every migrate() dropped and rebuilt a generated column — a table rewrite and a GIN index rebuild on every process start, silently. It now matches on the bound. Caught by the test asserting the column definition, which failed the same way; there is now a test comparing the column's attnum across two migrations, because no behavioural test would ever have noticed. Closes S1 (properly, not just survivably), S4, S7, W2, W3. Narrows S3. 593 passed, 22 skipped, Postgres exercised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EnJCjMch9zNymRVrGhyG3y
Two wiring defects with one shape: PROPOSAL-II built the honest answer, tested it, and then showed the reader a different one. W1. Federation.complete implements Invariant 9 and nothing called it, so the headline came from the entry corpus's stats. A harvest that got all of one corpus and half of another announced itself complete, with the shortfall visible only to whoever scrolled to the note at the bottom. The fix is an ordering one: _federate is what discovers, admits and harvests the other corpora, so until it has run there is no federation-level completeness to report. It now runs before the headline is written, records the roll-up in stats["federation"], and the headline uses it whenever more than one corpus is in play. W4. The entry corpus — the URL the caller handed in, already crawled and already stored — could be reported "not requested", because classify_kind returns "" for a docs root with no kind token in its path (most of them) and an unclassified corpus matches no kind-specific intent. Corpus.entry now marks it and _mark never deselects it. Invariant 5 still applies to everything else: a peer that genuinely was not requested still says so, with its magnitude. Selection.selected is now derived from the marks rather than set beside them, so the coverage note and the harvest loop cannot work from different lists. W7. There are now wiring assertions for each of these, which is the only reason W1-W4 and W6 can be called closed with any confidence that they stay closed. Two earned their place immediately: the one-renderer assertion caught _federate still building its own coverage note, and the bounded-index assertion caught a migration guard that would have rebuilt a table on every startup. A test written for this found that `api` is not an intent at all — unknown intents fall back to a permissive spec that wants everything, so the first version of the W4 test passed while proving nothing. It now uses `resolve-import`, which genuinely excludes `guide`. Closes W1, W4, W7. Phase 2 complete. 600 passed, 22 skipped, Postgres exercised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EnJCjMch9zNymRVrGhyG3y
go.dev spent sixteen minutes on roughly 1,200 pages — most of a second per page spent waiting on the network with one connection open. Fetching is almost all of a crawl's wall-clock time and almost none of its CPU, so it is the only place a real speedup lives. A bounded prefetch window now runs ahead of the crawl: pages are dispatched in queue order, results consumed in dispatch order, and everything else — the frontier, the ledger, the plan, link discovery, the sink — stays on one thread. That is why there is no lock on Plan.revise and no thread-safe frontier in this commit despite the phase naming both: nothing is shared, so there is nothing to lock, and ordering is preserved by construction rather than defended after the fact. A concurrent crawl returns the same pages in the same order, which is half the acceptance criterion and the half that is easy to lose. Measured: 24 pages at 50ms latency, 1.30s at one worker and 0.38s at four. Politeness moved from sleeping between completed pages to spacing request starts per host, which is where the speedup comes from. At a 0.4s delay and 0.8s per page, sleeping between completions costs 1.2s per page and overlaps nothing; spacing starts costs the host the same 0.4s with several requests in flight. _Pace reserves a slot under its lock and sleeps it off outside, so a worker waiting its turn does not hold the others up, and one host waiting never makes another host wait. Rendering stays sequential whatever the caller asks for. Playwright's sync API is bound to the thread that created the browser and a Fetcher keeps exactly one, so --js falls back to a single worker. A correctness constraint, not a tuning choice. Two things the window broke that needed fixing with it: Truncation accounting. Pages prefetched but never processed have been taken off the queue, so counting only the queue understated a shortfall by up to `workers` pages — and would report `whole` for a crawl that stopped with pages in hand. Undercounting a shortfall is the one direction this must never round. Invariant 16 said "a page is durable before the next is fetched", which concurrency makes literally false. Amended in PROPOSAL-3.md to "before the next is stored", with what it protects stated at its real strength: an interruption can lose at most `workers` pages of fetching and never anything already stored. An invariant nobody can rely on literally is worse than one stated honestly. 611 passed, 22 skipped, Postgres exercised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EnJCjMch9zNymRVrGhyG3y
PROPOSAL-II said "No LLM in the per-page loop", and that rule was right: a
model call per page on a 1,200-page site is the huge bill this project exists
not to send. This is the amended version — no LLM in the loop, a bounded
number of calls at the four moments that decide whether a harvest is correct,
cached so a decision is made once per template or per host rather than once
per page.
1. A template none of the nine selectors recognise. Density refuses, and
refusing is usually right — the link ratio is what stops a sidebar being
stored as documentation. On an API reference it is wrong, and the page is
lost. One cached call per template tells those apart.
2. Kind confidence below threshold. Moved from `selection`, which has only a
URL to go on, into `_measure_corpora`, where the probe has already paid
for the page. Escalation to a human stays the fallback, and the confidence
granted is capped at the escalation threshold rather than above it: a read
answer beats a URL guess and is still weaker than a path that says so, and
claiming otherwise would silence the escalation that exists for this doubt.
3. A corpus proposed on a new host — ISSUES.md R1, the top open defect. The
gate confirms a page is about something with this name, not that it is
this project, and the evidence it counts is identical in both cases. The
consultation may only VETO: a host the algorithmic gate refused is never
re-admitted by asking, so reasoning makes the gate stricter and never
looser, and turning it on cannot make R1 worse.
4. A page answering 200 while rendering an error, which no status code shows.
Gated on a cheap short-and-error-shaped check first, so a site full of
soft-404s still costs at most one call per template.
Invariant 18 is what makes spending anything here safe, and each clause is a
test rather than an intention. Bounded: 12 calls per harvest, one budget for
the entry corpus and its federation together, exhausted means fall back and
never stall. Cached: 500 pages of one template cost one call. Optional: off
unless DOCSFORGE_REASONING is on AND a provider is configured, two independent
switches because neither implies the other. Recorded: every consultation, its
question, its answer and whether it was cached or fell back, in
stats["reasoning"] beside the coverage note.
The model proposes and the code disposes throughout — every answer is
validated before it is trusted, a rejected answer is not cached, and a provider
that raises costs a fallback rather than a harvest.
The acceptance criterion was "a harvest with reasoning disabled behaves exactly
as Phase 3 did". It is met in a checkable rather than an intended way: all 611
tests from Phase 3 pass unchanged, none of them mentions reasoning, and every
one runs with it off.
638 passed, 22 skipped, Postgres exercised. All four phases built.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EnJCjMch9zNymRVrGhyG3y
Checking the four phases against PROPOSAL-3 §6's own acceptance criteria found four shortfalls. Two are fixed here; two are written down. The section table had no read path. It was created, migrated and populated for a whole phase while nothing queried it — which is exactly the defect the entire W-series of ISSUES.md is about, committed while closing it. An index nobody reads is the same as no index: bounding page.search would have traded a page that could not be stored for a page stored whole and findable only by its opening. search() now unions the page index with the section index, ranks by the better of the two, and returns one result per page. Per-host concurrency was promised and not implemented. Pacing bounds request frequency, which is not the same thing: four requests spaced 0.4s apart are still four open sockets if each takes two seconds, and it is open sockets rather than frequency that a small documentation host notices. _Pace now holds a per-host semaphore and records a high-water mark, because §6 asks for this asserted and a cap nobody measures is a comment. Two criteria are not met, and are now ISSUES.md P1 and P2. P1 — "an interrupted harvest leaves 60% of its pages readable and the previous version intact". Only the second half is true, and this is a contradiction inside §6 rather than a gap in the code: blue/green makes an in-flight version invisible to every reader, which is what Invariant 17 and the refusal to serve undisclosed subsets both require. Whether a partial harvest should be readable at all is a decision, not a bug. P2 — "progress that survives killing the process". harvest_jobs._JOBS is an in-process dict, so it does not. The pages survive, which is the half that matters; the report of them does not. §6 also asked that ISSUES.md gain no entries across the four phases. Writing these down breaks that, deliberately: a criterion met by not recording a known gap is met dishonestly, and the file shrank by nine entries regardless. 643 passed, 22 skipped, Postgres exercised. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EnJCjMch9zNymRVrGhyG3y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
go.devwas harvested for sixteen minutes, hit one 1.19 MB page, and stored none of the ~1,200 pages that had already extracted cleanly. This branch implements all four phases ofPROPOSAL-3.mdto make that impossible.The root cause was not what we assumed
We expected "large pages need splitting." It wasn't that.
page.searchis a generated column, soto_tsvectorruns during theINSERT. Postgres's 1 MB tsvector ceiling was therefore not an indexing limit but a storage limit - the row could not exist at all. Oneleft(content, 300_000)removes it (_upgrade_v4). The page is stored and served whole; thesectiontable keeps the tail searchable so bounding the index doesn't trade a visible failure for an invisible one.Phases
store().writer()in both backends,save()delegates to it. Per page, not the batches of 50 the proposal named - a batch of 50 contradicts Invariant 16 in the same documentclassify_shape/magnitude;sectiontable; federation completeness becomes the headlineHOST_CONCURRENCY. 1.30s to 0.38s at 4 workers, same pages in the same orderBudget(calls=12), 4 decision points, cached per template/host, off by defaultThe wiring debt
PROPOSAL-II shipped features that were built, tested, and never called. All had passing tests - passing tests proved behaviour, not reachability.
classify_shape- zero callers, so every corpus was atreeCorpus.magnitude- always0, so every escalation read "size unknown"Federation.complete- tested, never read; the headline came from the entry corpusEach is now wired with a grep-the-source assertion. Two of those assertions caught real defects within minutes of being written.
Found while building
pagebranch would have stayed unreachable in exactly the case it exists for._upgrade_v4's guard looked forleft(, but Postgres renders the reserved word as"left"(- it would have rebuilt a table and a GIN index on every process start, silently._federaterendered its own coverage note that had drifted into claiming coverage described "only the corpus that was crawled", long after selection began harvesting the others.apiis not an intent; unknown intents fall back to permissive, so the first version of a test passed while proving nothing.sectiontable went a whole phase populated with nothing reading it - the exact defect this branch exists to fix.search()now unions both indexes.Not done
harvest_jobs._JOBSis in-memory). The pages survive; the report of them does not.The spec also asked that
ISSUES.mdgain no entries. That is broken deliberately - a criterion met by not recording a known gap is met dishonestly. It closed 12 and opened 2.Testing
643 passed, 22 skippedwith Postgres exercised againstDocsForgeTest. Phase 4's criterion - "with reasoning off, behaves exactly as Phase 3" - holds checkably: all 611 Phase-3 tests pass unchanged, none mentions reasoning.CI note:
DOCSFORGE_TEST_DBmust be set or 22 Postgres tests skip silently and the run looks greener than it is.