PROPOSAL-3: storage that streams, measured shape, concurrency, bounded reasoning

[R204570]

Why

go.dev was harvested for sixteen minutes, hit one 1.19 MB page, and stored none of the ~1,200 pages that had already extracted cleanly. This branch implements all four phases of PROPOSAL-3.md to make that impossible, then fixes what building it exposed.

The root cause was not what we assumed

We expected "large pages need splitting." It wasn't that.

page.search is a generated column, so to_tsvector runs during the INSERT. Postgres's 1 MB tsvector ceiling was therefore not an indexing limit but a storage limit - the row could not exist at all. One left(content, 300_000) removes it (_upgrade_v4). The page is stored and served whole; the new section table keeps the tail searchable, so bounding the index does not trade a visible failure for an invisible one.

Phases

Phase	Change
1 - streaming storage	store().writer() in both backends, save() delegates to it. Per page, not the batches of 50 the proposal named - a batch of 50 contradicts Invariant 16 in the same document
2 - shape + bounded index	Probe to classify_shape / magnitude; section table; federation completeness becomes the headline
3 - concurrency	Prefetch window, per-host pacing, HOST_CONCURRENCY. 1.30s to 0.38s at 4 workers, same pages in the same order
4 - bounded reasoning	Budget(calls=12), 4 decision points, cached per template/host, off by default

The wiring debt

PROPOSAL-II shipped features that were built, tested, and never called. All had passing tests - passing tests proved behaviour, not reachability.

• classify_shape - zero callers, so every corpus was a tree
• Corpus.magnitude - always 0, so every escalation read "size unknown"
• Federation.complete - tested, never read; the headline came from the entry corpus
• the entry corpus could be reported "not requested" despite being stored

Each is now wired with a grep-the-source assertion. Two of those assertions caught real defects within minutes of being written.

Found while building

• A corpus was compared against a median it was itself in - so the one giant document could never be six times it, and the page branch would have stayed unreachable in exactly the case it exists for.
• _upgrade_v4's guard looked for left(, but Postgres renders the reserved word as "left"( - it would have rebuilt a table and a GIN index on every process start, silently.
• _federate rendered its own coverage note that had drifted into claiming coverage described "only the corpus that was crawled", long after selection began harvesting the others.
• api is not an intent; unknown intents fall back to permissive, so the first version of a test passed while proving nothing.
• The section table went a whole phase populated with nothing reading it - the exact defect this branch exists to fix. search() now unions both indexes.

Also in this branch

A finished harvest was reported as a failure. A google-adk harvest stored 1,799 pages, complete, and the web UI said "that did not go through". The Claude Code provider raised on any is_error result, and error_max_turns discarded everything already streamed. Running out of turns is not failing - the store is the proof, since a page is durable before the next is stored. Now a notice; every other error still raises. The turn cap went 12 to 24, and the tool allowlist - drifted to 3 of 14, missing harvest_docs itself - is now derived from forge_tools.TOOLS.

Design docs moved to Project Development/, README.md staying at the root. Moved with git mv, so history follows.

The README was corrected against the source. It had drifted into contradicting itself - "Serving a purpose" described federation working while "Known limits" still called single-corpus harvesting the most serious known defect. Also fixed a real footgun: the DOCSFORGE_TEST_DB example pointed at the database holding real harvests, directly under a comment promising tests could never touch them.

Project Development/llmsfinder.md - a new proposal, not implemented here. Measured against adk.dev: the same corpus is available in 1 request (llms-full.txt, 3.45 MB), 229 requests (.md twins, 100% of the index's links), or the 1,799 HTML fetches a real harvest actually made.

Not done

• P1 - the spec wants an interrupted harvest to leave 60% readable. Only "previous version intact" is true. This is a contradiction inside the spec: blue/green invisibility is what Invariant 17 requires. Needs a human decision.
• P2 - job progress does not survive a process kill (harvest_jobs._JOBS is in-memory). The pages survive; the report of them does not.
• R1 is still open - the identity gate confirms the name, not the project. Reasoning can veto a wrong admission but is off by default, so the default path is unchanged. Still the top defect.

The spec also asked that ISSUES.md gain no entries. That is broken deliberately - a criterion met by not recording a known gap is met dishonestly. It closed 12 and opened 2.

Testing

648 passed, 22 skipped with Postgres against DocsForgeTest; 611 passed, 59 skipped without. Phase 4's criterion - "with reasoning off, behaves exactly as Phase 3" - holds checkably: all 611 Phase-3 tests pass unchanged, none mentions reasoning.

CI note: DOCSFORGE_TEST_DB must be set or the 37 Postgres tests skip silently and the run looks greener than it is.