chain-serve pagination: EncodeBlocksUpTo + design (working-set fix — PE approach-review before the async loop) - #466
chain-serve pagination: EncodeBlocksUpTo + design (working-set fix — PE approach-review before the async loop)#466nerolabs wants to merge 1 commit into
Conversation
…in sync (working-set fix)
The chain-serve OOM driver: a node serving MsgGetChain{Height:0} marshals its WHOLE
bond-reg-laden chain into ONE buffer (chain.EncodeBlocks = 144MB LIVE, heap-profiled).
EncodeBlocksUpTo encodes the longest PREFIX that fits a byte cap (always >=1; sizing
blocks one at a time so it never marshals more than the returned window) — bounding the
serve buffer instead of the whole chain. A syncing peer requests successive windows and
reassembles; Reconcile validates the reassembled linkage (ErrWrongParent), so a windowed
fetch CANNOT corrupt the chain (verified — this is why it's tractable).
Tests: TestEncodeBlocksUpToBoundsTheWindow (bounded prefix, >=1 on oversize, legacy
whole-chain at maxBytes<=0) + TestWindowedReassemblyEqualsFullChain (successive windows
rebuild the exact chain across multiple round-trips — the requester-loop logic pinned
deterministically, without the async wiring).
The DELICATE half — wiring the MsgGetChain handler + the async fetchFull window-loop — is
flagged for PE sign-off on the approach FIRST (sync-correctness path; the atomic
server+requester rollout / mixed-fleet question). Design + safety argument + PE questions:
docs/thinking/2026-08-18-paginate-chain-sync-design.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TqsAc2sNZY7Pg3VgCUKm2P
✅ Deploy Preview for kaleidoscopic-pegasus-e9fec4 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Triage 2026-08-19 (post D-M1-PIVOT): kept open — NOT superseded by #470. Checked against main: the serve handler still encodes the whole requested suffix in one buffer ( Status unchanged: awaiting PE approach-review before the requester loop lands (per the design doc's own guardrail). Roadmap placement: Phase 3-adjacent (cheap heights shrink the same per-height cost from the other side). Will need a mechanical rebase when picked up (main is at dd19203; the PR's chain.go addition is additive, so conflicts should be minimal). |
1 participant
The chain-serve OOM driver (heap-profiled): a node serving
MsgGetChain{Height:0}marshals its whole bond-reg-laden chain into ONE buffer (chain.EncodeBlocks= 144 MB LIVE). This bounds it.What's here (safe, tested, groundwork)
chain.EncodeBlocksUpTo(blocks, maxBytes)— encodes the longest prefix fitting a byte cap (always ≥1; sizes blocks one at a time so it never marshals more than the returned window). Bounds the serve buffer instead of the whole chain.TestEncodeBlocksUpToBoundsTheWindow(bounded prefix / oversize / legacy) +TestWindowedReassemblyEqualsFullChain(successive windows rebuild the exact chain — the requester-loop logic pinned deterministically, without the async wiring).Deliberately NOT here — flagged for PE approach-review first
The async requester loop (
fetchFullwindowed fetch) + the handler wiring is the delicate sync-correctness change. Safety hinge (verified):Reconcilevalidates full block linkage (ErrWrongParent), so a windowed reassembly that splices inconsistent windows fails closed — pagination cannot corrupt the chain. But the atomic server+requester rollout (mixed-fleet) is a design question.3 questions for PE (in the design doc —
docs/thinking/2026-08-18-paginate-chain-sync-design.md):maxChainReplyBytes= 8 MiB — endorse or derive from network-durability?This is the interim memory fix; #299 succinct proofs is the structural win that makes it (and pruning) minor. Scope + safety + failure-modes + test plan in the design doc.
🤖 Generated with Claude Code