feat(chain,node): H2 rolling retention horizon + suffix-sync — the MATURING OOM return-to-2GB (slices 1–5) - #470
Merged
Merged
Conversation
…(H2 slices 1-2, OOM return-to-2GB) The MATURING daemon OOM's residual driver is chain retention of the ~1.5 MB space-time bond proof (BondReg.Answer) growing O(all history) — build-immutable #8 says a validator must fit a 2 GB box. The fix (PE-ruled in-scope, research-certified safetyDepth): bound the RESIDENT heavy payload to a recent finalized window via payload-selective pruning. This lands the two non-consensus substrate slices; the Reconcile Q2 gate + the prune itself follow. Slice 1 — RetentionHorizon() = finalizedHead - 2*BondTTL, floored to an epoch boundary (validator-set snapshot, #357 Cond A); 0 without BFT finality (no anchor to trust below). Pure, read-only, exhaustively unit-tested. Slice 2 — Opt 1 pruned-block representation: Block.Prune() drops the heavy BondReg.Answer and stores the pre-prune hash (Pruned field, cbor 14, additive/omitempty, excluded from the Hash preimage), so a pruned block still hash-links and still carries its consensus signatures (unbounded late-reveal slashing) while shedding the ~1.5 MB/reg. Hash() returns the stored value for a pruned block; a full block hashes exactly as before (no BlockVersion bump). Dormant — nothing prunes or accepts pruned blocks until the Q2-gated slice. Rulings/cert: principle-engineer/{rolling-horizon-oom,pruned-block-representation}-ruling-PE-2026-08-18.md; research-outcome/safetyDepth-retention-horizon-RESEARCH-CERTIFICATION-2026-08-18.md. Design: docs/thinking/2026-08-18-serve-retain-from-checkpoint-oom-fix.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WfnrdXxUnaDRbZusLjGCii
✅ Deploy Preview for kaleidoscopic-pegasus-e9fec4 ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
… below the node's OWN anchor (H2 slice 3, C1 merge gate)
Slice 3 of the rolling-retention-horizon OOM box-fix: make the re-validation paths
TOLERATE a payload-pruned (Answer-less) block without letting a peer forge standing.
Once nodes prune BondReg.Answer below the horizon (slice 4), any replay that accepts a
pruned block SKIPS the ~1.5 MB space-time proof re-verify — so the skip must be gated on
the node's OWN finalized anchor, never a peer's claim, or an attacker strips Answer to
dodge verification and forge consensus standing (a C1/M0 no-discount break). Still
DORMANT: nothing prunes or emits a pruned block yet.
The gate (validateBondRegs, chain.go):
- pruned & Height >= trustFloor() -> ErrPrunedAboveHorizon (the C1 reject)
- pruned & Height < trustFloor() -> skip the space-time re-verify (finalized-trusted);
belt: a pruned block still carrying an Answer -> ErrMalformedPruned
- full (Answer present) -> full verification at any height, unchanged
trustFloor() = max(WSCheckpoint.Height, RetentionHorizon()) (retention.go), with a
trustFloorOverride the Reconcile replay pins to the RECEIVING node's snapshot — so the
throwaway tmp replica uses the receiver's fixed anchor, never a height the attacker's
fork could supply.
ONE gate site, not two (evidence over the design doc): only Reconcile re-verifies bonds
(Reconcile -> tmp.Append -> ValidateCommit -> ValidateProposal -> validateBondRegs ->
verifyBond). Reload (validateStructural) never re-verifies bonds — it checks sigs against
Hash(), which returns the stored pre-prune hash (slice 2) — so a pruned own-disk block
already replays correctly. Proven by TestReloadPrunedBlockRoundTrips; no change there.
Threading is load-bearing for LIVENESS (post-build finding, corrects the plan): the gate
runs DURING replay when tmp's head reaches only the current block, so tmp's incremental
self-floor is always < height and would OVER-reject legitimately-pruned history (break
sync), not over-trust. Pinning tmp to the receiver's FIXED floor accepts a pruned block
genuinely below the receiver's finalized anchor; safety is backstopped by the finality
gate's hash-chaining (ErrPreFinalityReorg forces the fork to contain the receiver's
finalized head, so every below-floor block hash-equals already-verified history). Proven
load-bearing by ABLATION — removing the two threading lines turns the positive-floor
oracle RED ("floor 1" at h3 vs the receiver's fixed 5).
Consensus-invariants (I1-I5, binding PR-body rule): touches I5 — a pruned block is STILL
valid late-reveal equivocation evidence (header + consensus sigs kept; Hash() returns the
stored value), asserted by TestQ2_PrunedBlockStillSlashable. Reads I3/I4's finalized
anchor. Changes NO quorum sizing (I1), NO signing ledger (I2), NO fork-choice function
(I5 determinism). The C1/M0 protection is exactly the Q2 gate: proof-skip only strictly
below the node's own anchor.
Failing-first oracles (q2_gate_test.go, all RED-first then GREEN):
- above/at-floor rejected; below-floor accepted; fresh-node (floor 0) trusts nothing
- malformed pruned+Answer rejected; full block unaffected at any height
- Reconcile gates a pruned block in the replay against the receiver's floor
- positive-receiver-floor (threading; ablation-load-bearing) + at-floor security
- pruned-block-still-slashable (I5); Reload-round-trips (the one-site finding)
Full core/adapters/cmd suites green (incl. core/node model-checks); vet clean.
CHANGELOG + regenerated website/changelog.html; check_links OK.
Plan + post-build finding: docs/thinking/2026-08-18-slice3-q2-gate-plan.md.
PE rulings: principle-engineer/{rolling-horizon-oom,pruned-block-representation}-ruling-PE-2026-08-18.md.
One-line PE ack still wanted on the single-site refinement (Reload unchanged) at PR time.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LDPQNH9BvDewSQ8KLuQWRN
… degenerate-BondTTL guard (H2 slice 4)
Slice 4 of the rolling-retention-horizon OOM box-fix: the actual in-place shed of the
~1.5 MB space-time proof (BondReg.Answer) from finalized blocks strictly below the prune
floor, keeping header + consensus sigs (Block.Prune, slice 2). Because chainstore.Save and
the chainrole serve path both read c.blocks, this one in-place shed bounds resident,
on-disk, AND served heavy payload to a recent finalized window.
DORMANT by design: nothing in production calls pruneBelowHorizon() yet. Grounding the
prune in SyncChain/Reconcile showed enabling it is a SYNC-protocol change, not a memory
tweak — mesh catch-up is a full-chain-from-genesis Reconcile (chainrole.go:1099 asks
{Height:0}; no suffix-diff), which the slice-3 Q2 gate rejects for a behind peer's pruned
gap [Fr, Fs). The PE ruled (slice4-sync-redirect-ruling-PE-2026-08-18):
- REJECT trusting a peer-served finalized head to authenticate a pruned prefix — a C1 /
long-range break (garbage-Answer bonds + forged super-quorum, pruned).
- Safe unblock = suffix-sync from the node's OWN finalized head (behind < safetyDepth) +
an out-of-band WS-checkpoint / archive node (behind > safetyDepth) — the existing weak-
subjectivity window made the sync boundary, no new trust.
- Opt C: bank this dormant, tested shed now; the sync redirect is the enablement gate.
So the shed ships behind that gate; interim OOM air stays e2-medium + GOMEMLIMIT + -inbound-cap.
The prune floor (retention.go, pure pruneFloorAt + live pruneFloor):
- retains max(2*BondTTL, BondRegHeadWindow+margin) below the finalized head, epoch-aligned
- 0 without BFT finality (no finalized anchor); 0 when BondTTL is degenerate (0) — the PE-
acked guard against the horizon collapsing onto the tip and stranding the re-verify window
- always <= RetentionHorizon() <= trustFloor(), so a shed block is always below the Q2
gate's trust floor (we prune more conservatively than we trust — never shed then reject)
Consensus posture: dormant, so no consensus behavior changes (no caller). The shed itself
touches no quorum/signing/fork-choice rule; it drops a hashed payload but Block.Prune keeps
Hash() (stored) and the sigs, so I5 late-reveal slashing survives and Prev-linkage holds —
asserted by the Reload round-trip test. Enablement (the sync redirect) will carry its own
I1/I3/I4 statement + the anchor-safety merge-gate oracle the PE specified.
Failing-first oracles (prune_test.go, RED against a stub then GREEN):
- pruneFloorAt arithmetic (degenerate/guard-governs/horizon-governs/underflow/epoch-floor)
- sheds every heavy block below the floor, retains all at/above
- degenerate BondTTL=0 prunes nothing (the guard)
- pruned own chain preserves Hash/Prev-linkage and Reloads intact
- idempotent (already-pruned + entry-only blocks skipped)
Full core/adapters/cmd suites green; go vet clean (CI is vet+test, no staticcheck).
CHANGELOG updated (slices 1-4) + regenerated website/changelog.html; check_links OK.
Finding + PE ruling: docs/thinking/2026-08-18-slice4-prune-blocked-on-sync-redirect.md;
principle-engineer/slice4-sync-redirect-ruling-PE-2026-08-18.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LDPQNH9BvDewSQ8KLuQWRN
…turn-to-2GB (H2 slice 5)
Slice 5 is the safe sync redirect the PE gated the prune on (slice4/5-sync-redirect
rulings): it turns pruning ON. Mesh catch-up was a full-chain-from-genesis Reconcile
({Height:0}), which the slice-3 Q2 gate rejects for a behind peer's pruned gap. Now a
node suffix-syncs from its OWN finalized head and anchors on its own verified prefix —
never a peer-served head (peer-served-head trust was REJECTED as a C1/long-range Sybil
break: a garbage-Answer bond + forged super-quorum, served pruned, has no purchase when
the node supplies its own prefix and re-verifies the suffix).
Mechanism (M1, PE-acked — reuses the audited Reconcile + finality gate + slice-3 Q2 gate,
NO new consensus trust path):
- chain.FinalizedHeight(): len-1 under BFT finality (committed==final in both regimes),
(0,false) without — sync then falls back to full-genesis + no prune, unchanged.
- fetchFull requests {Height: FinalizedHeight()} instead of {Height:0}; reconstructFork
prepends our own prefix below the SERVED start height (keys off served[0].Height, so it
is robust to peers that honor the suffix AND to genesis-rooted servers / adversaries).
Our own pruned prefix (below our floor) is accepted on replay because Reconcile already
pins tmp.trustFloorOverride = c.trustFloor() (slice 3) — the override is ALWAYS our own
anchor, never the fork's (asserted: TestQ2_ReconcileFloorIsReceiversNotForks — a taller
fork can't inflate our floor).
- Deep-cold (behind > the WS window, peer pruned the gap): ErrNeedCheckpoint +
ChainSyncNeedCheckpoint stat + operator log naming the remedy (a recent -ws-checkpoint
or an archive node) — surfaced, never a silent failure-to-adopt (I4/S5).
Enablement (5b): PruneBelowHorizon exported and wired into both commit sites (pruneOnCommit),
so a validator sheds heavy BondReg.Answer below its floor as finality advances — the line
that returns the MATURING box to 2 GB.
Consensus-invariants (I1-I5): I4 headline — a node within the WS window MUST catch up
(TestSuffixSync_CatchUpAroundPrunedPeer), a node beyond it correctly stalls-and-signals
(TestSuffixSync_DeepColdSignalsNeedCheckpoint), never silently adopting a forgery. I3 — the
trusted set below the horizon is our OWN finalized snapshot (we supply our own prefix),
never a peer claim. I1 preserved (no quorum re-sizing; existing finality gate governs
adoption). I5 — equivocation-on-detection now covers heights >= our finalized head (the
suffix); sub-finalized forks can't exist under finality (PE reasoning), and the #184
recent-tip drill is preserved.
A real bug the suite caught: the first reconstructFork assumed the served run starts at
reqHeight and broke the #184 equivocation-over-sync drill (the adversary serves a
genesis-rooted fork). Fix: key off served[0].Height. Ablation confirms the suffix-request
is load-bearing — forcing {Height:0} makes the catch-up RED (stuck at 13, wants 18).
Full core/adapters/cmd suites green; go vet clean. CHANGELOG (slices 1-5, now ENABLED) +
regenerated website/changelog.html; check_links OK. Interim field air (e2-medium +
GOMEMLIMIT + -inbound-cap) can retire once a pruned-net field grade is green (PE: not to a
date). Plan + PE acks: docs/thinking/2026-08-18-slice5-sync-redirect-plan.md;
principle-engineer/slice5-suffix-mechanism-ruling-PE-2026-08-18.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LDPQNH9BvDewSQ8KLuQWRN
…berate SIGKILL as an OOM (false-failed clean runs) scan_node_liveness read the SERVICE journal (`journalctl -u silt`) for `Main process exited, code=killed, status=9` as its OOM signal. But status=9 is SIGKILL — which the chaos drill sends DELIBERATELY to store-2 (flow_chaos_crash, lib.sh SIGKILL of the storage node) — so a scripted test kill was counted as a kernel OOM and false-failed otherwise-clean runs. Evidence (this misgrade is demonstrable, not theoretical): - Run cd1a719-98020 (base sheet): chaos-reprovide PASSED — store-2 was healthy and re-announced its 23 held chunks — yet infra-node-liveness still counted store-2×1. The only kill was the chaos SIGKILL; there was NO kernel oom-killer line anywhere. - Run cd1a719-26323 (MATURING): same store-2×1, kill at 18:29:02 matching the chaos drill's SIGKILL window (fail recorded 300s later at 18:34:11Z), no oom-killer signature. - store-2 held 23 chunks with mem-limit 1500M — no memory pressure to OOM on a 2GB box. - The historical "store-2×1, store-1×0" pattern is the same artifact: store-2 is the chaos target, store-1 is not, so only store-2 shows a status=9. The consequence was material: it masked the real result. Across BOTH graded runs the CONSENSUS cohort (val/maturer/sybil) had ZERO real OOMs on e2-small — where it historically crash-looped (90→74→50→15 kills). The MATURING consensus OOM is fixed; the "FAIL" was this harness false-positive. Fix: count REAL OOMs by their authoritative signatures over the whole boot journal (`journalctl -b`) — the kernel oom-killer ("Out of memory: Killed process", "invoked oom-killer", "oom-kill:constraint=") OR the Go runtime ("fatal error: runtime: out of memory"). Neither can be produced by a userspace kill -9, so the chaos SIGKILL is correctly excluded while every genuine OOM is still caught (a real oom-kill also emits status=9, so nothing is lost). Harness-honesty discipline: an oracle must not conflate a deliberate test action with a real failure. Shell-lint + bash -n clean; the next field run confirms infra-node-liveness now PASSes on a chaos-inclusive sheet. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LDPQNH9BvDewSQ8KLuQWRN
nerolabs
changed the title
nerolabs
marked this pull request as ready for review
August 18, 2026 21:52
The slice-5 integration test was written without gofmt; the CI fmt check flagged one struct-field alignment. No behavior change; tests unchanged and green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LDPQNH9BvDewSQ8KLuQWRN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
H2 — the MATURING OOM return-to-2GB: rolling retention horizon + suffix-sync
A validator's chain grows
O(all history)in the ~1.5 MB space-time bond proof(
BondReg.Answer) carried by every registration — retained forever and re-encoded wholeon every chain-serve. On a 2 GB box (build-immutable #8, the hobbyist floor) the MATURING
consensus cohort OOM-crash-looped (90→74→50→15 kills across prior field runs). This branch
bounds resident, on-disk, and served heavy payload to a recent finalized window, and makes
mesh sync safe across the pruned gap.
The five slices
6f3d39f) —RetentionHorizon() = finalizedHead − 2·BondTTL(epoch-floored,research-certified safetyDepth); Opt-1 pruned-block representation (
Block.Prune()dropsthe heavy
Answer, keeps header + consensus sigs, stores the pre-prune hash so a prunedblock still hash-links and stays valid late-reveal slashing evidence).
76fc7ef) — the Q2 gate: a pruned block is trusted (space-time re-verifyskipped) ONLY strictly below the node's OWN
trustFloor; at/above →ErrPrunedAboveHorizon;pruned-but-carries-Answer →
ErrMalformedPruned. DuringReconcilethe floor is pinned tothe RECEIVER's anchor (
trustFloorOverride), never the peer's fork — the C1/no-discount guard.a36cbd4) —PruneBelowHorizon(): the in-placeAnswershed below the prune floor(
max(2·BondTTL, BondRegHeadWindow+margin), epoch-aligned; 0 without finality or degenerateBondTTL — the PE-acked over-prune guard).
cd1a719) — suffix-sync enables the prune (PE-acked M1):fetchFullrequests{Height: FinalizedHeight()}and prepends the node's own verified prefix, reusing theaudited genesis-rooted
Reconcile; a behind node catches up around a peer's pruned window;a deep-cold node beyond the WS window gets
ErrNeedCheckpoint(never silent).pruneBelowHorizonwired into the commit path.
Consensus-invariants (I1–I5)
never a peer's claim.
Also in this branch
fix(cloudtest)(f146b45) —infra-node-livenesswas miscounting the chaos drill'sdeliberate
kill -9of store-2 (code=killed, status=9) as an OOM, false-failing cleanruns. Fixed to match the authoritative OOM signatures (kernel oom-killer + Go-runtime OOM),
never bare
status=9.Field grade (3 GCP runs on e2-small / 2 GB)
crash-looped. Slice-5 code confirmed on the wire:
184-partition(suffix-sync catch-up,"a catch-up NOT a reorg") and
10c-ws-cold-syncboth passed.Honest caveats
engages past 64. At h63 retention is ~94 MB (fine), so the prune is proven-correct local
insurance (unit tests + ablation) for deeper runs; the impactful h63 fix is the suffix-sync
serve bound + Memory-safety: the MATURING OOM fix + a boundedness sweep (bounded-then-fast) #465 backpressure + GOMEMLIMIT.
chaos-reprovide/fetch(Daemon restart orphans discoverability of stored content (provider records + placement proofs are in-memory only) #69) and10a-stall-drilltiming are pre-existing residuals,not this arc and not regressions from it.
Testing
Full
core/…suite + allmodelcheck_*green;go vetclean. New:q2_gate_test,prune_test,prune_suffix_sync_test(catch-up-around-pruned-peer + deep-cold, ablation-provenload-bearing — forcing
{Height:0}turns the catch-up RED). Plans + PE rulings/acks underdocs/thinking/2026-08-18-slice{3,4,5}-*.mdandsilt-reviews/principle-engineer/.🤖 Generated with Claude Code
https://claude.ai/code/session_01LDPQNH9BvDewSQ8KLuQWRN