fix(arweave-scheduler): recover from chain reorganizations - #10
fix(arweave-scheduler): recover from chain reorganizations#10xylophonez wants to merge 5 commits into
Conversation
The scheduler validates each block's previous_block against the last indexed indep_hash. If it ever indexes a block the network later orphans, every canonical successor fails with 'does not extend the indexed chain' (409) forever - the frontier wedges permanently and no process can advance to tip. Observed twice on real deployments; purge+resync only re-arms it. Add bounded reorg recovery: on the conflict, walk the canonical chain down (bounded by arweave-scheduler-reorg-depth, default 50; never below 'from'), comparing network indep_hash to the locally indexed block, replacing orphaned cached blocks above the fork point, rewinding the global record to the fork, and re-indexing the canonical tail. Attempted once per pass; fails closed on no match or a fetch error. Adds reorg_fork_point_test, reorg_recovery_integration_test, and a live reorg_canonical_fetch_shape_test_.
Env-gated (REORG_SOAK=N) soak that seeds an orphaned tail at random fork depths against the live confirmed tip and asserts recover_from_reorg lands exactly on the fork point. Fresh store per iteration. 25/25 correct.
|
Proof status (lab, local): 14/14 device tests green including a live reorg-injection soak ( |
|
Overnight soak complete: 500/500 recoveries correct (14/14 tests green), random fork depths against live arweave.net data, fresh store per iteration. No wrong-fork-point or failure cases. A natural-reorg watcher continues (none observed yet — expected; deep reorgs are rare). |
|
Pushed e9e5d59: recovery redesigned after field testing on a constrained device. The fork-point walk previously fetched canonical blocks sequentially per height with the full forward-path retry budget (20x250ms), on the synchronous path that Recovery now fetches the whole bounded window in one concurrent round-trip (workers sized to cover the window, capped at 64), with a small retry budget (min(3, fetch-attempts)) so an unreachable gateway fails closed in seconds independent of depth. Measured on a clean node: fresh sync 1.2s; 1-block recovery 0.64s; 15-block 1.23s; full 51-block window against a black-holed network fails closed in ~15.5s (one bounded round-trip) vs ~103s/height before. |
…n-blocking The reorg-recovery patch placed recover_from_reorg on the synchronous sync path that ~process@1.0/now depends on, and recovered by walking the canonical chain one height at a time via fetch_block_remote (uncached, up to fetch_attempts=20 retries per height) over up to reorg_depth=50 heights. On a flaky network this stalls each recovery attempt for minutes, holds the exclusive sync lock, and starves ~process@1.0/now (zero progress) -- the field symptom. The logic was correct (500/500 soak) but the wall-time was O(depth x attempts x per-request) and a single transient fetch failure aborted the whole recovery. Recovery now fetches the bounded recovery window ([Floor..To]) in ONE concurrent round-trip via fetch_canonical_window/2 (recover_window_workers/2 gives it enough workers to cover the whole window, capped at RECOVERY_MAX_WORKERS), with a small per-height retry budget (recover_fetch_attempts/1 = min(RECOVERY_FETCH_ATTEMPTS=3, fetch_attempts)) so an unreachable gateway fails the window closed fast instead of blocking for minutes. fork_point/4 is now read-only: it locates the deepest matching height from the pre-fetched batch and returns the orphaned heights above it, and only once a valid fork is found does replace_orphans/2 overwrite them and the record rewind -- so the failure path performs no partial write. Correctness guarantees are preserved (deepest-match fork, orphan overwrite above fork, rewind to fork, fail-closed past floor / on unresolvable divergence). Verified: preloaded-store packages cleanly (no device_compile_failed); reorg_fork_point_test passes; on an isolated VPS node normal fresh sync (1.2s), shallow (0.6s) and deep 15-block (1.2s) reorg recovery all succeed and restore orphaned blocks to canonical; a full 51-block window under a black-holed network fails closed in ~15.5s (one 3-attempt round-trip) regardless of depth, versus the old sequential ~103s/height.
e9e5d59 to
bdfad3d
Compare
A preserved store can hold a stale cached block just above the indexed frontier -- an orphan a previous runtime indexed and left behind -- while the indexed chain itself is canonical. The forward fetch path is cache-first, so every sync pass re-reads the stale block, fails the previous-hash check with the does-not-extend 409, and invokes reorg recovery, which cannot clear it: recovery only repairs cached blocks at or below the frontier, and with the indexed chain already canonical it finds the fork at the frontier itself with nothing to replace. The conflict then recurs forever with zero progress. When the chain check fails, commit_batch now refetches the failing height from the network once, with the bounded recovery retry budget. If the canonical block differs from the batch's copy, the copy was stale: the poisoned cache entry is overwritten and the batch revalidated once. If the canonical block is identical -- a real reorg -- or cannot be fetched, the conflict surfaces unchanged and reorg recovery proceeds exactly as before. One refetch per pass, fail-closed, no partial writes. stale_cache_heal_test reproduces the field scenario offline: canonical chain to the frontier, stale orphan cached above it; the batch heals the cache, validates, and advances without invoking recovery, while identical (real-conflict) and unfetchable refetches surface the original 409 untouched. Verified live on an isolated node: a poisoned frontier+1 heals and resyncs in one pass (0.36s), and a true indexed-orphan reorg still recovers through the unchanged path (0.68s). The preloaded store packages cleanly and the fork-point and recovery integration tests stay green.
|
Pushed 65dbfe7: field testing surfaced a third failure mode the recovery alone cannot clear — a stale cached block above the frontier. If a block the network later orphaned is already in the local block cache at frontier+1 (e.g. cached before a restart or runtime swap), the cache-first fetch keeps serving it: validation fails, but the indexed chain below is fully canonical, so recovery correctly finds nothing to rewind — and the same poisoned cache entry wedges every subsequent pass. The commit adds a one-shot heal on the does-not-extend conflict: refetch the failing height remotely (bounded retry budget), and only if the canonical hash differs from the cached copy, overwrite the cache entry and revalidate once. Identical hash (a real reorg) or fetch failure leaves behaviour byte-identical to before — the conflict surfaces and the existing recovery runs. New regression test (stale_cache_heal_test) covers the heal, the real-conflict pass-through, and the unfetchable-refetch fail-closed case. Verified live: poisoned-cache state heals and resyncs in 0.36s; a true indexed reorg still recovers via the fork-point path (0.68s). |
The stale-cache heal keyed on the conflict height alone cannot clear an orphan RUN: a preserved store can cache an orphan that EXTENDS the canonical frontier -- passing the previous-hash check -- with the divergence only surfacing one height later. Refetching just the conflict height then yields the canonical block, which still fails to extend the stale orphan ahead of it in the batch, and the pass wedges on the 409 forever exactly as before. On a does-not-extend conflict, commit_batch now distrusts the cache for the whole batch: every height is refetched remote-only in one bounded parallel round-trip (the recovery window fetcher; a batch is at most the block-batch size), each cached entry whose canonical hash differs is overwritten, the batch is rebuilt from the fresh blocks, and validation runs once more. If the fully-remote batch still conflicts, the divergence is the indexed chain's own -- a genuine reorganization -- and the conflict surfaces unchanged for reorg recovery. A refetch that cannot be established, misaligns, or fails to write fails closed on the original conflict, and the global record is never touched on a failure path. stale_cache_run_heal_test reproduces the field shape offline: a canonical indexed chain with a two-block stale orphan run above the frontier whose first orphan extends the canonical tip; the batch heal replaces both poisoned entries and advances without invoking recovery. The single-block heal, real-conflict pass-through, and fetch-failure fail-closed cases stay green, as do the fork-point and recovery integration tests. Verified live on an isolated node: the injected orphan run heals and resyncs in one pass (0.60s, both cache entries restored to canonical), and a true indexed-orphan reorg still recovers through the unchanged path (0.78s). The preloaded store packages cleanly.
|
Pushed d08cb1b: the field device exposed a limitation of the single-height heal — the stale cache held an orphan run, and the first orphan extended the canonical frontier, so it validated cleanly and the conflict surfaced one height later. Healing only the conflict height refetches a canonical block that then fails against the stale orphan ahead of it in the same batch, wedging again. The heal now distrusts the cache for the whole batch on a does-not-extend conflict: every height of the batch (≤ block-batch, default 8) is refetched remote-only in one bounded parallel round-trip, cache entries whose hashes differ are overwritten, the batch is rebuilt from the fresh blocks, and validation retries once. A fully-canonical remote batch that still conflicts means the divergence is in the indexed chain → the existing recovery runs unchanged. Fetch failure or misalignment fails closed on the original conflict; the global record is untouched on every failure path. New regression stale_cache_run_heal_test reproduces the field shape exactly (orphan run whose first block extends the canonical tip). Verified live: the poisoned-run state heals and resyncs in 0.60s; single-block heal, real-reorg pass-through, and fail-closed cases remain green. |
1 participant
Problem
The Arweave scheduler indexes the confirmed chain by validating each block's
previous_blockagainst the last indexed block'sindep_hash. If the nodeever indexes a block the network later orphans in a reorganization, every
subsequent canonical block fails the previous-hash check:
There is no recovery path, so the frontier wedges permanently — every
~process@1.0/now500s while already-materialized reads keep working. Only apurge and full re-sync clears it, and that only re-arms the same failure.
Observed three times on a real deployment (Pixel-class node), including once
within hours of a full clean re-sync.
On confirmation depth: the wedge-era node ran
arweave-scheduler-confirmation-depth: 10(the stock default — deeper than theprod machines' 3), so this is not explained by a missing depth setting. The
suspected aggravator is
arweave-scheduler-gateway-first: true:current/heightand blocks are sourced from public gateways, and a lagging or fork-side gateway
view skews the
tip - depthconfirmation math, letting an orphan through evenat depth 10. Nodes reading through gateways (rather than direct peers) appear
substantially more exposed, which would explain why prod peers rarely hit it.
Regardless of the trigger, once wedged there is no way back — that is the
defect this PR addresses.
Fix
Bounded, parallel, fail-closed reorg recovery in the sync loop. On a
does not extendconflict (at most once per sync pass):[max(from, to - depth) .. to]from thenetwork in one concurrent round-trip (workers sized to cover the window,
capped at 64), with a small retry budget (
min(3, fetch-attempts)) so anunreachable gateway fails the recovery closed in seconds, independent of
window depth — it never holds the sync lock for minutes;
canonical
indep_hash(read-only over the pre-fetched batch);above it, rewind the persisted record to the fork point, and let the ordinary
forward path re-index the canonical tail. The failure path writes nothing.
If no indexed block within the bound matches, recovery fails closed and the
original 409 is surfaced unchanged. Window bounded by
arweave-scheduler-reorg-depth(default 50, never belowfrom).Scope
Single file
src/preloaded/process/dev_arweave_scheduler_sync.erl. One newconfig knob (
arweave-scheduler-reorg-depth). No change to steady-statebehaviour when no reorg occurs.
Tests (all green via
rebar3 device test)reorg_fork_point_test— fork-point location over a batch: deepest match,fail-closed past the floor, orphans identified above the fork.
reorg_recovery_integration_test— full path against a real store: rewind +persist + orphan overwrite; fail-fast window and no-partial-write cases.
reorg_canonical_fetch_shape_test_— live check that canonical blocks carrybinary
indep_hashchaining viaprevious_block(skips offline).reorg_soak_test_— reorg injection at random depths against thelive chain; 500/500 correct recoveries.
1.23s; full 51-block window against a black-holed network fails closed in
~15.5s (one bounded round-trip).