Observed on hanzo-mainnet (36963), v1.36.57/58, 2026-08-06.
A node whose resync was repeatedly OOM-interrupted (8×) ended up holding and GOSSIPING a consensus-certified chain at heights ~4115–4336 that is NOT an ancestor of the network's finalized chain (finalized height 6698). Clean peers then died in two ways, in the same second: OOM (serving), or the v1.36.58 fail-closed:
error VM accepted head is CONSENSUS-CERTIFIED and the finalized block is NOT on the certified chain — refusing to orphan it
fatal SetPreference would orphan a CONSENSUS-CERTIFIED block — refusing (fail-closed)
"cannot orphan finalized block at height: 6698 to common block at height: 4336"
The fail-closed is CORRECT (it protects finality) — but the poison PERSISTS into every peer's consensus store that processed those certs, so the fatal repeats on every boot (CrashLoopBackOff) and nothing cleans it. Recovery required a node-by-node state transplant of all five validators.
Ask: (1) don't persist certified markers for chains not connected to local finality without quarantine; (2) an operator tool to inspect/purge alternate certified lineages (cmd/voteguard-era tooling is gone); (3) treat an OOM-interrupted resync's partial state as untrusted on restart.
Observed on hanzo-mainnet (36963), v1.36.57/58, 2026-08-06.
A node whose resync was repeatedly OOM-interrupted (8×) ended up holding and GOSSIPING a consensus-certified chain at heights ~4115–4336 that is NOT an ancestor of the network's finalized chain (finalized height 6698). Clean peers then died in two ways, in the same second: OOM (serving), or the v1.36.58 fail-closed:
The fail-closed is CORRECT (it protects finality) — but the poison PERSISTS into every peer's consensus store that processed those certs, so the fatal repeats on every boot (CrashLoopBackOff) and nothing cleans it. Recovery required a node-by-node state transplant of all five validators.
Ask: (1) don't persist certified markers for chains not connected to local finality without quarantine; (2) an operator tool to inspect/purge alternate certified lineages (cmd/voteguard-era tooling is gone); (3) treat an OOM-interrupted resync's partial state as untrusted on restart.