Advertise-path corpus scan holds the single DB connection while page-faulting: a canonical dies after ~22h and the worker floor does not help

[emooreatx]

In situ: the advertise path's corpus scan takes a node down for good after ~22 h

Filed from the CIRIS canonical (ciris-canonical-1, 2 vCPU / 3.9 GB, seven
containers). Captured while the node was hung, not reconstructed afterwards.

Companion to CIRISServer#506; this is the edge-side half.

What the node looks like right now

started  2026-08-28T22:22:41Z     restarts=0
now      2026-08-29T20:41:21Z     → 22.3 h uptime
:4243    Recv-Q=1                 health=000 after 6.0 s (from inside the netns)
disk     35 MB/s sustained read, continuous
majflt   3,146,406 cumulative

Thread table at the moment of the hang — the runtime is not CPU-starved:

675252  ciris-node   state=D  wchan=wait_on_page_bit_common   ← the holder
675254  ciris-node   state=S  wchan=futex_wait_queue_me
675256  ciris-node   state=S  wchan=futex_wait_queue_me
675259  ciris-node   state=S  wchan=futex_wait_queue_me
323269  ciris-node   state=S  wchan=futex_wait_queue_me
324450  ciris-node   state=S  wchan=futex_wait_queue_me
454223  ciris-node   state=S  wchan=futex_wait_queue_me

One thread in uninterruptible page-wait. Six idle. HTTP still dead.

The mechanism is a lock convoy, not scheduling starvation

This distinction matters for which fix works, and we got it wrong once already.

CIRISServer#501 was scheduling starvation — one worker pegged at 99.9%, cured by a
worker-count floor (MIN_WORKER_THREADS = 4, shipped in server 0.5.194). That
floor is live here — you can count four ciris-node workers above — and this
node is down anyway.

Because the contention is not CPU. SqliteBackend holds
conn: Arc<Mutex<Connection>> — one connection for the entire node — and any
large scan takes conn.lock() and holds it across the read. While the holder is
in wait_on_page_bit_common, every other DB-touching task parks in
futex_wait_queue_me behind that mutex. Adding workers adds queue positions, not
throughput. That is exactly the shape above: one D, six S, nothing served.

/v1/health reads the store, so the node's own liveness surface is inside the
convoy. That is why this ran 22 hours unnoticed.

Why the advertise path is the producer

list_attestations_since(None, u32::MAX) per plane, per peer, per round is an
unbounded corpus scan on a cadence. The arithmetic on this host makes the outcome
inevitable rather than unlucky:

corpus (server db)        1.3 GB
page cache (whole box)    808 MB          ← already smaller than one node's corpus
free                      138 MB
transport_destinations   11,034 rows      (against 748 federation keys)
primed peers                430

Every round re-reads a corpus that cannot be cached, so every round page-faults
from disk, and each fault happens holding the single connection mutex.

Escalation is visible against your own earlier measurement: #480 recorded
40.8 MB/s; this node has been observed at 81–118 MB/s and is at 35 MB/s while
otherwise idle. The delta tracks directory growth, so this gets worse on its own.

Corroborating: edge logged 1 round and 1 deliver in ten minutes — the
replication plane is barely progressing while the disk runs flat out — alongside
28 retry/backpressure warnings in 30 minutes.

Impact

• A canonical goes permanently unreachable after ~22 h and does not self-recover.
  Only a restart clears it, and the clock starts again.
• It is silent: no crash, no panic, restarts=0, and the health endpoint that
  would report it is itself blocked.
• It scales the wrong way — the bigger the directory, the sooner it arrives.

Ranked asks

1. Bound the advertise scan (CIRISEdge#531's predicate hoist). A scan whose
   cost is O(corpus) per peer per round cannot be made safe by anything downstream.
2. Do not hold the connection guard across a large read. Even bounded, a scan
   that holds the only connection while faulting will convoy. Chunking with the
   lock released between pages would cap the convoy at one page rather than one
   scan.
3. A liveness surface that survives a blocked runtime — this is the reason for
   the 22 hours. Any health path that touches the store is inside the failure it is
   meant to report.

Our side

Not clean either, and it compounds this: the server runs storage_summary()
(SUM(pgsize) FROM dbstat — a walk of every page) from trace_plane_watch every
15 min and retention hourly, on the same single connection. On this box
ciris-status embeds the server too and is thrashing on its own 374 MB store
(majflt=768,585), so roughly ten full-page walks an hour across two nodes keep
the cache cold between your rounds. Tracked as CIRISPersist#775, and we are moving
our own reads off the runtime workers.

That is a compounding contributor, not the cause: our walks are periodic, the
81–118 MB/s is continuous.

Repro

Uptime. Leave a canonical-sized node running ~22 h with a live peer set; watch
/proc/<pid>/task/*/wchan for wait_on_page_bit_common and Recv-Q on the read
API port climb while CPU sits at zero.

Happy to give access to the affected host — ask and we will arrange it rather than
publishing connection details here.

Related: CIRISServer#506 · CIRISServer#501 (the other mechanism, already fixed) ·
CIRISEdge#531 · CIRISEdge#480 · CIRISPersist#775

[emooreatx]

Version this was observed against, which the report should have stated up front: ciris-edge v18.11.0.

The body names only server 0.5.194 (for the MIN_WORKER_THREADS = 4 worker floor) and no edge version at all, which leaves the wrong thing to guess on an edge issue. Filling that in:

edge	v18.11.0 (released 2026-08-28; still the latest release)
persist	v38.6.0
server on the affected canonical	0.5.194
issue filed	2026-08-29

Verified from the pin rather than from memory — git show v0.5.194:Cargo.toml gives ciris-edge = { … tag = "v18.11.0" }.

Worth noting for anyone reproducing: server 0.5.195 pins the same edge tag, v18.11.0. That matters because the two nodes in the comparison that produced this evidence differed only in the server version, so edge skew is ruled out as a cause rather than merely assumed — the contention reproduces on identical substrate.

Since v18.11.0 is still the current release, this is against the present tree; nothing here has aged out.

[emooreatx]

Confirmed on v18.11.0, and the mechanism is not the one the report cites. Worth correcting, because it changes the fix.

The cited call no longer exists

list_attestations_since(None, u32::MAX) is the pre-v18.7.0 shape. On v18.11.0 the advertise read is cursored and bounded: a per-(peer, plane) watermark supplies since, and sweep_page_budget() caps a page at min(operational_page_limit, sweep_page_rows) — default 1024 rows. v18.11.0 additionally removed the per-row serde_json::to_value from that loop.

So the send axis is already fixed. Something else is producing the 35–118 MB/s.

What is actually re-reading the corpus every round

The holdings (receive) axis, and bridge.rs:2197 states the constraint plainly:

the holdings view is the OTHER whole-table read per plane per round (want = remote ∖ holdings) … Every arm here is now PAGED. But NEVER watermarked: want is computed against this set, so a partial holdings view leaves held rows in want forever … SweepWindow::Full is the type-level statement of that: pages for memory, complete in result.

That is the whole finding. #531 made holdings paged, which bounded memory — and deliberately did not bound I/O, because want = remote ∖ holdings is only correct against a complete set. A watermark there would recreate #416's non-convergence.

session.rs:482 calls local_holdings(kind) per round; that resolves to a SweepWindow::Full drain looping to MAX_FULL_DRAIN_PAGES (4096) × 1024 rows. Every round, every plane, the full corpus is read from disk in 1024-row chunks — bounded memory, unbounded I/O. On a 1.3 GB corpus with 808 MB of page cache for the whole box, every one of those pages faults, and each fault happens holding persist's single connection.

So your ask 1 ("bound the advertise scan") is already done and is not the producer; ask 2 ("do not hold the connection across a large read") is partly done — the scan is chunked and the sweep permit is released between pages, which caps the convoy at one page rather than one scan, exactly as you asked. What remains is that the number of pages is O(corpus) per round.

Which means the fix is a different shape

Cursoring holdings is ruled out by the convergence invariant. The tractable direction is to stop deriving holdings from disk every round: seed the held-hash set once, maintain it incrementally on admit (edge sees every ApplyOutcome), and re-seed periodically so a writer outside edge's view cannot make it silently stale. That converts O(corpus) per round into O(admits) per round plus an occasional full pass — amortised, not eliminated, because correctness here really does require completeness.

I have not built that yet; it is a real change with a real staleness question (CIRISServer writes CEG rows too, so the re-seed interval is the correctness knob, not a nicety). Flagging the diagnosis first so we agree on the mechanism before I pick an interval.

Shipping now for ask 3

Ask 3 is version-independent and I have it in the next cut: a store-free liveness surface. seconds_since_last_round() reads ONE atomic — no store, and no lock that a store-touching path can hold — so it still answers while the runtime is convoyed. On the hung canonical it would have read 22 hours stale from the first minute. It says the runtime stopped progressing; it does not say why, which is the honest scope for a liveness probe.