seller node heartbeat/watchdog task parks silently — no ticks, no errors, 0% CPU (distinct from #171)

[orveth]

Second failure signature found during the #171 field investigation, on a different host from #171's reconnect wedge. DISTINCT from #171: there the watchdog task is alive and failing loudly every 60s tick; here the task stops running entirely.

Observation (3 independent dev seats @ e14942a, macOS, including one running a local #171 receiver-patch build — so this is upstream of the reconnect path)

• Process alive, 0.0% CPU, parked (sampler: main thread idle, worker threads in kevent awaiting network). Not spinning, not crashed.
• After boot: normal "seller node heartbeat+watchdog enabled (every 300s…)" + resubscribe backlog drain — then TOTAL silence for 8–17+ min (≫ the 300s interval): zero heartbeats published, zero failure lines, zero stall/recovery lines, zero relay errors.
• So the heartbeat+watchdog task is NOT firing at all. Consequence: the seller: in-process relay-stall watchdog — detect dead subscription via own heartbeat, reconnect with overlap (closes #142) #150 stall detector cannot detect anything — its own task is the thing that stopped. Seats only recover via an external log-staleness watchdog (process restart).

Hypothesis (labeled, not asserted)

The heartbeat task blocks on a relay publish that neither completes nor errors — e.g. a dead socket where the send future never resolves — parking mid-publish: no success line, no failure line, no further ticks. Fits 0% CPU + kevent + total silence. If so, the fix shape is a timeout on the publish (and any other await in the heartbeat/watchdog loop) so the task can never park permanently — the watchdog must be structurally unable to stop ticking.

A thread sample/spindump of a parked seat (showing the exact await) is being collected and will be attached.

Relationship to #171

Independent trigger-side bug. #171 = recovery path structurally fails once a stall is detected. This = the detector itself silently stops, so a stall is never even detected. Both must be fixed for unattended durability; a seat with only #171 fixed can still park forever with this one.

[orveth]

Operator note from field re-supervision of the fix (should land in ops docs / README alongside the fix):

The fix intentionally quiets per-heartbeat SUCCESS logging (only failures/stalls/recoveries log now). Consequence for external supervision: log staleness is no longer a health signal — a healthy idle seat is silent indefinitely. A staleness-based watchdog will needlessly restart healthy seats (observed in the field: healthy seats restarted every ~18min until the supervisor was changed).

Correct external supervision of the fixed daemon: process-liveness-based (pid presence / process aliveness), restarting only on a dead process — because the fix makes real problems log-and-recover in-process. The daemon changed what "healthy" looks like from the outside; supervisors must change with it.