seller-node cutover: mobee sell boots the durable seller node

[orveth]

seller-node cutover: mobee sell boots the durable seller node

Makes the durable seller node (crates/mobee-core/src/seller_node/) the real path for mobee sell: a sqlite store + outbox + reconcile_on_start replace the in-memory single-flight lifecycle. One process, one identity, restart-safe. The superseded in-memory seller_daemon is deleted — cleancut: the code reads as if the node were always the seller path (no transitional shim, no cfg-gated fallback).

Off the phase-1 checkpoint; rebased onto origin/main (includes the #165 telemetry-flake fix). Diff is cutover-only (no drive-by refactors), net ≈ −6.3k lines. 8 commits, each green at its tip.

Invariant → tooth

Inv	Property	Tooth	Bite
1	Award re-bind across restart	restart_then_award_rebinds_the_parked_claim (#154) kept green through the wiring	phase-1
2	Delivery re-push determinism	resume_redelivery_is_deterministic_and_never_double_publishes + seller_git tooth_delivery_snapshot_uses_journaled_date	STRONG: neuter deliver_and_enqueue dedup → red (rc=101)
3	Payment ordering	redeem_classification_finalizes_and_never_forges_from_a_breadcrumb + seal_sender_must_be_the_bound_offer_buyer + realized_mint_must_be_in_the_stored_creq + receipt_collect_dedups_a_replayed_payment	STRONG: neuter never-forge branch → Finalize → red (rc=101)
4	Session resume	resume pass re-drives awarded/executing jobs at boot: resume_selects_awarded_or_executing_not_delivered_or_terminal + boot_resume_re_drives_awarded_undelivered_job_delivery_lands_once	delivery-once via deliver_and_enqueue dedup
5	Reconcile does not release parked claims	phase-1 #154 adjacent teeth (open_claims preserved)	phase-1
6	Claim GC (#153)	phase-1	phase-1
7	buzz leg degrades, never bricks	by construction: the run loop wires no start_buzz(); the node never contacts buzz, so a down buzz relay cannot brick boot	—
8	creq bound at claim time (N-4)	delivery_preimage_binds_stored_creq_not_drifted_config + unit delivery_hash_binds_stored_creq_not_drifted_config	strong (unit)

Watchdog preserved (#150 + #162)

The #150 relay-stall watchdog is ported verbatim: own-heartbeat (kind-30340) each heartbeat tick; if no own heartbeat round-trips within interval*missed_intervals, disconnect → reconnect → re-NIP-42 → resubscribe-all with since=overlap (last-good −60s). Subscriptions cleared only after a successful reconnect+auth (never reconnected-but-deaf). #162 folded in: connect-phase failure retried 3× short-backoff within one recovery; success logs RELAY-STALL recovery SUCCEEDED (attempts=N, outage=Ss). Ported verbatim: stall_threshold_secs, subscription_stalled, self_heartbeat_subscription_filter, STALL_OVERLAP_MARGIN_SECS. Tooth: watchdog_stall_math_clamps_and_trips_only_at_threshold.

Custody

Seller identity key stays at two sites (signer actor + authenticated relay client). Push NIP-98 auth, receipt-hash signing, and the gift-wrap NIP-44 decode all route through the signer actor (http_auth_header, sign_receipt_hash, unwrap_payment_wrap). The cdk P2PK receive needs the raw cashu key (no signature-only path); it is derived inside the actor from the seller key and exported only as the derived payment secret — the identity key never leaves the actor.

Follow-up (recorded): move the cdk receive INSIDE the wallet actor so the wallet actor owns the cashu key entirely (actor-to-actor handoff, nothing exported). Deferred out of this slice.
Follow-up: ACP session/load resume (resume the agent's own session vs a fresh re-run + workdir inspection) — invariant 4 is shipped in its fallback form here.

Gates (rebased tip)

• money suite (-p mobee-core --no-default-features --features gateway,git-delivery,wallet --lib): 490/0 — release and debug (rc=0)
• default test (cargo test -p mobee-core, CI build-default): 162/0 (rc=0)
• acp full combo (cargo build -p mobee-core --features acp,gateway,git-delivery,wallet): rc=0
• release binary (cargo build -p mobee --release --features acp,wallet, CI build-acp): rc=0
• fmt / clippy: repo-wide rustfmt/clippy version drift — origin/main's own untouched files (e.g. budget.rs) fail fmt --check and clippy -D warnings under the pinned toolchain, and CI enforces neither. No reformat (would be a 56-file drive-by). New code is fmt/clippy-clean (one too_many_arguments allow on delivery_receipt_preimage, matching the store's existing allow). Repo-wide fmt/clippy+toolchain-pin+CI is filed as its own chore.

Live smoke — PENDING

Testnut end-to-end through the node path (post→claim→award→deliver→receipt), receipt id to be posted as a PR comment. Prereqs verified: relay mobee-relay.orveth.dev reachable, testnut mint reachable, claude agent available (via npx). Running against fresh throwaway homes.

acceptance
- [x] PR open against main, branch rebased, diff = cutover only (no drive-by refactors)
- [x] all invariants covered by tests; strong-form bite pasted for 2/3 new (re-push determinism, payment ordering)
- [x] 5-gate set: test gates + release build green (rcs above); fmt/clippy = pre-existing repo-wide drift (documented)
- [x] old in-memory lifecycle deleted; grep proves no dead remnant behind a cfg
- [ ] live smoke: testnut end-to-end through the node path, receipt id pasted  ← PENDING (PR comment)

🤖 Generated with Claude Code

[orveth]

Paired cutover smoke — BUYER side (PAID through the seller-node path) ✅

Live testnut trade through the durable seller node (this PR's tip binary), driven by the
buyer daemon from PR #166 (2-call auto-award). Fresh throwaway buyer, budget hard-capped
total_budget_sats=6, on wss://mobee-relay.orveth.dev / testnut.cashudevkit.org.

Buyer fc36098e6ecb0c82dc51dc1849f3a3be6f7f26387facec66fb7506fdda0dbcae ·
Seller node 231ba69a102a164cc9938c9ec520f0100f2692c5a567ddbe45dcffbf00becfa9 ·
Job 1d1df7e2b54d0684be1d433d947ed734b5983a874df3480cf91c594607860820

Two agent calls (post_job targeted @4 sat + collect); the daemon drove the award:

• auto-award (no award_claim): pending_awards=awarded, reservation reserved:4 while the node executed.
• node delivered commit bb55cb136faeffd07e1e55c9e1b7b0b469d4f457; files ["hello.txt","seller-run.jsonl"]; hello.txt == hello from the mobee seller node smoke (agent ran correctly).
• collect paid: receipt 1bec6e722c6caade397c1c220d3e94e79cb1a6472ced83cf849fa92885d804f5 (kind-3400 co-signed), attempt 8de68ff64ee455e24c6033488a1f2f0c6ee6e67bb366df53a6a73bdc77956cd0, state=Closed, spent_total=4.

Invariants observed live:

• reserved → spent: reservation 1d1df7e2 reserved:4 → spent:4 across the collect.
• pays exactly once: re-collect returned the same attempt_id with spent_total still 4; spent.jsonl one entry. Tip-match ran inside the sealed pay path (never auto-filled).

First attempt — a real finding (not a node bug)

The first job (e21568c4…) failed on the seller side: the claude-agent-acp adapter's bundled
claude binary hit the NixOS dynamic-link code 127 (the known CLAUDE_CODE_EXECUTABLE gotcha).
The node claimed → awarded → executing → invoked the agent correctly; only the harness couldn't
start. Buyer-observed gap: the node marked the job failed without publishing a buyer-facing
kind-3404 feedback, so my resolver saw silence, not a decline — the same buyer-facing-feedback-
dropped class the gate flagged. cutover2 is folding the 3404-on-exec-failure into the delta. After
the CLAUDE_CODE_EXECUTABLE fix + a fresh seller home, the retry above ran clean.

Cost: 4 sats testnut (well under the ≤10-sat shared ceiling). Both sides exercised through the node path.

[orveth]

Live testnut smoke — PASSED (paired with #166's buyer)

Seller = this PR's tip binary (--features acp,wallet) booting the durable node on a fresh throwaway home; buyer = worker on #166 (its receipt comment is above). Live on wss://mobee-relay.orveth.dev + testnut. This is the deployable-path proof: every stage ran through the cutover code.

Seller-side sequence (node path), job 1d1df7e2…:

seller node claimed   job_id=1d1df7e2… buyer=fc36098e… amount=4 (awaiting award)
seller node awarded   job_id=1d1df7e2… — executing        # agent ran (claude via npx)
seller node delivered job_id=1d1df7e2… commit=bb55cb136faeffd07e1e55c9e1b7b0b469d4f457 result enqueued
seller node wrap seen event=dd56e5d48bfe66f6631ee4a88dc71c11804c9d90ef254a5efa227d490dd54340
seller node collect ok: job_id=1d1df7e2… amount_received=4 expected=4 mint=https://testnut.cashudevkit.org
seller node paid       job_id=1d1df7e2… amount=4 receipt=dd56e5d48bfe66f6631ee4a88dc71c11804c9d90ef254a5efa227d490dd54340

Buyer side (above): kind-3400 co-signed receipt 1bec6e72…, commit bb55cb13… collected, hello.txt == "hello from the mobee seller node smoke", reserved:4 → spent:4, re-collect same attempt / spent still 4 → pays exactly once. Seller node redeemed the gift-wrap (collect ok 4/4 → paid); the money-safe pay path (breadcrumb → swap → receipt dedup) settled.

Also proven live at boot: reconcile, NIP-42 auth, relay-git delivery-repo announce+seed, discoverability, and the ported watchdog (heartbeat+watchdog enabled: kind-30340 every 300s; reconnect within 900s).

Honest findings the live run surfaced (candidates for the delta commit / follow-up)

1. Env, not code: first attempt failed at the agent spawn — claude-agent-acp bundles a dynamically-linked claude NixOS can't exec (code 127). Fixed with CLAUDE_CODE_EXECUTABLE. The node did everything right up to spawning the agent. (The buyer: MCP connect-or-spawn — daemon-owned money path + 2-call trade loop #166 worker hit the identical blocker independently.)
2. Idempotency gap (safe, but worth a guard): a second award event (different award_id) for the already-delivered job re-triggered execute_job (record_award returns Awarded::New because the job row INSERT OR IGNOREs while a fresh award_id inserts). The redundant re-execute re-ran the agent and its push was correctly rejected (non-fastforwardable). No money impact — pays-once held and collect_receipt marked paid regardless — but execute_job should early-return when the job is already executing/delivered/paid. Proposing to fold the guard into the delta commit.
3. Buyer-facing feedback on execution failure: on a failed execute the node marks the job failed silently (no 3404), so the buyer sees silence — the same class as the gate's under-rate-feedback bounce. Folding a fail-feedback line into the delta.

[orveth]

Delta ccfcdc2 — gate bounce + smoke findings fixed (fix-before-merge)

The gate's two items plus the two robustness gaps the live smoke surfaced, one commit, all teethed (money suite 494/0):

1. UnsupportedVersion classification — on_offer now matches OfferParseError::UnsupportedVersion distinctly (was collapsed into generic "unparseable", regressing seller_daemon: distinct UnsupportedVersion offer-skip reason-code (closes #111) #146 / protocol: machine-readable reject taxonomy — rate-declines, version-rejects, and work-errors are indistinguishable on the wire #117). Tooth: unsupported_version_offer_is_a_distinct_parse_refusal.
2. Under-rate refusal feedback — a targeted-to-self under-rate offer emits a kind-feedback status=error to the buyer (open-pool under-rate stays log-only; a lapsed skip never emits). ClaimDecision::Skip now carries a typed SkipReason. Tooth: under_rate_feedback_only_for_targeted_under_rate_rate_gate.
3. Execute idempotency guard (money-adjacent) — refined root cause, credit to the buyer worker's tag-level dig: the duplicate execute is not a race. mobee publishes two kind-3405s per settled trade by design — the award at selection AND the accept at settlement — indistinguishable on the wire (both status=accepted, same offer+claim e-tags; c57046fd=award, 7a77a6c5=accept). Without a guard the accept-3405 re-triggered execute_job for the already-delivered job (record_award returns Awarded::New because the jobs row INSERT-OR-IGNOREs while the fresh award_id inserts), re-running the agent — operator compute paid twice on every trade — with the push correctly rejected non-fast-forward and the delivered job transiently marked failed. execute_job now early-returns unless the job is still awarded/executing. Tooth: delivered_or_paid_job_is_not_re_executed, STRONG BITE (neuter should_resume_execution to admit terminal states → rc=101 red).
4. Execution-failure feedback — every post-award execute failure now sends the buyer a status=error (generic, path-free reason) so they don't wait on a delivery that never comes — the silence the smoke's first attempt exposed. Tooth: execution_failure_feedback_is_a_buyer_addressed_error.

Follow-up (tracking issue, later slice): mobee-core should differentiate award vs accept on the wire (accept e-tags the RESULT id) so sellers can tell binding from selection without the guard.

Gate re-check requested. All the money/delivery invariants + prior teeth stay green; strong bites hold on inv-2 (deliver dedup), inv-3 (never-forge), and now the idempotency guard.

[orveth]

Review-fix delta @563672ed — the watchdog can now detect, and can now recover (#171)

Scope note up front, because this delta is wider than "fix the reconnect": review of #171 turned up two structural bugs plus, in the same subscription path, a filter set the cutover had dropped. Sections 3 and 4 are deliberate scope, ruled in rather than assumed — flagging them so the gate reviews them consciously.

Everything below is one commit, 563672ed, on the frozen tip.

---

1. The trigger — the watchdog's liveness probe was unsatisfiable

RelayPool::send_event_to saves every published event into the client's own database (nostr-relay-pool-0.44.1/src/pool/mod.rs:767). When the relay echoes it back, the inbound handler checks the database first and returns without notifying for anything already present (relay/inner.rs:1215; the notification lives only in the NotExistent arm at :1256). A client can never be delivered its own published event.

The #150 watchdog's whole liveness signal was "my own kind-30340 heartbeat round-tripped." Unobservable by construction: last_self_heartbeat_seen was set once at loop start and never refreshed, so subscription_stalled went true at stall_threshold and stayed true — on every node, forever, healthy or not.

Field proof it was never a real stall: the devsoak specimen fired its first stall at 180s (line 21, earliest possible tick) and then went on to claim, deliver, and get paid twice (lines 29-101), receiving offers, awards, and kind-1059 payment wraps throughout. It was healthy the entire time and self-harming.

Why the old daemon didn't show this: its publish_heartbeat built a throwaway Client::new per publish (seller_daemon/mod.rs:2678-2684), so the event landed in a different client's database and the long-lived subscriber saw the echo as new. Old's probe therefore worked — and its detections were genuine: the only confirmed real stalls in the fleet data are two episodes on an old-binary seat (7/24-25), real subscription deaths, correctly detected, recovered by SDK background reconnect. Every dev-side detection is a false positive.

So the distinction is sharp: old's watchdog was satisfiable and therefore meaningful, but only accidentally so — it depended on a publish path that reads as pure inefficiency. The cutover's consolidation onto one authenticated client, right for custody and for not opening a socket per publish, silently converted a working detector into one that can only ever cry wolf. Behaviour preservation needs verifying per-invariant, not per-intent.

Replacement: a limit(0) REQ whose EOSE proves the relay is serving OUR subscriptions on THIS authenticated session. No cooperating publisher, no stored events, and it additionally catches a CLOSED-while-socket-up subscription death that the old probe could not have seen. The own-heartbeat subscription is removed with it — it could only ever have returned nothing. Heartbeat publishing stays: kind-30340 is presence/discovery other agents read, and that part always worked.

Tooth — liveness_probe_answers_only_on_an_authenticated_session, both directions, because a probe that can only answer one way is the defect being fixed:

• answers on a healthy authenticated session;
• reports loss within the window on a session the relay refuses to serve (auth-enforcing fixture + auto-auth off → CLOSED, never an EOSE). This half is what pins "on this session".

Bites (rc=101 each, tree restored and verified identical after): probe returns true on timeout → negative half red. Probe matches a different sub id → positive half red.

The negative half is also what keeps the probe honest against the real thing: the two genuine old-fleet stalls were subscription deaths, and "relay stops serving our REQs" is exactly the condition it asserts. A replay fixture built from those episodes' since= cursors would test overlap-backfill (that events published during an outage are re-delivered after recovery) rather than probe firing — a worthwhile separate tooth, not this one.

The SDK trap is filed as #174 — it's a standing hazard for any future client that both publishes and subscribes (the buyer daemon's delivery-watcher slice especially), and in tests it presents as a transport bug.

---

2. The wedge — recovery could never succeed

reconnect_and_resubscribe took the relay notification receiver before client.disconnect(), and Relay::disconnect emits RelayNotification::Shutdown into exactly that channel (relay/inner.rs:1288, sole emitters being disconnect() and ban() — not the reconnect(true) auto-actor, which only breaks on Shutdown at :1406). The auth wait consumed our own Shutdown and reported relay shutdown before NIP-42 authentication on a socket that had authenticated perfectly.

0 successes in 969 field attempts. Not flaky — structural. Wire trace from the relay operator confirms it from the other side: exactly 3 auth→close per 60s cycle, every close preceded by a successful NIP-42 auth, zero relay-side close reasons. Client-initiated, both sides describing the same bug.

Presented as a silent wedge rather than a crash because that Shutdown is sent with external = false (inner.rs:395-420) and so is never converted into a RelayPoolNotification::Shutdown — the run loop watches the pool channel, never saw it, and kept heartbeating with what it believed were dead subscriptions.

Fix: disconnect first, then take the receiver — still before connect, so the one-shot Authenticated cannot be missed either. Both halves load-bearing. Rejected tearing down and rebuilding the Client: re-reads the identity key and rebuilds the publisher for zero added correctness, a bad trade on a custody path.

Tooth — reconnect_reauthenticates_and_delivery_resumes_in_process. Two things make it real:

• The fixture enforces NIP-42 before serving a REQ (RelayBuilderNip42Mode::Both). The previous watchdog teeth ran against a LocalRelay that served reads unauthenticated, which made the auth step decorative — that parity gap is how the ordering bug shipped green.
• It asserts event delivery resumes, not that the function returned Ok. A live socket coexists happily with dead subscriptions; a return-code assertion would be the same false green.

Bite: restore the original ordering → red with the field's exact string, RelayAuthError("relay shutdown before NIP-42 authentication") (rc=101).

---

3. Restored: the open-pool offer filter and the #t=mobee guard (deliberate scope)

The cutover dropped both. origin/main's offer_subscription_filters (seller_daemon/mod.rs:3148-3182) carried a targeted #p==self filter plus an un-pinned open-pool filter under claim_open_pool; the node subscribed targeted-only, unconditionally. claim_open_pool survived only in the classify gate (run.rs:286), so an opted-in seller ran a claim gate over offers its REQ could never deliver, and offer_backfill_secs (a defaulted config knob) was ignored outright. The award filter kept #t=mobee; the offer filter had lost it.

Both filters ride ONE subscription, as main's comment insists they must — registered separately, the un-pinned filter delivers stored events but never live ones.

Ruled in rather than deferred: the chartered CLOSED degrade-to-targeted is a no-op without an open-pool half to degrade, and both live in the exact function the CLOSED arm touches, so a follow-up PR would re-open the same lines. Reported to #172, which this rescopes: dev's open-pool sellers weren't deaf-on-CLOSED, they were deaf by omission from boot.

Tooth — open_pool_filter_rides_the_offer_req_iff_opted_in: un-pinned filter present iff claim_open_pool; #t=mobee on both; offer_backfill_secs = 0 → live-only (since(now) + limit(0)); a window → since(now - window) + limit(OFFER_BACKFILL_LIMIT); post-stall resubscribe → overlap cursor on both.

---

4. Observability: a CLOSED arm (deliberate scope, minimal)

A relay CLOSED kills one subscription while the socket stays up, so no socket-level check can see it — close the 1059 leg and the node heartbeats happily while every payment misses. The loop had only Event and Shutdown arms, so CLOSED meant silently deaf.

Now: loud log naming which leg died (stable per-role subscription ids — mobee-offers / mobee-awards / mobee-wraps / mobee-liveness-probe — instead of generated ones, which is what made a CLOSED anonymous); the offer REQ degrades to targeted-only rather than being lost wholesale, re-armed by the next successful recovery; everything else repairs through the one paced recovery path rather than a second ad-hoc resubscribe. Never fatal.

Kept minimal per the rescope — CLOSED was refuted as the wedge trigger, and the per-subscription reconciler stays #172's slice.

---

5. Pacing

Exponential backoff between recovery attempts, capped so one bounded recovery still fits inside a heartbeat interval. The flat 2s interval is what let each wedged node re-dial shared infrastructure three times a minute, indefinitely.

Tooth — recovery_backoff_grows_and_stays_capped.

Also: the log now names which path restored the receive leg — manual recovery or the SDK's background reconnect. Their indistinguishability is how a manual path that never once succeeded went unnoticed.

---

6. Ride-alongs

• UnsupportedVersion tooth re-pinned to the routing. It asserted parse_offer's error enum, so a revert that collapsed on_offer's version arm into the generic bucket passed green. Parse refusals now route through one function (offer_parse_refusal), and the tooth asserts the emitted refusals differ and that the version one names the version — collapsing them is red.
• The paid line no longer mislabels the wrap id as a receipt. It logged the kind-1059 gift-wrap id as receipt= (visible in the specimen: wrap seen event=580d5ddd… then paid … receipt=580d5ddd…). The seller never sees the co-signed kind-3400 id on this path — the buyer publishes it — so the field is named payment_wrap= for what it actually is, rather than sending an operator grepping for a 3400 that will never match.

---

Gates — frozen tip 563672ed, pasted rcs

• money suite (-p mobee-core --no-default-features --features gateway,git-delivery,wallet --lib): 498 passed, 0 failed — rc=0
• default (cargo test -p mobee-core): rc=0
• acp full combo, all targets (--features gateway,git-delivery,wallet,acp --lib --tests): rc=0
• release binary (cargo build -p mobee --release --features acp,wallet): rc=0
• fmt/clippy: unchanged from this PR's existing note — pre-existing repo-wide toolchain drift, CI enforces neither, no drive-by reformat. Zero warnings from the touched file.

Builds ran in a separate CARGO_TARGET_DIR, never the workspace's live target.

Red-on-revert summary

revert	tooth that goes red	rc
receiver taken before disconnect()	reconnect / delivery-resumes	101
probe returns true on timeout	probe, negative half	101
probe matches a different sub id	probe, positive half	101

Tree restored to the commit and verified byte-identical after each.

Not addressed here

• The live testnut smoke in this PR's acceptance block is unchanged and still pending.
• seller node heartbeat/watchdog task parks silently — no ticks, no errors, 0% CPU (distinct from #171) #173 (heartbeat task parking on the nous fleet) is a separate defect, upstream of all of this — a parked tick means the detector never runs at all. Being investigated next.

[orveth]

Probe coverage — resolved. Class (A) evidence-backed; B1/B2 documented as residual

Amending the probe section of the delta above. I raised a coverage question against my own change before the gate saw it; this closes it. No code change resulted — the resolution is that the probe covers the only signature the field has ever actually produced.

The question

The probe has to do two things at once:

1. Suppress the own-echo false positive that made every node cry wolf.
2. Preserve firing on a real receive-side death — "publish flowing, round-trip absent".

(1) is satisfied by construction: the EOSE probe has no dependency on observing our own events, so that failure mode cannot recur.

(2) I could not claim in general, because the probe mints a fresh limit(0) REQ each tick. That splits the real signature:

• (A) the relay stops serving this session's REQs at all — fresh REQ gets no EOSE → fires correctly.
• (B1) the SDK deleted a long-lived subscription from its own map, but the relay still serves new REQs — the fresh probe REQ is answered, so the probe reports alive and does not fire.
• (B2) the relay forgot a subscription while the client still lists it — both bookkeeping views agree and are both wrong; no client-side check can see it.

Uncomfortably, in (B1) the old own-echo probe — had it been satisfiable — was strictly more sensitive than this one, because it watched a real long-lived subscription instead of creating a new one.

The classification, and why it lands on (A)

The two candidate real stalls (old-binary codex seat, 7/24-25) present as (B)-shaped: offers and awards arrived and trades completed during both windows while the detector reported no round-trip. But the only demonstrably absent quantity in either window is the seat's own kind-30340 echo, gone ~22 min, then back.

B1 is refuted by one fact: on the old binary, a terminally-removed subscription can never come back.

• A terminal CLOSED (error: / restricted:) maps to HandleClosedMsg::Remove, which deletes the subscription from the map (relay/inner.rs:1021, :1028, :1043).
• should_resubscribe returns false for a subscription that is not in the map — the None arm (relay/inner.rs:373-379). The SDK's post_connection / post-auth resubscribe() therefore cannot restore it.
• The manual unsubscribe_all + resubscribe path, the only thing that could re-add it, never completed on old: it carries the identical receiver-ordering bug (seller_daemon/mod.rs:3294 receiver, :3298 disconnect) and its success line appears zero times in old-fleet logs.

So terminal removal on old is permanent for the life of the process. The echo returned. The subscription was never removed.

What does produce a bounded, self-healing echo gap: kind 30340 is addressable/replaceable, so the relay keeps only the newest event per (kind, pubkey, d) and by NIP-01 ignores an incoming one older than what it holds — ignored meaning not stored and not broadcast. Anything that stamps that address into the future (an NTP correction, or a second process publishing under the same key and d) silently suppresses every later heartbeat from that seat while leaving offers, awards and payment wraps untouched, and it heals the moment wall-clock passes the stored stamp. A fixed-duration outage that resolves with no reconnect and no intervention is a clock offset's signature; a dead subscription has no reason to last ~22 minutes and then fix itself.

These episodes are therefore transient own-echo artifacts, not subscription deaths. The dataset contains zero confirmed genuine subscription stalls — every stall signal we have is either this artifact or the unsatisfiable-probe false positive described above.

The design error, stated for whoever writes the next probe

The old probe was wrong twice over: accidentally satisfiable (it only worked because publishing went through a throwaway client with a separate database — see #174), and inherently flaky even when satisfiable, because an addressable/replaceable kind is exactly the class of event a relay may legitimately decline to re-broadcast. Round-trip liveness must not be built on a replaceable event. The EOSE probe depends on neither property: it needs no cooperating publisher, no stored event, and no echo.

Coverage, as shipped

failure mode	detector
(A) relay stops serving this session's REQs	EOSE probe — the only class with field evidence
relay announces a subscription drop	CLOSED arm (this PR) — loud, names the leg, degrades offers to targeted-only, repairs via the paced path
(B1) SDK silently deleted a live subscription	not covered → #172's per-subscription reconciler is its designed closure (registered ids vs Relay::subscriptions()), analysis handed over there
(B2) relay forgot a subscription the client still lists	not closable client-side — would require observing our own events, which #174 documents the SDK denies

Stated plainly rather than papered over: this probe is not a universal subscription-liveness oracle. It asserts one property — the relay is serving my REQs on this authenticated session — and asserts it soundly, in both directions, against a fixture that enforces NIP-42 before serving a REQ. The residual classes are named, attributed, and tracked.

[orveth]

Delta 3 @408461c6 — the node can no longer park silently (#173)

A seller node could stop dead — heartbeat tick gone, relay-stall watchdog gone with it, every relay notification gone — at 0% CPU, with nothing logged anywhere. Same class as the two defects above, one layer deeper: the previous deltas fixed a detector that couldn't fire and a recovery that couldn't succeed; this one fixes the loop that runs them not running at all.

How it was found, and what the evidence excluded

A symbolicated frame from a parked seat (stable across two captures 60s apart) put the main thread at the runtime's own park point:

sell.rs block_on → current_thread Context::park → time::Driver → io::Driver::turn → kevent

No libgit2, reqwest or git_transport frame on any thread. That retired the first hypothesis — mine — that blocking relay-git HTTP was parking the runtime.

What replaced it came from one constraint: a permanent park proves no timer was pending, and any bounded await arms one. Run that filter over the candidates and almost everything falls:

• the publish await — excluded. nostr-sdk caps it internally: Relay::send_event → _send_event → wait_for_ok(.., WAIT_FOR_OK_TIMEOUT), and WAIT_FOR_OK_TIMEOUT = 10s (nostr-relay-pool/src/relay/constants.rs:10). A half-open socket makes it error in ten seconds, not hang. This was the originally-suspected cause; the timer argument rules it out.
• sitting at the select! — excluded. interval.tick() arms a timer, so the park would have a deadline and the tick would fire.
• loop terminated — excluded. sell.rs is block_on(runner.run()); a return exits the process, and the process didn't exit.

What survives is the timer-less awaits, and in this node those are the signer actor's oneshot round-trips: six sites, none bounded, every one reachable from inside a select! branch body. publish_heartbeat awaits one before it ever reaches the publish — so it sits in the tick path itself.

1. Bounded signer round-trips

All six now route through one round_trip helper, with both legs bounded — send parks forever on a full queue the actor isn't draining, rx.await parks forever if the actor is alive but never answers, and neither arms a timer. The error names the call and the leg (signer round-trip 'sign' did not complete: actor never answered) instead of a bare "actor gone".

The shipping argument, stated plainly: the fix is the diagnostic. A bound cannot make a stuck actor answer. What it does is convert an invisible permanent park into a named, logged, recoverable failure at the exact call site. That is why this ships without yet knowing which call stuck in the field — we don't need to know, to make parking silently impossible. If it recurs, the log will say where.

Tooth — a_stalled_signer_round_trip_is_bounded_and_leaves_the_caller_usable. The stalled actor receives each command and holds every reply sender forever, which is the only shape that hangs; dropping one would surface as a recv error, not a park. Two successive calls, because the property the run loop actually needs is that the caller is left usable, not merely that one call returned. Time is paused so the 30s bound elapses instantly, and an outer timeout is what makes a revert fail cleanly instead of hanging the suite.

Bite: restore the bare rx.await → rc=101 in 0.00s with the diagnostic panic, no hang.

2. Blocking git work off the runtime — long-standing, exposed here, not introduced here

init_empty_delivery_workdir, snapshot_delivery_at and push_branch_with_header are synchronous libgit2 — the push additionally driving blocking HTTP through the smart subtransport registered in git_transport — and execute_job called all three inline from async. They now run via spawn_blocking (*_off_runtime forms).

An async HTTP client would not have fixed this, and it's worth recording why: libgit2's push/fetch are synchronous C calls that own the thread for their duration regardless of what the transport does underneath, and bridging async inside the subtransport callback would reintroduce a nested-runtime block_on — the exact hazard whose guard is compiled out in release builds.

This defect predates the cutover. origin/main has the identical shape, including a current-thread runtime at seller_daemon/mod.rs:2969 (run_forever_blocking) and a byte-identical git_transport with no spawn_blocking anywhere. The cutover added call sites, not the structure. Anyone hunting a cutover diff for this will not find one.

It also explains the host differential cleanly: on a fast network a push returns in milliseconds, so the blocking window is invisible — turtle never parked. On a host where pushes hang mid-flight, a current-thread runtime turns "slow" into total: timers, relay I/O and the watchdog stop together. Same binary, same bug; only the network decides whether you ever see it.

3. Multi-thread runtime

mobee sell now runs on new_multi_thread().worker_threads(2). The real fixes are (1) and (2); a worker pool is the complement that stops one stuck task from taking the timer with it. On a current-thread runtime a blocking call doesn't merely delay other work — it stops time. No Send fallout.

4. A build you can actually profile

Adds [profile.release-debug] (inherits release, debug = true, strip = false). Stock release is stripped and thin-LTO'd, so a spindump of a stuck node yields raw addresses and nothing else — diagnosing this park required rebuilding the binary before any capture could be read at all. Same optimisation level as release so timing stays representative:

cargo build -p mobee --profile release-debug --features acp,wallet

Noticed, not touched

buyer/signer.rs:55 carries the identical unbounded rx.await.map_err(|_| SignerActorGone) pattern — the buyer daemon has the same timer-less-park defect in its own signer actor. Out of this PR's lane, flagged for the buyer's owner; it's long-lived too, and it's the same class as the #174 note that PR is already inheriting.

Gates — frozen tip 408461c6, pasted rcs

• money suite (-p mobee-core --no-default-features --features gateway,git-delivery,wallet --lib): 499 passed, 0 failed — rc=0
• default (cargo test -p mobee-core): 162 passed, 0 failed — rc=0
• acp full combo, all targets: rc=0
• release binary (cargo build -p mobee --release --features acp,wallet): rc=0
• fmt/clippy: unchanged pre-existing repo-wide drift, CI enforces neither. Zero warnings from the touched files.

Separate CARGO_TARGET_DIR throughout; never the workspace's live target.

[orveth]

Acceptance-block closer — live receipt chain on the frozen final tip 408461c6: the mid-soak smoke trade completed end-to-end on the fix binary and the buyer-side co-signed receipt is on the relay.

• offer 7e9012a3 → claim 776c8057 → award 4558aead → result eafc4ba1 → receipt (kind-3400) c2d38da23a80363d4ddf804e28be9012b9d32be6f3d4a148d93062244773b5d8
• delivery commit 3d155f3e, branch mobee/7e9012a3, content verified exact, 5 sats, pays-exactly-once (award-replay idempotency held ×2)

With this, every artifact in the acceptance block exists on the frozen tip: three deltas code-gated (idempotency guard, satisfiable probe + proven re-auth on an auth-enforcing fixture, park structurally removed), plus dual field validation on the host that reproduced both original bugs.

[orveth]

Delta 4 @0fece647 — restore the periodic payment-wrap backfill (money-path regression in this PR)

Found by asking a narrow observability question — does a healthy idle seat emit any periodic log line? The answer was no, and chasing why turned up something worse than a missing log line: this PR had dropped a money-path safety net.

The regression

grep -rn backfill crates/mobee-core/src/seller_node/ returned only offer-backfill hits. There was no payment-wrap backfill in the node at all — not periodic, not at boot. origin/main has one, on a 300s tick (WRAP_BACKFILL_INTERVAL_SECS, seller_daemon/mod.rs:83) with cursor, skew margin and a test seam. It was in the cutover plan; it never landed, and the 98-test accounting didn't surface it.

Why main carried it, from its own comment (mod.rs:74-81) — this is field-observed, not hygiene:

a fresh subscription delivers a wrap within ~1 min, but a subscription ~10+ min old was seen to go deaf and never deliver again — a payment then sat unredeemed until the daemon was manually restarted (which re-ran the boot backfill).

So on this branch a delivered job's payment could sit uncollected indefinitely, with no restart-free recovery. The seat looks healthy throughout: it claims, delivers, heartbeats, and answers every REQ.

This is a failure the liveness probe added in delta 1 cannot see — the session still serves our REQs, so the relay is alive by every measure the watchdog has. Worth stating plainly because it is the same shape as everything else in this PR: a check that looks like it covers the case, and doesn't.

The fix

A 300s tick that resolves a cursor, re-fetches stored wraps, and routes every one through the normal on_gift_wrap path — so all the pay-once guards apply unchanged (a re-seen wrap hits the receipt dedup; an already-spent token fails closed), then drains. Re-scanning a wide window is safe by construction, which is what makes this recovery cheap enough to run on a timer.

The cursor is the subtle part. It is the last collected receipt, clamped to the oldest delivered-but-unpaid job minus a skew margin. Last-receipt alone is wrong: a receipt for a NEWER job advances the cursor past an OLDER unsettled job and skips its still-uncollected wrap forever — the exact payment-stranding this exists to prevent. A store read error aborts the cycle rather than falling back to since = 0, which would turn a transient failure into a full-history rescan. Two new store readers back it: last_receipt_unix, oldest_unsettled_delivery_unix.

The operator tick, and why it's marked load-bearing

Answering the original question for the record: before this delta, a healthy idle node emitted nothing after boot. Every run-loop line is failure-conditioned — drain is Ok(_) => {} when idle, all four of publish_heartbeat's lines are error paths, the probe is silent on success. External supervision was left with pid presence (which a parked process satisfies — see delta 3) and absence-of-errors, which is not evidence of health.

The backfill's "fetching" line is emitted unconditionally, before the fetch, and is commented as load-bearing for external supervision so a future cleanup doesn't quiet it as noise. It's a meaningful tick rather than a heartbeat-to-stderr: it says "I just re-asked the relay for payments", so it's a response, not a claim about ourselves. Quieting success lines is how this entire class of bug went unnoticed in the first place.

Three layers, deliberately not merged

Stated explicitly so a later cleanup doesn't "simplify" one away — none of these subsumes another:

layer	asserts	catches
EOSE probe (delta 1)	the session answers our REQs	relay stops serving this session
wrap backfill (this delta)	the money leg recovers value	a silently-deaf 1059 subscription, without a restart — field-observed
subscription reconciler (#172)	the subscription map is intact	a removed or closed-but-present subscription, and names which leg

The backfill is a workaround, not a diagnosis: it restores the payment without ever telling an operator a subscription died, and it only covers kind-1059 — a dropped offer or award subscription still has no repair. That's #172's job, and I've noted there that it should prioritise the work legs now that the money leg has a net.

Correction posted to #172

I had written that the reconciler's footing was "real but unobserved". Wrong — main's comment above records the class happening, with a stranded payment. Footing is stronger than I stated; corrected at issuecomment-5086679778.

Gates — frozen tip 0fece647, pasted rcs

• money suite (-p mobee-core --no-default-features --features gateway,git-delivery,wallet --lib): 501 passed, 0 failed — rc=0
• default (cargo test -p mobee-core): rc=0
• acp full combo, all targets: rc=0
• release binary (cargo build -p mobee --release --features acp,wallet): rc=0
• fmt/clippy: unchanged pre-existing repo-wide drift. Zero warnings from the touched files.

Teeth + bite: wrap_backfill_cursor_clamps_to_the_oldest_unsettled_delivery_and_fails_closed (pure: clamp behind an unsettled delivery, 0 legitimate on first boot, read error aborts) and unsettled_delivery_pins_the_cursor_until_its_receipt_lands (over real rows). Bite: drop the .min(oldest - margin) clamp → rc=101, cursor jumps to 10000 instead of 2400, i.e. straight past the older job's payment window. Tree restored and verified identical after.

Per the ruling, the acceptance smoke re-runs on this tip — the earlier evidence stands for what it tested, and the final tip gets its own receipt.

[orveth]

Four-delta acceptance-block closer — live receipt chain on the frozen final tip 0fece647:

• offer 38a538f8 → claim 7827a24d → award 0ff4918c → result f1c58b68 → receipt (kind-3400) 9eb2b0e4fb1c63a3e538f9eb9da9efbdd2943c20d3561174b0739ad5fffa1662
• delivery commit ba388d84, exact-content pass, 5 sats, pays-exactly-once, ~1 min delivery

This tip carries all four deltas: accept-replay execute-idempotency, the satisfiable liveness probe + proven reconnect re-auth, the structural park fix (off-runtime git + bounded signer + multi-thread runtime), and the wrap-backfill port with the cursor clamp pinning the stranded-payment class. Field-validated twice (dual validation on the reproducing host + this end-to-end trade).

[orveth]

Delta 4b @28011f13 — pin the backfill's unconditionality, not just its cursor

A second-order gap in delta 4, raised from the field and correct: delta 4's teeth guard what the backfill's log line says (cursor clamping, fail-closed on a read error). Nothing guarded that the fetch happens at all.

That distinction carries weight because of what the line is for. It is the only periodic output a healthy idle seat produces, so it's what external supervision watches — and the obvious future optimisation, skip the fetch when the store has nothing outstanding, would silence exactly the healthy idle seats an operator least suspects. Supervision goes straight back to absence-reasoning, which is where this whole PR started.

Delta 4 marked the line load-bearing in a comment. A comment does not survive a refactor; a test does.

The tooth

wrap_backfill_fetches_even_with_nothing_pending asserts at the wire, not in the log: a QueryPolicy on an in-process LocalRelay counts kind-1059 REQs, so a skip-when-empty guard cannot pass by keeping the eprintln! and dropping the fetch. (Precedent for the counting policy is already in-tree — CountMetadataQueries, job_lifecycle.rs:3098.)

It drives the real SellerNodeRunner::boot path against a pristine home — no deliveries, no receipts, nothing outstanding, explicitly asserted before the call — so what's covered is the deployable shape rather than a hand-assembled runner. Feasible because SellerNode::open is local-only: no network, and the wallet open touches local sqlite.

One note for whoever runs it: the test takes ~20s. A plain LocalRelay never issues a NIP-42 challenge, so boot waits out CONNECT_WAIT and logs its NoChallenge warning before proceeding. Expected, not a hang.

Bite: add if self.node.store().oldest_unsettled_delivery_unix().ok().flatten().is_none() { return; } at the top of run_wrap_backfill → rc=101, with the assertion naming why. Tree restored and verified identical after.

Gates — frozen tip 28011f13, pasted rcs

• money suite (-p mobee-core --no-default-features --features gateway,git-delivery,wallet --lib): 502 passed, 0 failed — rc=0
• default (cargo test -p mobee-core): rc=0
• acp full combo, all targets: rc=0
• release binary (cargo build -p mobee --release --features acp,wallet): rc=0
• fmt/clippy: unchanged pre-existing repo-wide drift. Zero warnings from the touched file.

This is the tip the acceptance smoke should run against.

[orveth]

Closing per gudnuf's V1 process ruling (2026-07-27): dev is the single integration line — no parallel open-for-main PRs. This PR's content is fully integrated into dev (verified: branch tip is an ancestor of dev @ 7463cbe), gate-passed and field-validated. Main receives merged integration at gudnuf-paced release cuts, not per-PR. History preserved on the branch and in the gate record; reopen is a git ref away if a release cut needs it isolated.