Skip to content

fix(seller): re-acquire a slot permit for jobs resumed after restart - #304

Merged
orveth merged 2 commits into
devfrom
seller/251-restart-slot-permit
Jul 30, 2026
Merged

fix(seller): re-acquire a slot permit for jobs resumed after restart#304
orveth merged 2 commits into
devfrom
seller/251-restart-slot-permit

Conversation

@orveth

@orveth orveth commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Closes #251 (#223 P4).

The defect

The restart path re-drove every non-terminal job with no permit:

runner.execute_job(&job_id, None).await;   // the `None` is the permit

Reservations cannot survive a process by constructionParkedSlot.reserved_at is a monotonic Instant and the parked map is in-memory — so every permit is free when resumed jobs start and take_for_execution has nothing to hand them. A node returning with K non-terminal jobs ran up to slots + K concurrent executions against a ceiling of slots, and could claim slots further offers on top because the gate reported full capacity. Invisible at slots = 1; the excess scales with K exactly as we raise slot counts.

The fix

Each resumed job waits for a real permit before executing, via SlotGate::acquire_for_resume. The wait happens inside the spawned task, never on the loop: a fan-out that awaited capacity at boot would deafen the node that just restarted (#223).

Decision 1 — what happens when K > slots

Queue the excess. Tokio's semaphore hands permits to waiters in order, so resumed jobs run in waves as earlier ones finish, and all K eventually run.

Rejected refuse to resume beyond capacity: it quietly abandons awarded work, and under award-is-payment those are sats the buyer has already committed. Rejected resume serially: it throws away the capacity an operator configured.

Consequence worth stating: while the backlog drains the node stops claiming new offers (try_reserve finds no free permit ⇒ SlotsBusy). That is the intended back-pressure and it is precisely why the bound is slots rather than slots + K — a restarted node with a backlog should be invisible to the market until it catches up.

Not silent: the resume log carries the count with its denominator (N job(s) to re-drive, bounded to M execution slot(s)), because an operator seeing only a count cannot tell whether waves are happening.

Decision 2 — the lapse clock for a re-acquired permit

There isn't one, and that is the answer rather than a gap in it.

The permit is never parked and no reserved_at is seeded. reserved_at bounds a claim waiting to be awarded; a resumed job is already awarded and executing, so seeding a fresh Instant::now() would start a timer over a state the sweep was never meant to measure — the "ceiling without a floor" the issue warns about. sweep_lapsed iterates only the parked map, so it cannot reclaim a resumed job's permit. Release is the existing RAII pairing: the execution future returning, unwind included.

Pinned by its own test (a_resumed_permit_is_not_parked_so_the_lapse_sweep_cannot_reclaim_it) with lapse_after = 0ms, so the sweep would fire on anything parked.

Red leg

Pass None instead of acquiring — the pre-fix path exactly — and a_restart_resumes_more_jobs_than_slots_without_exceeding_capacity fails:

assertion `left == right` failed: concurrent resumed executions must never exceed the configured slots
  left: 4
 right: 2

Four resumed jobs, two slots, all four concurrent. That is #251. 13 other gate tests stayed green, and the lapse-clock test also stayed green under this revert (it calls acquire_for_resume directly rather than the fan-out) — so the two tests are independent, not one property asserted twice.

What the test covers, and the one thing it does not

The fan-out is generic over its execution step so the part carrying the bound is reachable without a relay, an agent, or a store. The stub is sound rather than a shortcut: the bound is a property of the fan-out, not of what a job does, and stubbing is what makes peak concurrency observable at all.

The gap, stated rather than discovered in review: the test does not prove the production loop passes execute_job as its step. That is one call site, checked by the compiler and by reading. If you want that leg too it needs a fixture that instruments a real execute_job, and I'd rather scope it separately than claim it here.

The overlap is real, not incidental — the stub yields across an await, so a bound that held only because nothing ever ran concurrently would not pass. Drain is a bounded poll, not a fixed sleep, so a slow box cannot flake it.

Counts

cargo test -p mobee-core --features wallet --locked: 661 → 663 = +2, 0 failed, 3 ignored.

cargo test -p mobee-core --locked (CI job 1) stays at 196 — unchanged, and correctly so: seller_node is #[cfg(feature = "wallet")] with default = [], so both new tests live where the code lives and are not in the default denominator. The wallet build is the only one that exercises them.

🤖 Generated with Claude Code

A restart re-drove every non-terminal job through execution with no slot permit
(`execute_job(&job_id, None)`), while the gate reported full capacity. Reservations cannot
survive a process by construction — `ParkedSlot.reserved_at` is a monotonic `Instant` and the
parked map is in-memory — so a node that came back up holding K such jobs ran up to
`slots + K` concurrent executions against a configured ceiling of `slots`, and could still
claim `slots` further offers on top. Invisible at `slots = 1`, and the excess scales with K
exactly as we raise slot counts.

Each resumed job now waits for a real permit before executing. `SlotGate::acquire_for_resume`
supplies it; the wait happens INSIDE the spawned task, so boot is never blocked on capacity —
a fan-out that awaited on the loop would deafen the node that just restarted (#223).

K > slots QUEUES rather than dropping. Tokio's semaphore hands permits to waiters in order, so
the excess runs in waves as earlier jobs finish. Refusing to resume beyond capacity would
quietly abandon awarded work, and under award-is-payment those are sats the buyer has already
committed. While the backlog drains the node stops claiming new offers (no free permit ⇒
SlotsBusy) — the intended back-pressure, and the reason the bound is `slots` and not `slots + K`.

There is no lapse clock for a re-acquired permit, which is the answer to the second open
question rather than a gap in it. The permit is never parked and no `reserved_at` is seeded:
`reserved_at` bounds a CLAIM waiting to be awarded, and a resumed job is already awarded and
executing, so seeding a fresh `Instant` would start a timer over a state the sweep was never
meant to measure. `sweep_lapsed` iterates only the parked map and therefore cannot reclaim a
resumed job's permit; it is released exactly like every other executing slot, by the execution
future returning, unwind included.

The resume fan-out is generic over its execution step so the part carrying the bound is
reachable without a relay, an agent, or a store. Red-proved: pass `None` instead of acquiring
and peak concurrency reaches 4 against 2 slots — `left: 4, right: 2`.

Closes #251

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mobee Ready Ready Preview Jul 30, 2026 2:33pm

Request Review

The #251 back-pressure claim — a restarted node stops claiming while its resume backlog
drains — holds only if tokio's semaphore is fair, since the claim path uses `try_acquire`
while the resume path awaits `acquire`. If a barging `try_acquire` could take a permit
released to a queued waiter, resumed jobs could be starved indefinitely by a busy market
and the bound would weaken to "`slots` per wave".

Measured on the locked tokio: a permit released while a waiter is queued goes to the
waiter, and a `try_reserve` at that exact instant is refused.

The discriminating instant is the release itself, with the waiter not yet polled.
Asserting Full while the permit is still held would prove nothing — that is just "no
permits available", the state next to the one that matters.

Refs #251

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant