fix(seller): re-acquire a slot permit for jobs resumed after restart - #304
Merged
Conversation
A restart re-drove every non-terminal job through execution with no slot permit (`execute_job(&job_id, None)`), while the gate reported full capacity. Reservations cannot survive a process by construction — `ParkedSlot.reserved_at` is a monotonic `Instant` and the parked map is in-memory — so a node that came back up holding K such jobs ran up to `slots + K` concurrent executions against a configured ceiling of `slots`, and could still claim `slots` further offers on top. Invisible at `slots = 1`, and the excess scales with K exactly as we raise slot counts. Each resumed job now waits for a real permit before executing. `SlotGate::acquire_for_resume` supplies it; the wait happens INSIDE the spawned task, so boot is never blocked on capacity — a fan-out that awaited on the loop would deafen the node that just restarted (#223). K > slots QUEUES rather than dropping. Tokio's semaphore hands permits to waiters in order, so the excess runs in waves as earlier jobs finish. Refusing to resume beyond capacity would quietly abandon awarded work, and under award-is-payment those are sats the buyer has already committed. While the backlog drains the node stops claiming new offers (no free permit ⇒ SlotsBusy) — the intended back-pressure, and the reason the bound is `slots` and not `slots + K`. There is no lapse clock for a re-acquired permit, which is the answer to the second open question rather than a gap in it. The permit is never parked and no `reserved_at` is seeded: `reserved_at` bounds a CLAIM waiting to be awarded, and a resumed job is already awarded and executing, so seeding a fresh `Instant` would start a timer over a state the sweep was never meant to measure. `sweep_lapsed` iterates only the parked map and therefore cannot reclaim a resumed job's permit; it is released exactly like every other executing slot, by the execution future returning, unwind included. The resume fan-out is generic over its execution step so the part carrying the bound is reachable without a relay, an agent, or a store. Red-proved: pass `None` instead of acquiring and peak concurrency reaches 4 against 2 slots — `left: 4, right: 2`. Closes #251 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
The #251 back-pressure claim — a restarted node stops claiming while its resume backlog drains — holds only if tokio's semaphore is fair, since the claim path uses `try_acquire` while the resume path awaits `acquire`. If a barging `try_acquire` could take a permit released to a queued waiter, resumed jobs could be starved indefinitely by a busy market and the bound would weaken to "`slots` per wave". Measured on the locked tokio: a permit released while a waiter is queued goes to the waiter, and a `try_reserve` at that exact instant is refused. The discriminating instant is the release itself, with the waiter not yet polled. Asserting Full while the permit is still held would prove nothing — that is just "no permits available", the state next to the one that matters. Refs #251 Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #251 (#223 P4).
The defect
The restart path re-drove every non-terminal job with no permit:
Reservations cannot survive a process by construction —
ParkedSlot.reserved_atis a monotonicInstantand the parked map is in-memory — so every permit is free when resumed jobs start andtake_for_executionhas nothing to hand them. A node returning with K non-terminal jobs ran up toslots + Kconcurrent executions against a ceiling ofslots, and could claimslotsfurther offers on top because the gate reported full capacity. Invisible atslots = 1; the excess scales with K exactly as we raise slot counts.The fix
Each resumed job waits for a real permit before executing, via
SlotGate::acquire_for_resume. The wait happens inside the spawned task, never on the loop: a fan-out that awaited capacity at boot would deafen the node that just restarted (#223).Decision 1 — what happens when K >
slotsQueue the excess. Tokio's semaphore hands permits to waiters in order, so resumed jobs run in waves as earlier ones finish, and all K eventually run.
Rejected refuse to resume beyond capacity: it quietly abandons awarded work, and under award-is-payment those are sats the buyer has already committed. Rejected resume serially: it throws away the capacity an operator configured.
Consequence worth stating: while the backlog drains the node stops claiming new offers (
try_reservefinds no free permit ⇒SlotsBusy). That is the intended back-pressure and it is precisely why the bound isslotsrather thanslots + K— a restarted node with a backlog should be invisible to the market until it catches up.Not silent: the resume log carries the count with its denominator (
N job(s) to re-drive, bounded to M execution slot(s)), because an operator seeing only a count cannot tell whether waves are happening.Decision 2 — the lapse clock for a re-acquired permit
There isn't one, and that is the answer rather than a gap in it.
The permit is never parked and no
reserved_atis seeded.reserved_atbounds a claim waiting to be awarded; a resumed job is already awarded and executing, so seeding a freshInstant::now()would start a timer over a state the sweep was never meant to measure — the "ceiling without a floor" the issue warns about.sweep_lapsediterates only the parked map, so it cannot reclaim a resumed job's permit. Release is the existing RAII pairing: the execution future returning, unwind included.Pinned by its own test (
a_resumed_permit_is_not_parked_so_the_lapse_sweep_cannot_reclaim_it) withlapse_after = 0ms, so the sweep would fire on anything parked.Red leg
Pass
Noneinstead of acquiring — the pre-fix path exactly — anda_restart_resumes_more_jobs_than_slots_without_exceeding_capacityfails:Four resumed jobs, two slots, all four concurrent. That is #251. 13 other gate tests stayed green, and the lapse-clock test also stayed green under this revert (it calls
acquire_for_resumedirectly rather than the fan-out) — so the two tests are independent, not one property asserted twice.What the test covers, and the one thing it does not
The fan-out is generic over its execution step so the part carrying the bound is reachable without a relay, an agent, or a store. The stub is sound rather than a shortcut: the bound is a property of the fan-out, not of what a job does, and stubbing is what makes peak concurrency observable at all.
⚠ The gap, stated rather than discovered in review: the test does not prove the production loop passes
execute_jobas its step. That is one call site, checked by the compiler and by reading. If you want that leg too it needs a fixture that instruments a realexecute_job, and I'd rather scope it separately than claim it here.The overlap is real, not incidental — the stub yields across an await, so a bound that held only because nothing ever ran concurrently would not pass. Drain is a bounded poll, not a fixed sleep, so a slow box cannot flake it.
Counts
cargo test -p mobee-core --features wallet --locked: 661 → 663 = +2, 0 failed, 3 ignored.⚠
cargo test -p mobee-core --locked(CI job 1) stays at 196 — unchanged, and correctly so:seller_nodeis#[cfg(feature = "wallet")]withdefault = [], so both new tests live where the code lives and are not in the default denominator. The wallet build is the only one that exercises them.🤖 Generated with Claude Code