buyer: an empty relay read is CANNOT-VERIFY, never ABSENT (#253) - #257
Closed
orveth wants to merge 1 commit into
Closed
buyer: an empty relay read is CANNOT-VERIFY, never ABSENT (#253)#257orveth wants to merge 1 commit into
orveth wants to merge 1 commit into
Conversation
A relay fetch that returns nothing means one of two things — the event is not there, or the budget expired first — and `fetch_events` yields an empty set for both. The auto-award driver read that emptiness as absence, parked the intent with `"offer no longer on the relay"`, and returned. A park is terminal, so one slow answer permanently killed a valid trade. It happened. Five parked intents across two buyer homes carry that reason; four of those jobs were subsequently awarded, delivered, and in three cases paid — against the offer the driver called absent. They only completed because their awards were issued manually, and the manual path never reads the parked intent. - The driver now retries on an unreadable offer instead of parking. Retrying is bounded by the offer's own deadline, which `post_job` now returns so the driver knows it without asking the relay; a re-armed intent that has never read its offer falls back to a bounded number of attempts. Giving up parks with a reason that claims only what is known — the offer was never read — rather than asserting absence. - `plan_missing_offer` carries that rule as a pure decision in `lifecycle`, where the money-load-bearing rules are testable without a relay. - Every offer read logs its elapsed time and the unverified-read count, because elapsed time is the only thing that distinguishes a timeout from an absence from outside the process. - `DEFAULT_FETCH_TIMEOUT_SECS` served both the `get_job` family, which is bounded from ABOVE by the MCP tool deadline, and the award/accept path, which needs to be bounded from BELOW by relay latency. One number could not satisfy both, so they are now `INTERACTIVE_FETCH_TIMEOUT_SECS` and `TRADE_FETCH_TIMEOUT_SECS`. A third constant of the same name in `profile` becomes `PROFILE_FETCH_TIMEOUT_SECS`: grepping that name used to find three unrelated budgets with three values. - `RELAY_TIMEOUT` for the background driver moves to 15s, matching the doctor's budget for the same relay. Raising the budgets is mitigation, not the fix: measured WebSocket handshake time to one relay ranges 867ms–7223ms depending on where the buyer sits, so any fixed budget is a coin flip whose odds depend on network position, and an award pays that handshake three times. The retry is what makes losing the flip cost five seconds instead of the trade. Reported-by: rocky (independent operator) — diagnosed the empty-vs-timeout substitution, measured the handshake cost, and located both fetch budgets.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This was referenced Jul 29, 2026
orveth
added a commit
that referenced
this pull request
Jul 30, 2026
…253, #257, #289) §1 detection: routes both selection-award publish paths (finalize_auto_award mod.rs:866 + manual MCP award mod.rs:577) through the single chokepoint award_with_reservation, and replaces the two-state has_award (empty-read collapsed to absent) with a THREE-state probe: present->skip, absent->proceed, UNVERIFIED(timeout/empty/relay-err)->REFUSE (fail-closed, publish closure unreachable). Discriminator is the awards job_id-PK row (a JOIN), never reservations.state alone. Detection only; repair is #286; no deploy/config change. GATED (adversarial verify on head 70e1645, reproduced locally): money-path 645/0/3 by the structural count rule (+4 runnable/+1 ignored vs 641; count moved = real run). No-bypass CONFIRMED for the selection award (award_claim_async has exactly two callers, both in the publish closure; accept_claim_async's 3405 is the separate expected accept, not a bypass). Three-state CONFIRMED (AskRelay arm returns Err on all three sub-cases). ATTACK A fail-closed unconditional (unreachable!()-closure tests pass); ATTACK B safe unconditionally (single reserve()? refuses AlreadySpent before publish, Spent terminal). Two benign follow-ups noted: (1) 'every 3405' is literally over-broad (informational); (2) the PresenceUnverified refusal names 'release the reservation', an operator action with no CLI/MCP command — recovery rides the auto-reconcile loop (tracked follow-up; money-safe).
Contributor
Author
|
Superseded by #289 — the same §1 award-presence chokepoint, rebased onto #287 (the buyer/lifecycle.rs race-test fix that moved the shared call site). #289 merged to dev as b9aedcf: three-state presence gate at award_with_reservation (unverified→REFUSE, fail-closed), no-bypass confirmed for the selection award, verified adversarially with money-path 645/0/3 reproduced. The two-condition gate on #253 is satisfied: (1) §1-at-chokepoint = #289; (2) seller post-Delivered test leg = #282 (merged). Detection only; repair tracked in #286; the fail-closed operator-exit wording tracked in #290. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the defect in #253: the auto-award driver rendered an unreadable relay read as an authoritative negative and parked the trade permanently.
What was wrong
fetch_eventsreturns an empty set both for "no such event" and for "the timeout expired first". The driver treated empty as absence:Twenty-four lines above it, the same function already takes the correct posture on the same uncertainty —
has_award_async(...).unwrap_or(false), with the comment "a relay error is 'unknown' … do not skip". The award check knew a failed read was unknown; the offer check, in the same loop, did not.A park is terminal (
list_pending_awardsselectsWHERE state='pending'; its own test asserts"parked is not pending"), so one slow answer permanently killed a valid trade.Evidence this is not hypothetical
Five parked intents across two buyer homes all carry
"offer no longer on the relay". Four of those jobs were subsequently awarded and delivered, three of them paid — the daemon's own log carriesdelivery watcher settled … paid 100 satfor three. A job cannot be delivered against an offer that is not on the relay.The clearest single case needs no timing argument at all: one job's intent reads
parked, "offer no longer on the relay"whileget_jobfor that same id, from the same daemon and the same reader one minute later, returnsoffer_present: truewith the deadline still in the future.They completed only because their awards were issued manually, and the manual path never reads the parked intent. An unattended buyer on this code loses the trade silently.
What this changes
The driver retries an unreadable offer instead of parking. Bounded by the offer's own deadline, which
post_jobnow returns so the driver knows it without a relay read. A re-armed intent that has never successfully read its offer falls back to a bounded attempt count, and giving up parks with a reason that claims only what is known — the offer could not be read — instead of asserting absence.The rule is a pure decision,
lifecycle::plan_missing_offer, in the module whose contract is that money-load-bearing rules are exhaustively testable without a relay or a mint.Offer reads log elapsed time and the unverified-read count. From outside the process, elapsed time is the only thing that separates a timeout from an absence. Nothing logged either, which is why this had to be reconstructed from park reasons and payment logs.
One constant that served two opposite contracts is now two.
DEFAULT_FETCH_TIMEOUT_SECSwas shared by theget_jobfamily — bounded from above by the MCP tool deadline, per its own comment — and the award/accept path, which needs to be bounded from below by relay latency. No single number satisfies both. They are nowINTERACTIVE_FETCH_TIMEOUT_SECS(5s, unchanged) andTRADE_FETCH_TIMEOUT_SECS(15s). A third constant of the same name inprofile.rs(value 8) becomesPROFILE_FETCH_TIMEOUT_SECS— grepping that one name used to find three unrelated budgets with three values, which is how a fix that raises two of three resurfaces later as "intermittent".RELAY_TIMEOUTfor the background driver moves 5s → 15s, matching the doctor's budget for the same relay.Raising the budgets is mitigation. It is not the fix.
Measured WebSocket handshake time to one relay ranges 867ms–7223ms from a remote buyer, while TCP+TLS to that same relay measured 0.14s from the relay's own host (five samples, <5ms variance). An 8× spread on the same operation means any fixed budget is a coin flip whose odds depend on where the buyer sits — a bigger number rescues nearby buyers and leaves distant ones racing the same handshake. An award pays that handshake three times (
has_award_async,fetch_job_view_async,publish_draft_asynceach build a fresh client), so it is three independent flips and the award needs all three.The retry is the part that makes the timeout value stop being load-bearing: losing a flip now costs five seconds instead of the trade.
The follow-up, with its argument already made
Connection reuse is the real fix, and there is a sharper reason for it than latency:
buyer/relay.rsalready carries a considered 45sPUBLISH_TIMEOUTthat the award path bypasses, becausejob_lifecyclebuilds its own ad-hoc client. The right value exists in the tree and is unreachable from the code that needs it. That is a different defect from "the number is too small", and it is not fixed here — deliberately, so this change stays reviewable and lands while the parks are still happening.Verification
cargo test -p mobee-core --lib --features wallet— 623 passed, 0 failed.left: ParkUnverified, right: Retry— then green again on the exact tip.One note on running it:
buyeris behind#[cfg(feature = "wallet")], so a plaincargo test -p mobee-core --libruns 185 tests and not one of them is a buyer test — it reportsokwhile measuring nothing in this module. The feature flag is required to test any of this.Formatting left as-is:
rustfmt1.94 wants to rewrap 13 lines of the existingplan_rearmtest at HEAD, so the un-wrapped assert style is the repo's convention and the new test matches its neighbours rather than the local toolchain.Reported-by: rocky, an independent operator running their own fleet — diagnosed the empty-vs-timeout substitution, measured the handshake cost and both fetch budgets, and retracted two of their own hypotheses before either was tested.