Skip to content

buyer: an empty relay read is CANNOT-VERIFY, never ABSENT (#253) - #257

Closed
orveth wants to merge 1 commit into
devfrom
fix/253-validation-read-not-absence
Closed

buyer: an empty relay read is CANNOT-VERIFY, never ABSENT (#253)#257
orveth wants to merge 1 commit into
devfrom
fix/253-validation-read-not-absence

Conversation

@orveth

@orveth orveth commented Jul 29, 2026

Copy link
Copy Markdown
Contributor

Fixes the defect in #253: the auto-award driver rendered an unreadable relay read as an authoritative negative and parked the trade permanently.

What was wrong

fetch_events returns an empty set both for "no such event" and for "the timeout expired first". The driver treated empty as absence:

let Some(offer) = view.offer.as_ref() else {
    let _ = context.store.mark_award_parked(job_id, "offer no longer on the relay", now_unix());
    return Ok(());
};

Twenty-four lines above it, the same function already takes the correct posture on the same uncertainty — has_award_async(...).unwrap_or(false), with the comment "a relay error is 'unknown' … do not skip". The award check knew a failed read was unknown; the offer check, in the same loop, did not.

A park is terminal (list_pending_awards selects WHERE state='pending'; its own test asserts "parked is not pending"), so one slow answer permanently killed a valid trade.

Evidence this is not hypothetical

Five parked intents across two buyer homes all carry "offer no longer on the relay". Four of those jobs were subsequently awarded and delivered, three of them paid — the daemon's own log carries delivery watcher settled … paid 100 sat for three. A job cannot be delivered against an offer that is not on the relay.

The clearest single case needs no timing argument at all: one job's intent reads parked, "offer no longer on the relay" while get_job for that same id, from the same daemon and the same reader one minute later, returns offer_present: true with the deadline still in the future.

They completed only because their awards were issued manually, and the manual path never reads the parked intent. An unattended buyer on this code loses the trade silently.

What this changes

The driver retries an unreadable offer instead of parking. Bounded by the offer's own deadline, which post_job now returns so the driver knows it without a relay read. A re-armed intent that has never successfully read its offer falls back to a bounded attempt count, and giving up parks with a reason that claims only what is known — the offer could not be read — instead of asserting absence.

The rule is a pure decision, lifecycle::plan_missing_offer, in the module whose contract is that money-load-bearing rules are exhaustively testable without a relay or a mint.

Offer reads log elapsed time and the unverified-read count. From outside the process, elapsed time is the only thing that separates a timeout from an absence. Nothing logged either, which is why this had to be reconstructed from park reasons and payment logs.

One constant that served two opposite contracts is now two. DEFAULT_FETCH_TIMEOUT_SECS was shared by the get_job family — bounded from above by the MCP tool deadline, per its own comment — and the award/accept path, which needs to be bounded from below by relay latency. No single number satisfies both. They are now INTERACTIVE_FETCH_TIMEOUT_SECS (5s, unchanged) and TRADE_FETCH_TIMEOUT_SECS (15s). A third constant of the same name in profile.rs (value 8) becomes PROFILE_FETCH_TIMEOUT_SECS — grepping that one name used to find three unrelated budgets with three values, which is how a fix that raises two of three resurfaces later as "intermittent".

RELAY_TIMEOUT for the background driver moves 5s → 15s, matching the doctor's budget for the same relay.

Raising the budgets is mitigation. It is not the fix.

Measured WebSocket handshake time to one relay ranges 867ms–7223ms from a remote buyer, while TCP+TLS to that same relay measured 0.14s from the relay's own host (five samples, <5ms variance). An 8× spread on the same operation means any fixed budget is a coin flip whose odds depend on where the buyer sits — a bigger number rescues nearby buyers and leaves distant ones racing the same handshake. An award pays that handshake three times (has_award_async, fetch_job_view_async, publish_draft_async each build a fresh client), so it is three independent flips and the award needs all three.

The retry is the part that makes the timeout value stop being load-bearing: losing a flip now costs five seconds instead of the trade.

The follow-up, with its argument already made

Connection reuse is the real fix, and there is a sharper reason for it than latency: buyer/relay.rs already carries a considered 45s PUBLISH_TIMEOUT that the award path bypasses, because job_lifecycle builds its own ad-hoc client. The right value exists in the tree and is unreachable from the code that needs it. That is a different defect from "the number is too small", and it is not fixed here — deliberately, so this change stays reviewable and lands while the parks are still happening.

Verification

  • cargo test -p mobee-core --lib --features wallet623 passed, 0 failed.
  • Red leg: reintroducing the defect (park on the first empty read) fails the new test on its first assertion — left: ParkUnverified, right: Retry — then green again on the exact tip.
  • The new test enumerates the failure modes rather than asserting the happy path: the first empty read must retry (that is the read that killed four trades); a known deadline must retry regardless of how many reads have failed; only a passed deadline parks on positive knowledge; a zero bound still cannot park a job whose deadline is known and unexpired.

One note on running it: buyer is behind #[cfg(feature = "wallet")], so a plain cargo test -p mobee-core --lib runs 185 tests and not one of them is a buyer test — it reports ok while measuring nothing in this module. The feature flag is required to test any of this.

Formatting left as-is: rustfmt 1.94 wants to rewrap 13 lines of the existing plan_rearm test at HEAD, so the un-wrapped assert style is the repo's convention and the new test matches its neighbours rather than the local toolchain.

Reported-by: rocky, an independent operator running their own fleet — diagnosed the empty-vs-timeout substitution, measured the handshake cost and both fetch budgets, and retracted two of their own hypotheses before either was tested.

A relay fetch that returns nothing means one of two things — the event is not
there, or the budget expired first — and `fetch_events` yields an empty set for
both. The auto-award driver read that emptiness as absence, parked the intent
with `"offer no longer on the relay"`, and returned. A park is terminal, so one
slow answer permanently killed a valid trade.

It happened. Five parked intents across two buyer homes carry that reason; four
of those jobs were subsequently awarded, delivered, and in three cases paid —
against the offer the driver called absent. They only completed because their
awards were issued manually, and the manual path never reads the parked intent.

- The driver now retries on an unreadable offer instead of parking. Retrying is
  bounded by the offer's own deadline, which `post_job` now returns so the
  driver knows it without asking the relay; a re-armed intent that has never
  read its offer falls back to a bounded number of attempts. Giving up parks
  with a reason that claims only what is known — the offer was never read —
  rather than asserting absence.
- `plan_missing_offer` carries that rule as a pure decision in `lifecycle`,
  where the money-load-bearing rules are testable without a relay.
- Every offer read logs its elapsed time and the unverified-read count, because
  elapsed time is the only thing that distinguishes a timeout from an absence
  from outside the process.
- `DEFAULT_FETCH_TIMEOUT_SECS` served both the `get_job` family, which is
  bounded from ABOVE by the MCP tool deadline, and the award/accept path, which
  needs to be bounded from BELOW by relay latency. One number could not satisfy
  both, so they are now `INTERACTIVE_FETCH_TIMEOUT_SECS` and
  `TRADE_FETCH_TIMEOUT_SECS`. A third constant of the same name in `profile`
  becomes `PROFILE_FETCH_TIMEOUT_SECS`: grepping that name used to find three
  unrelated budgets with three values.
- `RELAY_TIMEOUT` for the background driver moves to 15s, matching the doctor's
  budget for the same relay.

Raising the budgets is mitigation, not the fix: measured WebSocket handshake
time to one relay ranges 867ms–7223ms depending on where the buyer sits, so any
fixed budget is a coin flip whose odds depend on network position, and an award
pays that handshake three times. The retry is what makes losing the flip cost
five seconds instead of the trade.

Reported-by: rocky (independent operator) — diagnosed the empty-vs-timeout
substitution, measured the handshake cost, and located both fetch budgets.
@vercel

vercel Bot commented Jul 29, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mobee Ready Ready Preview Jul 29, 2026 9:47pm

Request Review

orveth added a commit that referenced this pull request Jul 30, 2026
…253, #257, #289)

§1 detection: routes both selection-award publish paths (finalize_auto_award mod.rs:866 + manual MCP award mod.rs:577) through the single chokepoint award_with_reservation, and replaces the two-state has_award (empty-read collapsed to absent) with a THREE-state probe: present->skip, absent->proceed, UNVERIFIED(timeout/empty/relay-err)->REFUSE (fail-closed, publish closure unreachable). Discriminator is the awards job_id-PK row (a JOIN), never reservations.state alone. Detection only; repair is #286; no deploy/config change.

GATED (adversarial verify on head 70e1645, reproduced locally): money-path 645/0/3 by the structural count rule (+4 runnable/+1 ignored vs 641; count moved = real run). No-bypass CONFIRMED for the selection award (award_claim_async has exactly two callers, both in the publish closure; accept_claim_async's 3405 is the separate expected accept, not a bypass). Three-state CONFIRMED (AskRelay arm returns Err on all three sub-cases). ATTACK A fail-closed unconditional (unreachable!()-closure tests pass); ATTACK B safe unconditionally (single reserve()? refuses AlreadySpent before publish, Spent terminal). Two benign follow-ups noted: (1) 'every 3405' is literally over-broad (informational); (2) the PresenceUnverified refusal names 'release the reservation', an operator action with no CLI/MCP command — recovery rides the auto-reconcile loop (tracked follow-up; money-safe).
@orveth

orveth commented Jul 30, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #289 — the same §1 award-presence chokepoint, rebased onto #287 (the buyer/lifecycle.rs race-test fix that moved the shared call site). #289 merged to dev as b9aedcf: three-state presence gate at award_with_reservation (unverified→REFUSE, fail-closed), no-bypass confirmed for the selection award, verified adversarially with money-path 645/0/3 reproduced. The two-condition gate on #253 is satisfied: (1) §1-at-chokepoint = #289; (2) seller post-Delivered test leg = #282 (merged). Detection only; repair tracked in #286; the fail-closed operator-exit wording tracked in #290.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants