Skip to content

buyer: refuse to award twice, decided from local state first (#268) - #289

Merged
orveth merged 1 commit into
devfrom
buyer/award-presence-chokepoint
Jul 30, 2026
Merged

buyer: refuse to award twice, decided from local state first (#268)#289
orveth merged 1 commit into
devfrom
buyer/award-presence-chokepoint

Conversation

@orveth

@orveth orveth commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Closes the detection half of #268. Follow-up for the repair half: #286.

The finding: the code documented a safety property it did not have

Not a missing check. plan_rearm's doc comment made a claim, and the claim was false in two independent ways:

A Reserved row with NO relay award is the crash window between reserve and publish — the award never went out, so it must be re-attempted (republished); award_with_reservation's reserve is idempotent for the same amount, so the re-attempt reserves once and publishes.

1. A Reserved row with no relay award is not only the crash window. It is equally the window where publish succeeded and record_award then failed — a state award_with_reservation creates and logs itself, twenty lines away in the same file:

buyer: award for {job_id} published ({award_event_id}) but recording it failed ({error});
this job will NOT be auto-settled on delivery — collect it manually

Two comments in one file, mutually inconsistent, both written as authoritative. The two states want opposite actions: republish, versus never republish.

2. Reserve-idempotency does not protect what was at risk. reserve short-circuits a same-amount replay with Ok(Reserved::Idempotent) — so the budget is not double-committed. It does nothing to stop publish() from running immediately afterwards and putting a second kind-3405 on the relay. And award_with_reservation discarded that value, so the one signal saying "you have been here before" was read and dropped.

⇒ The mitigation named in the comment protects the budget. The exposure was the protocol.

The signal could not carry the weight either

award_on_relay == false conflated three distinct things:

false because how
there is genuinely no award the honest case
the relay errored has_award_async(...).unwrap_or(false) at the call site
the read timed out fetch_events resolves Ok(empty) on timeout, inside the probe

The third is the one that matters: Ok(false) was already a lie before unwrap_or(false) ever ran. Fixing only the unwrap_or would have looked like a fix and left the conflation intact.

The fix: local-first, and the relay only where local truth cannot decide

The local awards row is read first because it is the only signal here that cannot lie by omission — a job_id-keyed row written at this chokepoint. A local point-query either finds the award or proves it was never recorded. There is no timeout that turns "present" into "absent".

reservations.state alone cannot make the call: Reserved is the state of a healthy in-flight award and of both failure windows. The discriminator is the JOIN against awards — the same shape awarded_unsettled_job_ids already relies on.

local row present                        -> AlreadyRecorded -> return it, publish nothing
no row, and no/released/spent reservation-> Publish
no row, reservation is `Reserved`        -> ask the relay:
      an award id  -> REFUSE  (PublishedButUnrecorded)
      None or Err  -> REFUSE  (PresenceUnverified)

Five of the six (row × reservation) cells never touch the network. Spent and Released deliberately fall through to reserve, which already refuses AlreadySpent and re-reserves a Released row — one authority for those, not a second opinion here.

⚠ One deliberate behaviour change, stated as a trade

A process that died between reserve and publish used to silently republish. It now refuses and stops for an operator.

That is the point, not a regression: None from the probe does not mean "no award exists" — absence and unreachability are the same value out of fetch_events — so publishing on it is the duplicate. Recovering costs one command; a duplicate award of real money costs the money. Both refusals name the operator action, and the tests assert the message does:

  • PublishedButUnrecorded → names the award id it found, and says collect manually.
  • PresenceUnverified → names both remedies, because the operator cannot know which state they are in without looking: collect if an award exists, release the reservation if none does.

On the auto-award path the two refusals are routed differently, following this file's existing convention rather than lumping them:

  • PublishedButUnrecorded is terminal (no retry makes the row appear) ⇒ park, which surfaces the refusal with its remedy in status via parked_awards.
  • PresenceUnverified / Presence is unknown, possibly a transient blip ⇒ return Err, leaving the intent pending to re-arm on next start. Safe by construction: the chokepoint refuses again on every unverified pass, so retrying can never publish a duplicate — only eventually succeed once the answer is knowable.

Return type

AwardOutcome::Published(AwardClaimOutcome) | AlreadyAwarded(AwardRecord).

AwardRecord does not persist quoted_mints. Collapsing both into AwardClaimOutcome would mean returning an empty mint list on the already-awarded path — and an empty quoted_mints already means "the claim carried no parseable creq". The type keeps "we don't store this" from being reported as "there were none" on a money path.

has_award_asyncaward_event_id_async, returning Option<String> so a refusal can name the award it found. One relay read, one authority; the bool call site derives its answer.

Scope: detection only

Repairing the missing row is #286, deliberately. Refusal is the safety property — a silent duplicate becomes a loud, actionable stop — and the upcoming #t protocol-tag rename needs exactly and only the guarantee that a false absence cannot publish (after the flip, a probe filtering the new tag is blind to awards published under the old one). Self-healing adds nothing to that guarantee, and repair would write relay-parsed data into the money ledger, which deserves its own reviewed change.

This is also why the rename cannot ship a probe-only presence check: the buyer's awards table is tag-agnostic, so local-first survives the flag day.

Verification

test result: ok. 645 passed; 0 failed; 3 ignored; 0 measured; 0 filtered out

Against the 641 passed / 0 failed / 2 ignored that #287 reported on this same base (eea0b63): +4 runnable, +1 ignored.

New tests:

  1. award_precheck_asks_the_relay_only_when_local_state_is_ambiguous — the pure decision table, all six (row × reservation) cells pinned individually. Exactly one may reach the relay.
  2. a_recorded_award_publishes_nothing_and_reserves_nothing — returns the recorded award (original awarded_at_unix, not this call's), and the reserved total does not move.
  3. an_unrecorded_public_award_refuses_instead_of_publishing_a_duplicate — constructs the real record_award-failed window, asserts the refusal names the award and the action, and asserts the row is still absent (this gate detects; buyer: repair the missing awards row when an award is on the relay but not in the local ledger (follow-up to the §1 presence gate) #286 repairs).
  4. an_unverifiable_presence_refuses_rather_than_publishing — both uninformative answers (Ok(None), Err), asserting both remedies appear.

Red-on-revert: the publish closure in tests that must not publish is unreachable!(). Delete the precheck and the concurrency re-award leg panics, rather than continuing to pass on the reserved total alone — the reserved total was never the thing at risk.

#[ignore]d, requires a live relay and a real home:

  1. red_prove_has_award_async_returns_true_for_a_known_award — ships as the permanent positive control for the probe. It runs a same-construction control first (fetch the known award by id) so an empty probe result cannot be misread as absence, then asserts the probe returns that award id, not merely some award.

Result of running it against the live relay tonight: CONTROL fetch-award-by-id -> 1 event, PROBE -> Ok(true). The probe genuinely works — so this PR is a three-state-logic fix, not a broken-probe fix, and "never award twice" was not decorative.

⚠ Caveat on that run, worth stating because it nearly produced a false pass: the first attempt printed test result: ok. 0 passed; 185 filtered out with exit 0 — the test had never compiled, because --features wallet was omitted. The only thing separating "ran" from "never built" was the count moving. Any local run cited on this crate needs its count.

Rebase note

Rebased onto #287 (eea0b63). Reconciled at the re-award line, taking #287's winners[0] — this PR's unreachable! publish closure only means something if a real winner is named, so the two fixes are orthogonal and both required. The collision was visible only from the file list, not from either PR alone.

🤖 Generated with Claude Code

The reserve-then-award chokepoint now establishes whether a job is ALREADY
awarded before it reserves or publishes, reading the local `awards` row first
and consulting the relay only in the one state local truth cannot settle.

The bug was not a missing check so much as a false safety claim. `plan_rearm`
documented that "a `Reserved` row with NO relay award is the crash window
between reserve and publish -- the award never went out", and justified acting
on it with "`award_with_reservation`'s reserve is idempotent for the same
amount". Both halves fail:

  - A `Reserved` row with no relay award is EQUALLY the window where `publish`
    succeeded and `record_award` then failed -- a state `award_with_reservation`
    creates and logs itself, twenty lines away. The two want opposite actions.
  - Reserve-idempotency protects the BUDGET, not the PROTOCOL. `reserve`
    returns `Reserved::Idempotent` and does not stop `publish` from putting a
    second kind-3405 on the relay. The value was discarded at the call site.

And the signal it acted on could not carry the weight: `award_on_relay == false`
conflated three things -- no award, a relay error (`unwrap_or(false)`), and a
read that TIMED OUT, because `fetch_events` resolves `Ok(empty)` on timeout. So
`Ok(false)` was already a lie before `unwrap_or` ever ran.

The fix reads the local `awards` row first, because it is the only signal here
that cannot lie by omission: a `job_id`-keyed row written AT this chokepoint. A
local point-query either finds the award or proves it was never recorded.
`reservations.state` alone cannot make the call -- `Reserved` is the state of a
healthy in-flight award and of both failure windows -- so the discriminator is
the JOIN against `awards`, the shape `awarded_unsettled_job_ids` already uses.

  local row present                      -> AlreadyRecorded -> return it
  no row, no/released/spent reservation  -> Publish
  no row, reservation `Reserved`         -> ask the relay:
        an award id -> REFUSE (PublishedButUnrecorded)
        None or Err -> REFUSE (PresenceUnverified)

Both relay branches refuse, and one is a deliberate behaviour change: a process
that died between `reserve` and `publish` used to silently republish and now
stops for an operator. That is the point rather than a regression -- `None` from
the probe does not mean "no award exists", so publishing on it is the duplicate.
Recovering costs one command; a duplicate award costs the money. Both refusals
name the operator action, and both are asserted to.

Return type becomes `AwardOutcome::Published(AwardClaimOutcome) |
AlreadyAwarded(AwardRecord)`. `AwardRecord` does not persist `quoted_mints`, and
an empty `quoted_mints` already MEANS "the claim carried no parseable creq" --
so collapsing both into one type would report "we do not store this" as "there
were none" on a money path.

The relay leg is injected as a closure like `publish`, so `lifecycle` stays free
of relay I/O and the common paths cost no network. Tests that must decide from
local state pass a probe that panics if awaited, making "no network needed" an
assertion rather than a comment. `has_award_async` becomes
`award_event_id_async`, returning the id so a refusal can name the award it
found -- one relay read, one authority.

Detection only. Repairing the missing row would write relay-parsed data into the
money ledger, which deserves its own reviewed change: #286.

Tests: 645 passed / 0 failed / 3 ignored, against the 641/0/2 #287 reported on
this same base -- +4 runnable, +1 `#[ignore]`d live-relay red-prove. The
concurrency re-award leg now uses an `unreachable!` publish closure, so removing
the gate panics there instead of passing on the reserved total alone.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mobee Ready Ready Preview Jul 30, 2026 1:29am

Request Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant