Skip to content

feat(seller): stop claiming for a harness that cannot deliver, and restore it only on a passing self-probe - #300

Merged
orveth merged 3 commits into
devfrom
seller/roster-drop-254
Jul 30, 2026
Merged

feat(seller): stop claiming for a harness that cannot deliver, and restore it only on a passing self-probe#300
orveth merged 3 commits into
devfrom
seller/roster-drop-254

Conversation

@orveth

@orveth orveth commented Jul 30, 2026

Copy link
Copy Markdown
Contributor

Closes part of #254. Read the scope note at the bottom first — this narrows the quota-dead hole, it does not close it, and I am not claiming otherwise.

The problem

Boot verifies that a harness launches. Launching is not delivering. A harness whose account is at a spend limit, whose provider was never configured, or whose binary lacks the acp feature boots perfectly and then cannot do any work — and the node kept claiming for it. Under award-is-payment the buyer's sats are committed at award, so every such claim costs a buyer money before the failure is visible to anyone.

What this does

seller_roster::LiveRoster owns the boot registry together with live availability, exposing the surface the node already read (advertised / dispatch / serves). A dropped harness therefore leaves the advertisement and the dispatch table in the same motion.

Wrapping rather than adding a health object alongside the registry is the load-bearing choice. The registry's stated contract is "advertised is exactly what dispatch serves"; a separate object leaves every caller free to read advertised() unfiltered and publish a harness the node would then refuse. This way the invariant holds by construction, not by discipline.

Three states, because CANNOT RUN is not FAILED:

state meaning probe?
serving advertised and dispatchable
Dropped failed unattributably; transient and structural arrive identically, so it does not guess scheduled, 15m → 1h → 4h
Incapable a named missing capability never — no retry adds a build feature

Incapable derives its remedy from the capability rather than hardcoding one. A fixed "rebuild owed" would be a lie for a harness whose barrier is an unset provider, and the disposition is only worth emitting if an operator can act on it. The capability is a typed enum, so the compiler refuses a new variant without a remedy.

Collapsing the last two states is the defect this shape exists to prevent: an acp-less seat can never pass a probe, so a two-state design drops it on its first offer and never restores it — converting a fixable build problem into a permanent roster hole with no signal saying which it was.

The backoff window gates the PROBE, never the restoration. probe_due_at passing restores nothing; only a probe that actually passed does. If the window merely expired, the next real offer would become our diagnostic and a buyer would pay for it.

State is keyed by registry index, not name, so the unlabelled --agent-argv hatch is droppable too. A name-keyed map could not drop it at all — it has no name — leaving exactly one configuration permanently unable to stop claiming.

The self-probe is decided by the ARTIFACT

This is the part worth reviewing hardest. A harness whose account is exhausted ends its turn completed, exits 0, and returns a perfectly non-empty message telling you to upgrade your plan. Exit status, turn state and response length are all green for exactly the harness we most need to catch — measured on cursor, whose ACP path completes in ~2s with a billing notice and no error anywhere in mobee's own job log.

So the probe asks for one artifact carrying a token minted for that probe and requires the token back. Fresh each time, so neither a stale workdir nor a replayed transcript satisfies one. It runs under the same sandbox policy as a real job — probing an unsandboxed path while jobs run sandboxed would verify a path no paid job ever takes. Its failures go through the same attribution classifier a real job's failure uses, so a probe against an acp-less binary marks the harness Incapable and stops being probed rather than re-asking a settled question every few hours.

Probes are spawned off the event loop, never awaited on it — a probe runs a whole agent turn, and awaiting one on the loop would deafen the node to offers, awards and payments (#223). Since the tick that starts them fires every 5s while a probe takes minutes, claim_due_probes marks a harness in-flight in the same locked step that hands it over; otherwise the node would stack a dozen concurrent turns of its own tokens on one harness. Either verdict releases the mark.

The trigger is multi-site, scoped by ATTRIBUTION not severity

Hooking only the agent-run Err misses the site that actually fires on a quota-dead harness — its turn completes, so that arm sees no error at all. But execution failures are not uniformly harness-attributable, and dropping a harness for the wrong one is an outage the node inflicts on itself:

  • agent run errored — harness
  • delivery snapshot refused, empty tree — harness ✅ (returned success having produced nothing)
  • git push failed — remote/network ❌
  • receipt sign refused / signer gone — ours ❌
  • workdir init failed — local fs ❌
  • ExecError::Policy — our own refusal ❌

Verification

Command, verbatim, so the numbers are reproducible against a stated base:

cargo test -p mobee-core --features acp,wallet,gateway --lib -- --list
  • base (dev a1e14de, same tree, same command): 655
  • this branch: 669+14, all of them new tests for this change
  • full run: 666 passed, 0 failed, 3 ignored; default-feature run 196 passed, 0 failed
  • acp_concurrency 2.01s — the CI: acp+money-path job is externally cancelled ~2s into acp_concurrency.rs — baseline on dev+main since ~22:17, blinds every money-path row #271 flake did not fire
  • compiled clean on four feature combos: default, wallet, acp, acp,wallet,gateway (the pre-existing launch_argv dead-code warning without acp is not from this change)
  • seller_roster is ungated, so its tests run in the default cargo test denominator rather than only under --features — deliberate, after being bitten twice by feature-gated tests reporting ok while running none

Red-proved, two legs, both targeted:

  1. Remove the availability filter from the live view → 6 tests fail, including the wiring test a_dropped_harness_stops_the_node_claiming_for_it.
  2. Decide the probe on workdir non-emptiness instead of the token → the_self_probe_is_decided_by_the_token_not_by_a_completed_turn fails, i.e. the billing-notice case passes and a harness that can do no work is restored to the roster.

The classifier and in-flight-guard tests stayed green under both, so these are negative controls rather than a blanket break.

⚠ Scope: narrowed, not closed

A premise I published on #254 and that was adopted from me was wrong, and I corrected it there (comment 5128642935) before writing this code. I had claimed the quota-dead class produces "zero failures". It does not: seller_git.rs:280-293 is a completion gate the node already runs, called with base = None, so the live floor is !tree.is_empty() and a harness that writes nothing already fails at run.rs. Cursor's observed empty-cwd behaviour would have been refused.

I got that wrong because I measured with mobee run, which has no delivery-snapshot step at all, and reported the result as a property of the node's execute path. The probe and the path differ precisely in the step that catches the class.

What survives is narrower and still real: a quota-dead harness that writes ANY file. Commit one file saying upgrade your plan and tree-non-empty passes, push passes, the receipt signs, the watcher pays. So the node's existing artifact predicate is a global non-emptiness check — the known-insufficient granularity — and this PR wires that existing gate into the roster drop rather than adding a new one.

Still owed, deliberately not in this PR: the prompt (seller_exec.rs:174-180) already tells every agent "Make one or more non-empty commits… Anything not committed to git will not be delivered", and that sentence is currently false — the node add_alls the whole workdir and makes its own commit. Enforcing the contract it already publishes is ruled GO, gated on first counting how many current deliveries would break. That count is its own PR. A genuinely per-job mandatory predicate is a market-wide contract change and is not mine to take.

Refs #254 · unblocks #252, which was holding for this seam.

🤖 Generated with Claude Code

orveth and others added 2 commits July 30, 2026 02:08
…annot deliver

A harness that boots fine can still be unable to work — its account is at a spend
limit, its provider is unconfigured, the binary lacks the `acp` feature. Boot can
only verify that a harness LAUNCHES, so a node kept claiming for such a harness and
became a black hole for awards: under award-is-payment the buyer's sats are already
committed by the time the failure surfaces.

`seller_roster::LiveRoster` owns the boot registry together with live availability and
exposes the surface the node already read (`advertised` / `dispatch` / `serves`), so a
dropped harness leaves the advertisement and the dispatch table in the same motion.
Wrapping rather than adding a separate health object is what keeps the registry's
"advertised is exactly what dispatch serves" contract true by construction — alongside
it, any caller reading `advertised()` unfiltered would publish a harness the node would
then refuse.

Three states, because CANNOT RUN is not FAILED. An unattributable failure DROPS the
harness and schedules a self-probe. A named missing capability marks it INCAPABLE and
schedules nothing, because no number of retries adds a build feature or picks a
provider; the remedy is derived from the capability, since a hardcoded "rebuild" is a
lie for a harness whose barrier is configuration. Collapsing the two would drop an
acp-less seat on its first offer and never restore it, silently converting a fixable
build problem into a permanent roster hole.

The backoff window gates the PROBE, never the restoration. `probe_due_at` passing
restores nothing; only a probe that actually passed does. Otherwise the next real offer
would become our diagnostic and a buyer would pay for it.

State is keyed by registry index, not by name, so the unlabelled `--agent-argv` hatch is
droppable too — a name-keyed map could not drop it at all, leaving exactly one
configuration permanently unable to stop claiming.

The trigger is multi-site and scoped by ATTRIBUTION rather than by severity. An agent
run that errored and a delivery snapshot refused for an empty tree both implicate the
harness; the empty-tree site is the one that fires on a quota-dead harness, whose turn
"completes" so the agent-run arm sees no error at all. A failed push, a receipt our own
signer refused, a workdir that would not initialise, and a policy WE declined are all
execution failures that say nothing about the harness — dropping it for those would be
an outage the node inflicts on itself.

Refs #254

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rs its token

A drop with no way back converts a transient failure into a permanent roster hole: one
flaky timeout would take a single-harness seat dark until someone restarted it. The
recovery path is a self-probe, and what the probe is DECIDED BY is the whole point.

A harness whose account is exhausted ends its turn `completed`, exits 0, and returns a
perfectly non-empty message explaining that you should upgrade your plan. Exit status,
turn state, and response length are all green for exactly the harness we most need to
catch — measured on cursor, whose ACP path completes in ~2s with a billing notice and no
error anywhere in the job log. So the probe asks for one artifact carrying a token minted
for that probe, and requires the token back. A harness that cannot work cannot produce
it. The token is fresh each time, so neither a stale workdir nor a replayed transcript
can satisfy one.

The probe runs under the same sandbox policy as a real job, because probing an
unsandboxed path while jobs run sandboxed would verify a path no paid job ever takes.

Its failures are classified by the same attribution function a real job's failure uses,
so a probe against an `acp`-less binary marks the harness INCAPABLE and stops being
probed at all, rather than re-asking a settled question every few hours.

Probes are spawned off the event loop, never awaited on it: a probe runs a whole agent
turn, and awaiting one on the loop would deafen the node to offers, awards and payments
for its duration (#223). Because the tick that starts them fires every few seconds while
a probe takes minutes, `claim_due_probes` marks a harness in-flight in the same locked
step that hands it over — otherwise the node would stack a dozen concurrent turns of its
own tokens on one harness. Whichever verdict lands releases the mark.

Refs #254

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Jul 30, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
mobee Ready Ready Preview Jul 30, 2026 10:02am

Request Review

…kdir path

Review finding R1. The workdir was named after the sentinel while the readback matched
file CONTENT, so the sentinel lived in the harness's own cwd. Any harness that echoes a
path into a file — an error trace, a log header, a partial write — reproduced the sentinel
without doing the task, and a dead harness was restored to the roster as "serving". That
defeats the one property the probe exists to establish.

A discriminator must not appear in the environment it discriminates. So a probe now mints
TWO values: a non-secret label that names the workdir and may be echoed freely, and the
sentinel, which reaches the harness only through the prompt and carries sub-second entropy
the label does not. Neither is derivable from the other.

The readback also subtracts the workdir's own path from file content before matching. That
is belt to the mint's braces: it makes the property hold for ANY caller, so reintroducing
the leak at a call site cannot silently revive a dead harness. Red-proved — remove the
subtraction and a file containing nothing but the cwd path passes a probe the harness did
no work for.

Folded in with the rename this was found alongside: the probe's nonce is a SENTINEL, never
a "token". In this crate a token is Cashu ecash — `payload.to_token()`, `token_hash`,
"already-spent token" all appear in this same file — so two identically-named locals sat
here with one of them holding money, and a reviewer had to stop and ask whether probing
spends sats. It spends nothing but our own inference. A name that forces that question on a
money path is wrong whatever the answer is. The prefix and the mint each have exactly one
definition, so no second literal can drift out of step.

Refs #254

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant