feat(sponsorship): pay the Celestia fee only for offers priced below reference - #55
Conversation
e80fe37 to
b84d34e
Compare
Rebased onto the corrected base (Q-10: USDM is
|
| Check | Before the rebase (e80fe37) |
Now (be29306) |
|---|---|---|
bun test packages/database packages/node packages/offer-guard packages/price-feed packages/batcher packages/validator |
541 pass / 2 skip / 0 fail | 542 pass / 2 skip / 0 fail, 3801 expects, 54 files |
bun run typecheck |
0 / 0 / 0 | node 0, solver-set 0, price-feed 0 |
bun run check:pgtypes |
green | green (this branch touches no SQL) |
The +1 is inherited from the base branch — a new schema test asserting the asset_prices.source CHECK rejects 'fixed'. Nothing in this branch's own test set changed except the source list in price-cache.test.ts.
Unchanged: the 422 NOT_SPONSORED contract, the warn + allow shipped defaults, and the pinned note to retarget this PR to main once #54 merges. The live SC-002 run recorded on this PR was made against the pre-correction base; only the stablecoin's asset id and price changed underneath it, so the gate's behaviour at reference and at 2.5% below is unaffected — it was not re-run.
b84d34e to
be29306
Compare
…rence The batcher holds the wallet that pays for every published offer, and until now it sponsored anything structurally valid: checkForCelestiaSponsorship was a no-op stub. It now applies the same rule the node's quote already shows the maker — evaluateSponsorship() from @zswap-da/offer-guard — so the UI cannot promise sponsorship the batcher then refuses, or vice versa. The batcher has no database, so it polls the node's GET /v1/prices (PriceCache, every 10 minutes by default) instead of resolving prices itself. A failed refresh keeps the last good snapshot and lets it age, which is what makes "the node answered ten minutes ago" distinguishable from "the node has been down for two days". Policy, all validated at startup so a typo throws instead of silently sponsoring everything: BATCHER_SPONSOR_POLICY enforce | warn | off (default warn) BATCHER_SPONSOR_UNPRICED allow | reject (default allow) BATCHER_NODE_API_URL, BATCHER_PRICE_REFRESH_MS, BATCHER_PRICE_MAX_AGE_MS SPONSOR_DISCOUNT_BPS bootstrap only, until the node answers once Defaults keep every current deployment behaving as it does today: warn logs what enforce would have refused, and unpriced test tokens keep flowing. validateInput is now async. The SDK core already awaited it, so nothing on the production path changes; direct callers (the tests) await it now.
…'s refusals
A maker whose offer is not worth a Celestia fee learned it, until now, as a
500 INTERNAL after the blob had already crossed the network — a status that
tells them "server problem, retry unchanged", which is the opposite of what to
do. POST /v1/offers now asks the question itself, before liveness and proof
verification, and answers 422 with the numbers:
{ "error": "NOT_SPONSORED", "reason": "wants 1.0% below reference,
sponsorship needs >= 2.5% below", "give_usd": ..., "want_usd": ...,
"implied_discount": 0.01, "sponsor_discount": 0.025 }
It is a mirror of the batcher's gate, not a second opinion: the same
evaluateSponsorship over the same resolution order the quote and /v1/prices
use, and the same two policy variables the batcher reads
(BATCHER_SPONSOR_POLICY, BATCHER_SPONSOR_UNPRICED). Under the default `warn`
the node logs what `enforce` would have refused and forwards, so this release
changes no deployment's behaviour until the policy is switched.
The pre-check is READ-ONLY: unlike the quote it never writes a demo price row,
so a refused submission cannot create the row that makes the next one look
priced.
When the batcher refuses after the node forwarded (a node in `warn` with a
batcher in `enforce`, or a stale batcher snapshot), NOT_SPONSORED /
UNPRICED_TOKEN / PRICE_UNAVAILABLE now surface as 422 carrying the batcher's
own message, instead of 500.
Also: policy parsing moved into @zswap-da/offer-guard so the two processes
cannot disagree about what a value means; compose and the three env examples
carry the new variables; API.md's "step 5 is not implemented" paragraph is
replaced by what was actually built, including why this is NOT enforced at STM
ingestion (the namespace is permissionless — an unsponsored offer posted
directly is still indexed).
Found by running a real batcher against a real node (Brief B, B4), not by
reading code. The batcher SDK wraps an adapter refusal:
{"success":false,"error":"Validation failed",
"message":"NOT_SPONSORED: wants 0.0% below reference, …"}
`error` is the generic wrapper text, so the mapping added in the previous
commit matched nothing and the maker still got 500 INTERNAL — the exact defect
it was written to remove. Both fields are now scanned, and the test fixture is
the verbatim body a live batcher returned.
Follow-on from the Q-10 correction on the base branch: `PriceSource` no longer has a `fixed` member, so listing it in MARKET_SOURCES stopped typechecking. Every asset price is now a fetched USD price and nothing is pinned to a dollar, so the market sources are `feed | seed | manual`. A `fixed` row from an older node is not silently trusted after this: it falls into the existing "unknown source" path and is downgraded to `fallback`, i.e. the offer is unpriced and BATCHER_SPONSOR_UNPRICED decides. That is the safe direction — no fee is paid against a price this build cannot vouch for.
…e table
Q-11: `PriceCache` polled the node's whole `GET /v1/prices` every ten minutes
and held a snapshot. That is fine for six tokens and wrong for thousands —
the payload grows with the registry, the batcher downloads prices for colours
no offer will ever mention, and a token minted since the last poll is
invisible until the next one.
`PriceLookup` replaces it. `validateInput` collects the offer's leg colours
and asks `GET /v1/prices?tokens=` for exactly those, in ONE request; answers
are cached per colour for `BATCHER_PRICE_TTL_MS` (default 10 min), so a busy
pair costs one request per TTL rather than one per offer, and an unfamiliar
colour is answered now instead of at the next poll. `sponsor_discount` and
the node's feed status are refreshed from every response.
A colour the node returns nothing for is cached as an ANSWER ("no price
here"), not as a gap — otherwise every offer on a test token would re-ask.
That keeps the two questions the policy already separated genuinely
separate: "the node has no market price for this" is
BATCHER_SPONSOR_UNPRICED's, "I could not ask" is BATCHER_SPONSOR_POLICY's.
Failure behaviour is unchanged in spirit and sharper in detail. A failed
re-ask does not erase what is cached: entries keep their original timestamp,
are still served while they are inside `BATCHER_PRICE_MAX_AGE_MS`, and only
then does the colour become unavailable (enforce → PRICE_UNAVAILABLE with
the colours named, warn → sponsor + log once a minute, off → skip). So a
brief node restart no longer makes every offer unavailable.
`BATCHER_PRICE_REFRESH_MS` is dropped with the poll it configured — setting
it is inert rather than an error, so a stale compose file does not block a
deploy. `BATCHER_PRICE_MAX_AGE_MS` keeps its name and gains a startup check
that it is >= the TTL: the reverse would strand colours that have a
perfectly recent answer in hand.
One optional request survives at startup — the NIGHT colour, seeded on every
network — so an operator can still read "am I price-aware?" off the startup
log. It is fire-and-forget and non-fatal.
Verified live against the real node API and the real batcher binary on
random free ports (Celestia deliberately unreachable, torn down after): one
probe at startup and nothing else; at reference the node answers 422 and the
batcher refuses with zero Celestia submissions, having asked only for the
want leg (the give leg was already cached); the same offer again asks the
node nothing; 2.5% below reference the offer queues.
be29306 to
89cfc3e
Compare
Q-11 scale amendment: the batcher looks prices up per offerHead is now Why:
|
| Check | be29306 |
89cfc3e |
|---|---|---|
bun test packages/database packages/node packages/offer-guard packages/price-feed packages/batcher packages/validator |
542 pass / 2 skip / 0 fail | 572 pass / 2 skip / 0 fail, 3909 expects, 54 files |
bun run typecheck (3 gates) |
0 / 0 / 0 | 0 / 0 / 0 |
bun run check:pgtypes |
green | green (no SQL here) |
+30 = +15 inherited from the rebased base and +15 here: price-lookup.test.ts 25 → 35, sponsorship-gate.test.ts 21 → 26.
Live check — the request shape, measured
Real node API + the real batcher binary over real sockets, random free ports (15939 PGlite / 56934 node / 47807-8 batcher), Celestia deliberately unreachable, torn down after (ports free, storage dir removed, git status clean, no docker artifacts). The harness added one fastify onRequest hook of its own to record every request the batcher made.
Startup — one probe and nothing else:
[zswap-da-batcher] prices: asked for 1 color(s), 1 priced, discount 2.50% from http://127.0.0.1:56934/v1/prices
[zswap-da-batcher] sponsorship: policy=enforce unpriced=allow prices=1 color(s) cached, last answer 0s ago, ttl=600s max_age=172800s discount=2.50% node=…
node requests during startup: ["GET /v1/prices?tokens=0000…0000"]
At reference, enforce on both sides:
POST /v1/offers → 422 {"error":"NOT_SPONSORED","reason":"wants 0.0% below reference, sponsorship needs ≥ 2.5% below","give_usd":19181.81,"want_usd":19181.81,"implied_discount":0,"sponsor_discount":0.025}
POST /send-input → 400 {"success":false,"error":"Validation failed","message":"NOT_SPONSORED: wants 0.0% below reference, … (give_usd 19181.81, want_usd 19181.81)"}
node requests for this offer: ["POST /v1/offers","GET /v1/prices?tokens=ffff…ffff"]
That is the amendment, live: one lookup, naming only the want leg — the give leg was already cached from the startup probe.
The same offer again: node requests: [].
2.5% below reference (fresh batcher, empty cache): 200 {"success":true,"message":"Input queued for batching","inputsProcessed":1}, 🔗 Processing 1 inputs for target: celestia, again exactly one ?tokens=ffff… lookup. Zero Celestia submissions across both phases — the DA transport was the only thing denied.
Every node request the batcher made, in order — GET /v1/prices never appears without ?tokens=:
GET /v1/prices?tokens=0000…0000 ← startup probe
GET /v1/prices?tokens=ffff…ffff ← at reference, the want leg
GET /v1/prices?tokens=0000…0000 ← fresh process, probe again
GET /v1/prices?tokens=ffff…ffff ← 2.5% below, the want leg
Two honest gaps in the harness, neither a gate finding: the 2.5%-below POST /v1/offers answered 500 relation "effectstream.effectstream_blocks" does not exist — the framework's own block store, which bare migrationTable does not create; the node's pre-check passed there (no 422, unlike at reference), and the batcher path was driven directly through /send-input. And implied_discount reads 0 where B4 recorded 2.22e-16, because this harness computes the exact at-reference price instead of a rounded literal.
Still unchanged: the shipped defaults (warn + allow), the 422 contract, and the pinned note to retarget this PR to main once #54 merges.
…feed feat(batcher): sponsorship gate — pay the Celestia fee only for offers priced below reference (re-land of #55 onto main)
1 participant
What
The batcher pays the Celestia fee only for offers whose wanted value is at least
the sponsorship discount below the given value at the node's reference prices —
and the node says the same thing earlier, and more clearly.
Until now
checkForCelestiaSponsorshipwas a no-op stub: every structurallyvalid offer was sponsored, including one priced exactly at reference or above
it. And when the batcher did refuse for any reason,
POST /v1/offersanswered500 INTERNAL— a status that tells the maker "server problem, retryunchanged", which is the opposite of what to do.
Builds on #54 (base is that branch; retarget to
mainonce #54 merges).Not breaking, by construction
The rollout defaults are
BATCHER_SPONSOR_POLICY=warnandBATCHER_SPONSOR_UNPRICED=allow, so on any deployment that sets nothing:warnonly logs whatenforcewould have refused;The intended rollout is: deploy with the defaults, read a day of
would refuse (policy=warn) — NOT_SPONSORED: …lines from both processes, thenset
enforce.The one caller-visible change:
ZswapCelestiaAdapter.validateInputis nowasync. The SDK core already awaited it (batcher-sdk core/batcher.ts:513), sonothing on the production path changes; direct callers (three test files) await
it now.
How
One rule, three readers, no drift.
evaluateSponsorship()in@zswap-da/offer-guard— added in #54 and already behind/v1/quote'ssponsoredflag — is now also what the batcher and the node's pre-check ask.packages/batcher/price-lookup.ts(89cfc3e; wasprice-cache.ts) — thebatcher has no database and does not mirror the price table: for each
offer it collects the leg colours and fetches the missing or expired ones in
ONE
GET /v1/prices?tokens=…request, cached per colour forBATCHER_PRICE_TTL_MS(10 min default).sponsor_discountand the feedstatus are refreshed from every response. A failed refresh keeps the cached
entry and lets it age; past
BATCHER_PRICE_MAX_AGE_MSit stops countingas an answer (a colour with no usable entry is unavailable; a colour the
node does not price is unpriced). One startup probe (
?tokens=<NIGHT>)answers "am I price-aware?" — the only pre-offer request.
ZswapCelestiaAdapter.validateInput— the gate runs LAST, after proofverification. It is the cheapest check in the ladder, but it needs
gives/wants, and refusing an offer with values read out of an unverifiedtransaction would be refusing it with numbers that were never real.
POST /v1/offers— a mirror, not a second opinion: same function, sameresolution order as the quote and
/v1/prices, same two env variables. Itruns after the byte-identical dedup probe and before liveness/crypto, and is
read-only — unlike the quote it never writes a demo price row, so a
refused submission cannot create the row that makes the next one look priced.
batcher-client.ts—NOT_SPONSORED/UNPRICED_TOKEN/PRICE_UNAVAILABLEfrom the batcher now surface as422carrying thebatcher's own message. Every other batcher failure is still a 500; it is not
the maker's fault.
New
422body onPOST /v1/offers:{ "error": "NOT_SPONSORED", "reason": "wants 0.0% below reference, sponsorship needs ≥ 2.5% below", "give_usd": 19181.81, "want_usd": 19181.81, "implied_discount": 0, "sponsor_discount": 0.025 }This is NOT enforced at STM ingestion, deliberately
The MIP-0006 namespace is permissionless: an unsponsored offer posted straight
to Celestia at the maker's own expense is still indexed. The gate decides who
rides the batcher's wallet for free, not what is a valid offer.
API.md's"step 5 — not implemented" paragraph is rewritten to say exactly this, so a UI
does not start assuming every indexed offer was sponsored.
Policy
BATCHER_SPONSOR_POLICYenforce|warn|offwarnBATCHER_SPONSOR_UNPRICEDallow|rejectallowBATCHER_NODE_API_URLhttp://127.0.0.1:9999BATCHER_PRICE_TTL_MS600000BATCHER_PRICE_MAX_AGE_MS172800000(48 h); must be ≥ the TTL (startup error otherwise)BATCHER_PRICE_REFRESH_MS89cfc3e(inert if still set)SPONSOR_DISCOUNT_BPS250The two processes read the same names, and the parsing lives in
@zswap-da/offer-guardso they cannot disagree about what a value means. Aninvalid value throws at startup: an operator who typed
enfrocewantsoffers refused, and silently sponsoring everything is what they were
preventing. On the batcher
SPONSOR_DISCOUNT_BPSis only a bootstrap — oncethe node answers, the node's
sponsor_discountwins, so the batcher cannotrefuse at a threshold the UI never showed the maker.
compose.ymlpasses the knobs to bothkernelandbatcher. There isdeliberately no
depends_onfrom batcher to kernel — the kernel alreadypoints back at the batcher and the pair would deadlock; the startup probe is
non-fatal by design.
Tests
bun test packages/database packages/node packages/offer-guard packages/price-feed packages/batcher packages/validator→ 541 pass / 2 skip / 0 fail, 3798 assertions, 54 files
(
mainbaseline 378, after #54 474 — +67 here).bun run typecheck0 diagnostics on all three gates;bun run check:pgtypesgreen.New suites:
packages/batcher/price-cache.test.ts(25),packages/batcher/sponsorship-gate.test.ts(21),packages/node/offers-sponsorship.test.ts(21).Everything is driven with the real proven offer fixture
(
packages/validator/fixtures/valid-offer.bech32, gives 1 000 000 NIGHT / wants5 000 000 of a test colour), so the price of the wanted leg is the only dial and
give_usd = 19181.81can be checked by hand.Red before green: on the base branch that fixture is sponsored
(
validateInput -> {"valid":true}) andPOST /v1/offersforwards it to thebatcher even with
BATCHER_SPONSOR_POLICY=enforceset, because nothing readsit. Both assertions now flip.
Verified against real processes
The real node and the real batcher binary were run as two processes over
real sockets on random free ports, with only Celestia absent (this host could
not take a second full devnet — another agent's Midnight stack was up). Real
HTTP between them throughout; torn down after.
89cfc3e): atstartup only the probe
GET /v1/prices?tokens=0000…0000; validating theat-reference offer produced exactly
GET /v1/prices?tokens=ffff…ffff(thewant leg — the give leg was already cached); the repeat produced no request
(TTL).
GET /v1/pricesnever appears without?tokens=. (The first run one80fe37used the table mirror:prices: 4 tokens, discount 2.50%.)422 NOT_SPONSORED; the same blob straight to/send-input→400 … "NOT_SPONSORED: wants 0.0% below reference …", andzero Celestia submissions — the SDK threw
InputValidationErroratcore/batcher.ts:513.(
Processing 1 inputs for target: celestia); the only failure left is thedeliberately unreachable Celestia RPC.
warn+ batcherenforce: the node logs, forwards, and answers422with the batcher's own sentence.
200 queued, plussponsoring an unpriced offer (… BATCHER_SPONSOR_UNPRICED=allow).That run found a real bug (fixed in
e80fe37): the SDK wraps an adapterrefusal as
{"error":"Validation failed","message":"NOT_SPONSORED: …"}, and thefirst version of the mapping read only
error— so it matched"Validation failed"and still answered 500. Both fields are scanned now, andthe test fixture is the verbatim body a live batcher returned.
Not done here
bun run test:grand— needs a fresh full Midnight + Celestia devnet(~88 minutes exclusive). This machine is shared and its Docker was hosting
another stack. This branch adds no tables and no SQL, so the suite's three
static schema lists are untouched, and under the shipped defaults neither
process refuses anything.
exit criterion on preprod, together with the
enforceswitch and thebrowser walkthrough.
Found in passing, not fixed
packages/batcheris in no typecheck gate and does not currently typecheck(4 pre-existing errors in
celestia.tsandmidnight-balancing.ts). One realerror in this branch's new code was caught only by an ad-hoc
tscrun. Writtenup separately; adding the gate means fixing those four first, which is not this
change's scope.