Skip to content

feat(deploy): offer poster — mint WBTC and post one exact-coin ZSwap offer per minute, re-offering released coins - #57

Merged
acedward merged 8 commits into
mainfrom
00007-offer-poster-service
Sep 3, 2026
Merged

feat(deploy): offer poster — mint WBTC and post one exact-coin ZSwap offer per minute, re-offering released coins#57
acedward merged 8 commits into
mainfrom
00007-offer-poster-service

Conversation

@acedward

@acedward acedward commented Sep 3, 2026

Copy link
Copy Markdown
Contributor

What

A long-running Docker Compose service, offer-poster, in deploy/, off by default (docker compose --profile poster up -d offer-poster). Given a DUST-funded wallet it mints GIVE_AMOUNT WBTC every POST_INTERVAL_MS (default 60s, fees from its own DUST) and posts one ZSwap offer whose only input is that exact coin, sized from the kernel's own quote so the batcher sponsors it (payFees:false). Released coins (offer expired/cancelled, or orphaned by a crash between mint and post) are re-offered before minting a new one. Every coin and every offer built from it is recorded in a durable, atomically-written journal on its own volume.

Spec: spec/00007-offer-poster-service.md in the organizer workspace. Plan and full research/measurement trail: plans/00007-offer-poster-service.md (same workspace) — every claim below has a path and line number there.

Exact-coin mechanism

Coin selection is not addressable through the public wallet API (initSwap takes {colour: amount} only; the default selector is smallest-first). This service assembles its own wallet facade (deploy/scripts/lib/pinned-wallet.ts) around CustomShieldedWallet(...).withCoinSelection(...), arming a pinned selector with the just-minted coin's nonce before building each offer and disarming it in a finally. After finalizeTransaction, the tick asserts collectNullifiers(tx) deep-equals [coin.nullifier] and the fallible offer has zero inputs — on any mismatch the recipe is reverted (wallet.revert) and nothing is posted. Verified live in P5: 10/10 posted offers' computed.inputNullifiers (read back from the kernel, not assumed) matched the journaled nullifier of the coin minted that tick; three pre-loaded "control" WBTC coins the poster never minted were never touched across a full day of testing (mint, re-offer, settlement, a mid-tick kill, and an enforce-mode run) — confirmed via a final availableCoins snapshot, byte-identical to the pre-test one.

Journal and re-offer semantics

A JSON file at POSTER_JOURNAL_FILE (own named volume, atomic temp+rename writes) keyed by coin nonce: {type, value, nullifier, mintTx, state: minted|offered|spent|lost, offers: [...]}. Every tick reconciles non-terminal offers against GET /v1/offers/:hash/status (the kernel emits exactly live|consumed|cancelled|expired|not_found — never rejected, which is poster-local for a 4xx-refused post) and only proposes a coin as a re-offer candidate when both its latest offer is kernel-terminal and the coin is back in the wallet's availableCoinscancelled does not by itself prove release (a partial/split settlement is also classified cancelled), so availableCoins is the actual proof, not the status. A crash between mint and post leaves the coin journaled as minted (written as the first action after the mint confirms, before quote/build/post) and it is picked up as a candidate on the next reconcile rather than leaked.

Live-verified (P5, SC-003/SC-004): a coin whose offer expired carried a second offer after mode=reoffer fired with zero mints in that tick; a docker kill issued right after the mint log line and before post produced a clean orphan re-offered on the very next tick after restart (RestartCount=0, no crash loop, no duplicate offer).

Want leg: sized from the kernel's own quote

Per build, GET /v1/quote?from_token=<give>&to_token=<want>&from_amount=<coin.value>suggested_to_amount already lands exactly on the sponsorship threshold, so the offer is sponsored by construction unless WANT_AMOUNT forces a fixed amount (then the poster only warns; the kernel is the one that actually refuses it under enforce). Verified under BATCHER_SPONSOR_POLICY=enforce on both kernel and batcher (SC-007): normal operation stayed accepted for 3 ticks (0 NOT_SPONSORED/UNPRICED_TOKEN in either service's logs); a forced, ~150x-over-threshold WANT_AMOUNT was refused by the kernel (422 NOT_SPONSORED), the recipe was reverted, and the offer id came back 404 — genuinely nothing was posted, not just "assumed refused."

Fees

Mints are balanced and submitted from the poster's own DUST (not routed through the batcher). ZSwap offers stay payFees:false — the taker balances, the batcher sponsors the Celestia fee.

P5 measurements (live undeployed stack, project op00007, 2026-09-03)

  • SC-001: 10/10 offers live, 10/10 computed.inputNullifiers matched, 0/12 ticks failed, control coins untouched throughout.
  • SC-002: one posted offer settled via a funded taker; journal flipped the coin to spent on the very next reconcile poll.
  • SC-003: re-offer demonstrated (a nonce carrying two offers, mints < ticks, mode=reoffer logged before any mint) — required aligning the kernel's own OFFER_TTL_SECONDS alongside the poster's TTL knob (see the TTL clarification below).
  • SC-004: kill between phase=mint and phase=post → clean orphan, re-offered within 1 tick (bound was 2), no duplicate, no crash loop.
  • SC-007: enforce mode accepted normal operation, refused a forced over-threshold amount as above.
  • Tick timing: p50TickMs=48407, p95TickMs=51978 — comfortably under the 60s interval, 0 overruns across 12 ticks.
  • DUST: at the tested funding level (4×5e12 NIGHT = 2e13 total, the same shape provision-solver-fees.ts uses) DUST generation outpaced one mint's cost by roughly three orders of magnitude — the wallet's DUST balance rose on every sampled before/after mint, never fell, over 12 consecutive 1-mint/min ticks. A per-mint fee could not be isolated from a balance diff at this generation rate (filed as issues/00014 — the mint path doesn't log tx.fees()/tx.cost() directly). POSTER_MIN_DUST is left at the code default (1, "any DUST at all") — empirically correct at this funding level, reasoning recorded in .env.example.
  • Sync time: negligible and journal-size-independent up to 43 coins / 53 offers on this devnet (chain height, not journal size, is the real driver — noted as non-extrapolatable to preprod scale).
  • Teardown: ./down.sh (no -v — an unknown flag would exit 2) left zero op00007 containers/volumes/networks, independently re-verified with fresh docker ps -a/volume ls/network ls filters.

New env knobs

All under deploy/.env.example's new "offer poster (profile: poster)" block: OFFER_POSTER_SEED/_MNEMONIC (required, xor), _GIVE_TOKEN (WBTC), _GIVE_AMOUNT (1000), _WANT_TOKEN (WETH), _WANT_AMOUNT (blank = quoted), _INTERVAL_MS (60000), _TTL_MINUTES (60), _MIN_DUST (blank = code default 1, P5-measured), _DRY_RUN, _JOURNAL_RESET, HOST_OFFER_POSTER_HEALTH_PORT (19977), plus a set of blank-means-default passthroughs (coin-visible timeout, reconcile interval, max re-offers/tick, shutdown grace, health-stale-ticks, sync/dust-wait timeouts, post/live retry knobs). Health endpoint on POSTER_HEALTH_PORT (literal 9977 in the container): /health, /metrics (Prometheus text), /journal.

TTL clarification (Q8)

OFFER_POSTER_TTL_MINUTES is the ttl passed to initSwap — the wallet's own local deadline for an unconfirmed recipe, governing when the wallet gives up on it and releases the coin back to availableCoins. It does not shorten a posted, live, shielded offer's kernel-advertised lifetime, which is min(ROOT_WINDOW_SECONDS, OFFER_TTL_SECONDS) on the kernel process (currently 1h on every network) and reads nothing per-offer from any client — traced in packages/node/state-machine.ts and packages/node/env.ts. This surprised SC-003's first attempt (candidates stayed at 0 for 7+ ticks with only the poster-side knob shortened) and is corrected in this PR's deploy/.env.example and deploy/README.md (previously they read as if the knob controlled both). Not a poster correctness bug — re-offer candidacy already fails closed on the kernel-side half, so a short poster TTL alone is safe, just produces no re-offers by itself. Full trace in plan Q8; OFFER_TTL_SECONDS/ROOT_WINDOW_SECONDS promotion to a documented stack-wide knob is filed separately as issues/00013 (kernel/batcher-wide concern, out of this PR's scope).

Changes outside deploy/

  • packages/solver-core/offer-files.ts: joinOfferFiles(walletResult, address?) gains an optional second argument, defaulting to the existing getContractAddress() behaviour. Additive — all 12 existing one-argument call sites (all in packages/tests) are unaffected. Needed because getContractAddress() hard-codes the undeployed compose stack's contract-share path, which is wrong for a preprod dry run (SC-006); the poster resolves its own address in priority order (MIDNIGHT_CONTRACT_ADDRESS → the shared contract-share file → the repo-local fallback).
  • Root package.json / bun.lock: eight @midnightntwrk/wallet-sdk-* packages promoted to root direct dependencies, at the versions the lockfile already resolved for @effectstream/midnight-contracts. Necessary because deploy/ is not a workspace member and resolves bare specifiers from the root only. The lockfile update also reshuffled @polkadot/types' fallback-root hoisting; verified inert — a full symlink-graph diff against the pre-change tree showed exactly 9 changed links (the 8 new root links plus the one fallback-root @polkadot/types link used only by undeclared imports; every package that declares its own @polkadot/types version is unaffected), and bun install --frozen-lockfile passes clean both before and after.
  • deploy/images/kernel/Dockerfile (pre-existing bug, not introduced by this branch, but fixed here because P5 needed a buildable image): the COPY block never included packages/price-feed's manifest, so no kernel image could build from main since the price-feed package was added (feat(prices): seeded asset prices, price-feed service (CoinGecko), /v1/prices, quote sources #54). One COPY line added; unit-tested (kernel-image-manifests.test.ts). Calling this out prominently since it means main itself has been unbuildable — worth confirming on the maintainer's side independent of this PR.
  • .github/workflows/ci.yml: deploy/scripts/lib added to the "Run unit suites" step. That directory is not a workspace member so it was previously invisible to CI entirely — none of this service's own unit tests (or the pinned-wallet/faucet-mint/journal/quote modules') ran anywhere before this.

Not a breaking change

With the poster profile off (the default), docker compose config --services is byte-identical to this branch's base — measured, not assumed. deploy/down.sh gained --profile poster on both its docker compose down invocations so the new service and its offer-poster-state volume are actually removed by a plain ./down.sh once the profile has been used (Compose only tears down what the selected profiles select).

CI caveat

deploy/scripts/lib (and every other suite) now runs on CI's pinned bun 1.3.3, while every measurement and test run above was taken on bun 1.3.11. That pin is deliberate (see the ci.yml comment) but a known WASM-segfault flake exists on 1.3.3 for wasm-heavy suites (this repo touches ledger wasm in faucet-mint.test.ts/pinned-wallet.test.ts and Bun.serve on port 0 in poster-health.test.ts) — if the first CI run on this PR shows a segfault, rerun once before investigating further.

Testing

  • bun test deploy/scripts/lib: 230/230 pass (9 files, 1229 expect() calls) — pinned-selector, faucet-mint colour derivation, journal round-trip/candidacy, quote sizing, config parsing, tick/scheduler/health logic, and the kernel-image-manifest fix, all in the space-free scratch clone.
  • CI's exact unit-suite command (with its Compact-runtime stub and check:pgtypes steps replicated locally): 1356 pass / 2 skip / 1 pre-existing failure (solver-offerfiles-real-image-secret-scan.test.ts's large-inventory case — reproduces on the pre-branch tree too; filed as issues/00018, not caused by this branch).
  • typecheck:backend, typecheck:solver, and the aggregate typecheck (adds typecheck:price-feed): 0 diagnostics each, with CI's declaration-only Compact stub. (deploy/ itself has no repo-level typecheck gate at all — pre-existing, filed as issues/00016; the new files were separately typechecked by hand throughout P1–P3 with 0 diagnostics.)
  • Live stack test (P5, undeployed, project op00007): SC-001, SC-002, SC-003, SC-004, SC-007 all verified against a real compose stack, as detailed above; SC-006 (preprod dry run) verified read-only against preprod in this PR's own handoff — see the plan for the exact result and the .env block for whoever deploys preprod next.

Related issues filed (out of this PR's scope)

issues/0001200020 in the organizer workspace, covering: compose image tags not project-scoped (a real cross-session collision hit during P5, worked around with a session-unique tag — not fixed here since it's a pre-existing, stack-wide hazard); the undocumented kernel TTL knobs; missing DUST-fee instrumentation on the mint path; a stale api-example; deploy/'s missing typecheck gate; several small API.md/README.md drifts; the flaky secret-scan test; one transient, self-recovered RPC error; and the lack of a reusable "settle one poster offer" smoke-test helper.

…et mint, journal, quote sizing

- pinned-wallet.ts: buildWalletFacade reproduced with CustomShieldedWallet +
  a coin selector that, when armed, returns exactly one (type, nonce) coin or
  nothing; the eight @midnightntwrk/wallet-sdk-* packages become root deps at
  the versions the lockfile already resolved (hoisting reshuffle only).
- faucet-mint.ts: zswap-da faucet colour derivation (rawTokenType over the
  FNV domain separator), mint_shielded wrapper returning the coin + nullifier,
  known-tokens registration that disambiguates the 409 cases.
- poster-journal.ts: atomic file-backed coin -> offers record with kernel
  status mapping and re-offer candidate selection gated on availableCoins.
- poster-quote.ts: want-leg sizing from GET /v1/quote (suggested_to_amount,
  sponsored verdict).

117 unit tests, no network.
…alth

The long-running poster (spec 00007, FR-001…FR-016). Every POST_INTERVAL_MS it
posts one Offer File whose ONLY input is a coin it can name: a coin an earlier
offer released (terminal-but-unspent AND back in availableCoins), or one it
mints that tick from its own DUST.

Decisions live behind injected dependencies so they are unit-testable without a
wallet, a node or a kernel:

  deploy/scripts/lib/poster-config.ts     env -> PosterConfig; POSTER_SEED xor
      POSTER_MNEMONIC (BIP-39 seed derived exactly as midnight-env does it),
      seven refused seed collisions, blank-is-unset, offline colour derivation,
      contract-address resolution (env -> share dir -> package dir, network
      aware), and redaction so a seed can never reach a log line
  deploy/scripts/lib/poster-tick.ts       reconcile + tick + offerCoin +
      mintCoin, the failure taxonomy, and the exact-coin assertion: the builder
      returns the nullifiers, the tick decides, reverts the recipe on any
      mismatch and posts nothing
  deploy/scripts/lib/poster-scheduler.ts  fixed-RATE, never-overlapping loop
      with injected clock/timers; overruns warn, count and start the next tick
      at once; rolling p50/p95
  deploy/scripts/lib/poster-health.ts     /health (503 only after
      HEALTH_STALE_TICKS consecutive FAILED ticks), /metrics, /journal
  deploy/scripts/offer-poster.ts          wiring only: the pinned facade, the
      sync that does not block on dust progress, DUST registration, the contract
      join, token-name registration, the journal, DRY_RUN, signals, shutdown.
      main() runs only under import.meta.main.

packages/solver-core/offer-files.ts gains an OPTIONAL contractAddress argument
on joinOfferFiles; getContractAddress() hard-codes the `undeployed` file name,
which is right for the compose stack and wrong for preprod. Every existing
one-argument caller is unaffected.

offer-poster.ts imports packages/solver-core/offer-files.ts DYNAMICALLY: that
module builds a CompiledContract at load time from Compact build output a fresh
clone does not have, so a static import would make the file unloadable in tests
and in CI. It also avoids rxjs, which does not resolve from deploy/.

Tests: 98 new (15 scheduler, 35 config, 36 tick, 12 health); the whole
deploy/scripts/lib suite is 215/215. Typecheck clean in all nine new files
under the repo's compiler options and under the stricter set. No change to the
full-suite failure set.
…`poster`)

Nothing changes with the profile off: `docker compose config --services` is
byte-identical, and no existing service, volume or port is touched.

- deploy/images/kernel/entrypoint-offer-poster.sh: sources entrypoint-common.sh,
  requires ZSWAP_API + MIDNIGHT_NETWORK_ID, fails fast (78) when neither
  POSTER_SEED nor POSTER_MNEMONIC is set (naming the OFFER_POSTER_* keys the
  operator actually edits), unsets every blank poster knob, adopts the contract
  address, waits for the kernel API, mkdir -p's the journal directory and
  `exec`s bun so PID 1 receives SIGTERM. No marker file: this is a loop, not a
  one-shot, and idempotence lives in the journal.
- deploy/images/kernel/Dockerfile: COPY the new entrypoint (after `COPY . .`, so
  the ~20 min proving-key layer stays cacheable); header list NINE -> TEN.
- deploy/compose.yml: `offer-poster` behind profile `poster`, restart
  unless-stopped, depends_on kernel healthy + offerfiles-deploy completed,
  offerfiles-deploy read-only + a new offer-poster-state volume, bun healthcheck
  on 9977 with a 15 m start_period, health port published on
  ${BIND_ADDR}:${HOST_OFFER_POSTER_HEALTH_PORT:-19977}. The Midnight endpoints
  are spelled out instead of merging *midnight-endpoints, because that anchor
  carries MIDNIGHT_WALLET_SEED and the poster refuses to share a seed.
- deploy/down.sh: --profile poster on both `docker compose down` lines, or the
  container and its chain-keyed volume survive teardown.
- deploy/.env.example: OFFER_POSTER_* block (dedicated seed, funding, quote-sized
  want leg, sponsorship must match on kernel and batcher) and the host port.
- deploy/README.md: "Offer poster" — exact-coin guarantee and how to verify it,
  journal/re-offer semantics, funding, one facade per seed, DRY_RUN, the
  /health /metrics /journal endpoints, and the preprod env block.
- .github/workflows/ci.yml: run `deploy/scripts/lib` in the unit job. Those
  suites (215 tests) have never been visible to CI.
The 00005 price service (bc5c673) added `packages/price-feed` as a workspace
member but never added it to the kernel Dockerfile's manifest-copy list, so the
`bun install --frozen-lockfile` in that layer resolved a different workspace
set from the one bun.lock records and refused every build from that merge
onwards:

    error: lockfile had changes, but lockfile is frozen

The image is normally reused from a cache, so nothing exercised the build until
00007's P5 needed one. Reproduced by replaying the Dockerfile's manifest-only
context by hand: it fails without packages/price-feed/package.json and installs
2231 packages in 3.9 s with it.

kernel-image-manifests.test.ts keeps the list honest — it expands the root
package.json's workspace globs and asserts a COPY line names each manifest. It
fails on the unfixed Dockerfile and passes on the fixed one.
…xample

DUST regenerates continuously from registered NIGHT at a rate roughly
three orders of magnitude faster than a single mint's fee at the
funding level P5 tested (4x5e12 NIGHT); the wallet's DUST balance rose
on every single before/after sample across 12 consecutive mints, so a
positive per-mint cost could not be isolated and the code default (1,
"any dust at all") is left as the recommended floor rather than
replaced with a fabricated number.
The knob bounds initSwap's ttl -- the wallet's local deadline for an
unconfirmed recipe, releasing the coin back to availableCoins if it never
confirms. It does not shorten a posted shielded offer's kernel-advertised
lifetime, which is min(ROOT_WINDOW_SECONDS, OFFER_TTL_SECONDS) on the kernel
process (currently 1h on every network) and reads nothing per-offer from the
client -- traced in packages/node/state-machine.ts and packages/node/env.ts,
per plan Q8 (resolution: option A, docs only).
Both sides added services, entrypoints and docs at the same anchors:
register-minted-tokens (main) and offer-poster (this branch) are kept side
by side; the packages/price-feed manifest fix landed on main independently,
so the duplicate comment is dropped and the COPY line kept once.
@acedward
acedward merged commit dc839db into main Sep 3, 2026
5 checks passed
This was referenced Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant