ledger-v9 port of the COW solver (stacked on 00001-ledger-v9) - #50
Draft
acedward wants to merge 16 commits into
Draft
ledger-v9 port of the COW solver (stacked on 00001-ledger-v9)#50acedward wants to merge 16 commits into
acedward wants to merge 16 commits into
Conversation
Moves the contract half of the ledger-v9 migration onto the node-2.x toolchain: compactc 0.33.0-rc.2 (language 0.25, runtime 0.18.0-rc.1, compact-js 2.5.5-rc.7). offer-files.compact compiles clean with no source edits -- the language-0.25 Field/Uint change does not bite, since the only arithmetic here is already annotated `as Uint<128>`. `compact compile +0.33.0-rc.2` is not an option: the compact version manager does not publish the 0.33 line (compact list tops out at 0.31.1). So infra/ fetches the pinned compactc release asset from LFDT-Minokawa/compact into a container, and infra/compact.sh runs it -- honouring $COMPACTC if a host install is available instead. We deliberately do NOT pass --feature-zkir-v3. compactc 0.33 defaults to ZKIR v2, and the emitted keys are tagged verifier-key[v6], which the ledger maps to ProofVersioned::V2 -- provable by the plain proof-server:9.0.0-rc.5. The _experimental proof-server build is needed only for the zkir-v3 lane. Adding that flag later means switching every deployment's proof server too. Worth noting: all twelve ZK artifacts are byte-identical to the compactc 0.31.0 output, so verifier keys are unchanged and this bump alone does not force a contract redeploy. Only the generated TypeScript moved. The rest of the migration -- renaming @midnight-ntwrk/ledger-v8 to @midnightntwrk/ledger-v9 across the workspace -- is NOT here, and cannot be until five @effectstream/* packages move off ledger-v8 upstream. Even packages/validator, which otherwise has no effectstream runtime coupling, would end up with two ledger wasm modules in one process because @effectstream/mip-zswap-offer hard-codes the v8 specifier in a value import, and two copies fail every cross-copy instanceof check. LEDGER-V9-MIGRATION.md records the full audit, the blocker, and the prepared v8->v9 API delta.
Moves the whole workspace off the ledger-v8 line onto the published
ledger-v9 set: @effectstream/*@0.200.1, @effectstream/mip-zswap-offer@
0.4.0-v9.0 (dist-tag ledger-v9), @midnightntwrk/ledger-v9@1.0.0-rc.3,
midnight-js 5.0.0-beta.6, wallet-sdk 2.0.0-beta.2 line, compact-runtime
0.18.0-rc.1 (compactc 0.33.0-rc.2 via infra/compact.sh, pinned earlier).
Runs against midnight-node 2.0.0-rc.4 / indexer 4.4.0-rc.1 (API v4
explicit) / proof-server 9.0.0-rc.5.
BREAKING: every package manifest changes scope/major for the Midnight
SDK line; the serialized transaction wire format moves from
transaction[v9](signature[v1],...) to transaction[v12](signature[v2],...),
so v8-era blobs and fixtures do not deserialize. The committed validator
fixture is regenerated against a live rc.4 stack.
Beyond the 31 import-site migration, the changes of substance:
* validator: v9's UtxoSpend.owner is a tagged SignatureVerifyingKey
({tag: 'schnorr'|'ecdsa', value}), not a bare hex string. derive.ts
passes the tagged object to addressFromKey (bare-string inputs are
wrapped as schnorr); shapes.testkit.ts builds tagged owners. A
rename-only migration compiles clean here and derives garbage.
* signer callbacks: the v2 SDK requires (data) => Promise<Signature>;
every unshieldedKeystore.signData callback moved to signDataAsync
(sync signData made Effect's evaluate().then() throw at sign time).
* e2e: new waitForWalletSettlement gate wherever one facade does a
second prove+submit — node rc.4 rejects stale-DUST reuse with
1010 Custom error 170 (MalformedError::InvalidDustSpendProof).
Deliberately NOT the full-quiescence gate the effectstream e2e uses:
this repo finalizes offer txs that are never submitted on-chain
(they live in Celestia until a taker settles), so pending.all and
shielded pendingCoins stay non-zero by design; the gate is
DUST-subtree quiescence + strict-complete progress.
* api tests: /v1/offers rows are MIP-0006 payloads carrying offerId
(the offer hash), not DB ids — resolve ids by hash while offers are
live; a byte-identical re-submit of an archived offer answers
409 DUPLICATE_OFFER (dedup probes offer_file_history first, by
design) so the negative test accepts that or 400 NULLIFIER_SPENT.
* external-stack mode: new start.external.ts + packages/node/
preflight-external.ts run ONLY the kernel's own processes (pglite,
contract deploy, mint, sync, batcher) against externally-operated
Midnight + Celestia infrastructure. The dev orchestrator cannot be
pointed at an external stack: launchMidnight is unconditional AND
declares stopProcessAtPort [9944, 8088, 6300], which would kill the
external stack's listeners. Celestia auth arrives by sourcing
/celestia/auth/celestia.env when present.
* indexer URLs move to explicit /api/v4 (v3 was an alias).
Verified: bun run test green — 62 tests, 0 failures, full Phase A + B
(deploy, mint, dual-chain sync, batcher settle, multi-token, unshielded-
only, negative paths) with zero error-170 submissions; packages/validator
93/93 including the crypto ladder against the regenerated fixture;
marker-dedup 8/8.
Two failures, both v9-migration fallout the e2e could not see:
* @effectstream/db 0.200.1's startPglite close() DESTROYS live sockets
(0.103.1 closed politely). The db test files deliberately never send a
client Terminate (PGlite WASM throws on it), so every file's teardown
now fired an unhandled 'error' on its still-open pg client — which bun
attributes to the NEXT file, cascading "Connection terminated
unexpectedly" across the whole run and eating ~20 tests' registration.
Fix: swallow the expected teardown 'error' on every client right after
connect() (13 files). CI's exact invocation now passes locally:
313 tests, 0 fail, 0 errors (was 286/4 with 14 errors).
* docs playground: midnight-js 5 moved FetchZkConfigProvider's custom
fetch from a positional argument into { fetchFunc } (and the v5
provider verifies artifacts against the compactc integrity manifest by
default). Same fix as the frontend template. docs tsc -b is clean.
@effectstream/*@0.200.2 (monorepo PR #882, "fix(db): make PGlite force-close opt-in") restores startPglite().close()'s graceful default: live client sockets are no longer destroyed unless force is requested. That was the regression fe61612 worked around by swallowing the expected teardown 'error' on all 13 db-backed test clients. Bump every @effectstream pin 0.200.1 -> 0.200.2 and DELETE all 13 swallow handlers — running without them is the verification. CI's exact invocation: 313 tests, 0 failures, 0 errors.
Integration branch 00001-solver-v9, stacked on 00001-ledger-v9. Both parents fork from 6c5ebab (= main HEAD), so this is a clean two-way merge; neither parent is rebased or rewritten. Conflict resolutions (7 files): - packages/tests/lib/{api,batcher,offer-files,wallet}.ts — took the solver side. The solver promoted these into packages/solver-core and left re-export shims; keeping the v9 implementations here would strand a second, divergent copy of code the solver no longer reads. The v9 rewrites were ported to their new home instead: * solver-core/wallet.ts — the solver's copy was byte-identical to base, so the v9 file (waitForWalletSettlement, the DUST-quiescence gate against rc.4 error 170, post-transfer settle) was taken verbatim. * solver-core/offer-files.ts — 3-way merged, no conflict: keeps the solver's fileURLToPath fix AND v9's @midnightntwrk scope rename. * solver-core/{api-client,batcher}.ts — diverged too far from base to port mechanically; they stay on v8 source and are S2's job. - package.json, packages/node/package.json — v9 pins win on everything the migration set; solver-only additions kept (typescript, ws, @types/ws, @zswap-da/solver-core, typebox 0.34.41, event-client). Dropped the solver's root @midnight-ntwrk/ledger-v8; bumped its root midnight-js-network-id 4.1.1 -> 5.0.0-beta.6 and node's @effectstream/event-client 0.103.1 -> 0.200.2. - bun.lock — regenerated from the v9 side by bun install. Also applied the v9 pin set to the two new workspaces (packages/solver, packages/solver-core), which arrived v8-pinned: @effectstream/* 0.200.2, mip-zswap-offer 0.4.0-v9.0, ledger-v8 8.1.0 -> @midnightntwrk/ledger-v9 1.0.0-rc.3, midnight-js-* 5.0.0-beta.6, compact-js 2.5.5-rc.7, wallet-sdk-address-format 4.0.0-beta.2, wallet-sdk-capabilities 4.0.0-beta.2. bun install completes: zero ledger-v8 on disk, exactly one physical ledger-v9 copy (the single-WASM override holds). Source is NOT yet migrated — that is S2.
The v8 -> v9 source migration turned out to be almost entirely a
specifier rename, because the playbook's "byte-for-byte identical"
list covers everything the solver packages touch: they use only
Transaction, FinalizedTransaction and ZswapLocalState.
- 8 files in packages/solver + packages/solver-core move from
@midnight-ntwrk/ledger-v8 to @midnightntwrk/ledger-v9 (note the
scope also loses its hyphen). Prose mentions swept with them, except
in grand-e2e where the comments cite v8 ledger source line numbers
and remain true statements about v8.
- The two S1-deferred ports collapsed into the same rename: the v9
rewrites of tests/lib/{api,batcher}.ts were themselves nothing but
the specifier change, so solver-core/{api-client,batcher}.ts are now
equivalent to the v9 side. No content was stranded.
- packages/node drops @effectstream/event-client. It was a solver-side
addition whose only consumer was deleted on that same branch
(6f9bde3, the orphaned MQTT app-event path).
Two playbook traps checked rather than assumed:
- ZswapChainState.postBlockUpdate's new required argument: zero call
sites in the solver packages. Not applicable.
- The signature-type change: one holdout each in packages/solver and
grand-e2e still passed the SYNCHRONOUS keystore.signData to
signRecipe, whose signer parameter is
SignSegment = (data) => Promise<Signature>. The v9 parent converted
every one of its own sites to signDataAsync; these two were the last
v8-era stragglers, and both call sites are cast to `any`, so no
typechecker would have caught them. Aligned with the parent.
Also fixes a regression from the S1 pin reconciliation: dropping the
solver's root @midnight-ntwrk/ledger-v8 left api-examples/11 unable to
resolve a ledger at root scope, which breaks this branch's own CI
example-bundling gate. Root now depends on @midnightntwrk/ledger-v9
1.0.0-rc.3 — the same version the override pins, so the single-WASM
guarantee is unaffected (verified: still exactly one physical copy).
Green: solver-core 159/0, solver 264/0, node 141/0 (564 total).
typecheck:backend still 0 diagnostics, dependency-diagnostic count
unchanged at 23. All five CI-bundled api-examples build.
…FERS
An offer whose maker is a CONTRACT CALL (e.g. an AA-Manager open swap) can
never pass wellFormed against the blank reference state: contract-call proof
verification reads the contract's verifier keys FROM STATE, and a blank state
has no contracts - every such offer dies with PROOF_INVALID
('call to non-existant contract'), measured live 2026-08-26 with a proven
selector-6 open-swap offer from the demo stack (which then SETTLED on-chain,
ledger-exact, via the stock balanceFinalizedTransaction taker path - the offer
class is genuine and settleable; only ingestion refused it).
verifyOfferCrypto now takes contractMakerRetry (default off): strict wellFormed
runs FIRST, and only the exact missing-contract failure class widens into one
retry with verifyContractProofs=false - native zswap proofs and signatures stay
verified; the contract-call proof is verified by the node itself at settlement.
Threaded from ALLOW_CONTRACT_MAKER_OFFERS in the node (POST /v1/offers and the
STM ingestion ladder) and the batcher's pre-fee gate. Default remains
fail-closed: the relaxed lane admits ingestion-time spam of unprovable
contract-call offers, bounded by liveness checks, size caps and the limiter.
Tests: validator 93/0, node api+marker-dedup 25/0.
Brings the parent branch's contract-maker carve-out onto the solver integration branch, so the solver mirrors a book that can actually contain AA-Manager open-swap offers. Without it the kernel this solver reads would still refuse every contract-call maker with PROOF_INVALID. Two conflicts, both additive: - packages/node/api.ts — the parent rewrote the ./env.ts import as a single line adding ALLOW_CONTRACT_MAKER_OFFERS; the solver branch had expanded the same import to a multi-line list (rate-limit, SSE and websocket connection caps, event-gate poll). Kept the solver's list and added ALLOW_CONTRACT_MAKER_OFFERS to it. - packages/validator/types.ts — both sides rewrote the same comment block above the liveness predicates. The solver's text describes evaluateOfferLiveness and the UTXO_NOT_LIVE normalization; the parent's appends the contract-maker rationale and the contractMakerRetry field. Both kept, parent's appended after the solver's with a separator. Default stays fail-closed: contractMakerRetry is off unless ALLOW_CONTRACT_MAKER_OFFERS is set. Green after merge: solver-core + solver + node + validator + batcher 677/0. typecheck:backend 0 diagnostics, dependency count still 23.
The DA write path authenticates; the health probe did not. Against an auth-enabled Celestia node the header.NetworkHead probe 401s, the tip reads null forever, and /v1/health/sync reports 'syncing' on a fully-caught-up kernel — starving any consumer that gates on currentness (measured live 2026-08-26 on the demo stack: a book-mirror client blocked at its readiness gate). Token comes from the same CELESTIA_AUTH_TOKEN the batcher config reads; absent (skip-auth deployments) the header is simply omitted.
Conflict in sync-health.ts resolved by grafting the auth header into this branch's fetchJsonWithDeadline idiom (the base's fix was written against the older AbortController shape). sync-health tests 8/0.
…stack The demo stack runs the kernel through `bunx orchestrator start start.external.ts`, which puts six processes in one container and re-deploys the offer-files contract on every recreate. Splitting that into one container per concern (demo-infra Phase 11) needs two things from this repo; both are additive and neither changes how `start.dev.ts` / `start.external.ts` behave. 1. packages/node/pglite.dev.ts — PGLite as a STANDALONE process. PGLite is a real Postgres-wire server the node dials over TCP, not an in-process library, so `main.dev.ts` cannot simply absorb it. Until now the only way to start it was the orchestrator's `launchPglite()` helper, which means a container that wants the store had to run the orchestrator too. This calls `startPglite()` directly rather than spawning the CLI — the upstream module guards its CLI behind `import.meta.main` and exports the function — so the kernel container is two real processes (this and main.dev.ts), not a wrapper babysitting a child. It closes the store on SIGTERM so a normal `docker stop` is the graceful path. Port precedence matches `launchPglite()`: --port, then DB_PORT, then 5432. 2. config.dev.ts / main.grand-b.ts — honour MIDNIGHT_CONTRACT_ADDRESS. When the deploy is its own one-shot container, the address exists before the sync node starts and has to be injectable. `api.ts` and `config.preview.ts` already read this variable; `config.dev.ts` and `main.grand-b.ts` did not — they dereferenced `midnightContract!` with a non-null assertion, so a missing artifact surfaced as a bare TypeError that says nothing about the cause. Both now resolve env-first, file second, and throw a message naming both ways to fix it. The two files are kept byte-identical in this region on purpose: grand-b-config-drift.test.ts asserts they do not diverge, and it caught this change when only one side had been edited. Verified: node + validator + batcher + database suites 264/0 (the drift test among them); pglite.dev.ts starts, accepts a TCP connection on a chosen port, and exits 0 on SIGTERM.
It was added yesterday for exactly one consumer: the split kernel container, which ran an embedded PGLite beside the sync node because the orchestrator was the only way to start one. The demo stack now runs a single shared PostgreSQL for every component (demo-infra T11.4), so the kernel container selects the real server with PGLITE=false plus DB_HOST/DB_PORT/DB_USER/DB_PW/DB_NAME — all of which @effectstream/db already supported — and starts one process. Nothing in this repo referenced the file. Note that `PGLITE` defaults to TRUE in @effectstream/utils, so a container that wants the real server must say so explicitly; the demo's entrypoint refuses to boot otherwise rather than silently falling back to an embedded store. PGLite is still used here for local, non-Docker development (start.dev.ts / start.external.ts / start.mainnet.ts via the orchestrator's launchPglite) and as the in-process database for the packages/database tests. Neither runs in a demo container. Tests: node + database suites unchanged.
midnight-js 5's FetchZkConfigProvider verifies every fetched key/zkir artifact against compiler/contract-manifest.json by default, fetched from the same origin as the keys. The zk-asset routes only registered /keys/* and /zkir/*, so the browser contract client died with "no ZK artifact manifest ... was found" before its first proof, even though compactc had emitted the manifest into managed/compiler/. Adds `compiler` as a third served kind — same path containment, same rate-limit exemption.
The balancing adapter validates every merged transaction through the wallet facade, whose validation service runs wellFormed against a BLANK ledger state. A blank state cannot hold any contract's verifier keys, so a delegated transaction that CALLS a contract — the zswap-da frontend's faucet mint being the first live case — always dies with "call to non-existant contract", the exact failure class packages/validator already handles for contract-maker offers via ALLOW_CONTRACT_MAKER_OFFERS. Mirror that lane on the batcher's wallet: strict validation runs first, and ONLY the missing-contract failure widens to a wellFormed retry with verifyContractProofs=false (parameters still applied from blockData, all other strictness preserved; fail-closed when blockData is absent). Native zswap/dust proofs and signatures are still verified on the retry; the node verifies the contract-call proof at settlement. Opt-in via BATCHER_ALLOW_CONTRACT_TX=true, default off. Wrapped here in first-party code because @effectstream/batcher-sdk 0.200.x (and current v-next) hardcodes the three validation flags with no extension point — upstreaming the lane into batcher-sdk is the follow-up.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Points the v9 line back into #48: this branch is
feat/cow-solver@ 381022d merged onto00001-ledger-v9(the ledger-v9 migration branch of #49), with the solver packages migrated from v8 to v9.What the port contained (it turned out mostly mechanical):
@effectstream/*@0.200.2,@effectstream/mip-zswap-offer@0.4.0-v9.0(dist-tagledger-v9),@midnightntwrk/ledger-v9@1.0.0-rc.3with the single-WASM root override, midnight-js5.0.0-beta.6, wallet-sdk4.0.0-beta.2.@midnight-ntwrk/ledger-v8→@midnightntwrk/ledger-v9across 8 files inpackages/solver+packages/solver-core(the imported names are byte-compatible).keystore.signDatapassed tosignRecipe(whose signer must return a Promise) at two sites, both behindas anyso no typechecker had flagged them.ALLOW_CONTRACT_MAKER_OFFERS(contract-maker offers at ingestion) and the sync-health Celestia bearer fix (the tip probe sent no auth, so/v1/health/syncreportedsyncingforever against an auth-enabled Celestia node — which starves this solver's readiness gate).Verification: solver-core 159/0, solver 264/0, node 141/0 (564/0 on the port gate); no fixture regenerated, no safety default flipped; single physical ledger-v9 WASM copy confirmed.
Running today: the demo stack (
acedward/midnight-2-offers) runs this branch's solver in OBSERVATION mode against a live ledger-v9 chain — it mirrors the book overGET /v1/offers/updates, and publishes realprice-levelsframes over the relay-WS boundary (received by a demo sink; no Intents relay, no settlement — fills happen through the offer-files backend instead). First live datapoint for this branch's code against v9 wire bytes.Draft on purpose: #48 is itself a draft with a NO-GO-for-real-funds posture, which this branch inherits unchanged. Intended as the v9 continuation point, not a request to land over the in-flight review.