feat!: migrate the kernel to ledger-v9 / node 2.x (BREAKING) - #49
Open
acedward wants to merge 8 commits into
Open
feat!: migrate the kernel to ledger-v9 / node 2.x (BREAKING)#49acedward wants to merge 8 commits into
acedward wants to merge 8 commits into
Conversation
Moves the contract half of the ledger-v9 migration onto the node-2.x toolchain: compactc 0.33.0-rc.2 (language 0.25, runtime 0.18.0-rc.1, compact-js 2.5.5-rc.7). offer-files.compact compiles clean with no source edits -- the language-0.25 Field/Uint change does not bite, since the only arithmetic here is already annotated `as Uint<128>`. `compact compile +0.33.0-rc.2` is not an option: the compact version manager does not publish the 0.33 line (compact list tops out at 0.31.1). So infra/ fetches the pinned compactc release asset from LFDT-Minokawa/compact into a container, and infra/compact.sh runs it -- honouring $COMPACTC if a host install is available instead. We deliberately do NOT pass --feature-zkir-v3. compactc 0.33 defaults to ZKIR v2, and the emitted keys are tagged verifier-key[v6], which the ledger maps to ProofVersioned::V2 -- provable by the plain proof-server:9.0.0-rc.5. The _experimental proof-server build is needed only for the zkir-v3 lane. Adding that flag later means switching every deployment's proof server too. Worth noting: all twelve ZK artifacts are byte-identical to the compactc 0.31.0 output, so verifier keys are unchanged and this bump alone does not force a contract redeploy. Only the generated TypeScript moved. The rest of the migration -- renaming @midnight-ntwrk/ledger-v8 to @midnightntwrk/ledger-v9 across the workspace -- is NOT here, and cannot be until five @effectstream/* packages move off ledger-v8 upstream. Even packages/validator, which otherwise has no effectstream runtime coupling, would end up with two ledger wasm modules in one process because @effectstream/mip-zswap-offer hard-codes the v8 specifier in a value import, and two copies fail every cross-copy instanceof check. LEDGER-V9-MIGRATION.md records the full audit, the blocker, and the prepared v8->v9 API delta.
Moves the whole workspace off the ledger-v8 line onto the published
ledger-v9 set: @effectstream/*@0.200.1, @effectstream/mip-zswap-offer@
0.4.0-v9.0 (dist-tag ledger-v9), @midnightntwrk/ledger-v9@1.0.0-rc.3,
midnight-js 5.0.0-beta.6, wallet-sdk 2.0.0-beta.2 line, compact-runtime
0.18.0-rc.1 (compactc 0.33.0-rc.2 via infra/compact.sh, pinned earlier).
Runs against midnight-node 2.0.0-rc.4 / indexer 4.4.0-rc.1 (API v4
explicit) / proof-server 9.0.0-rc.5.
BREAKING: every package manifest changes scope/major for the Midnight
SDK line; the serialized transaction wire format moves from
transaction[v9](signature[v1],...) to transaction[v12](signature[v2],...),
so v8-era blobs and fixtures do not deserialize. The committed validator
fixture is regenerated against a live rc.4 stack.
Beyond the 31 import-site migration, the changes of substance:
* validator: v9's UtxoSpend.owner is a tagged SignatureVerifyingKey
({tag: 'schnorr'|'ecdsa', value}), not a bare hex string. derive.ts
passes the tagged object to addressFromKey (bare-string inputs are
wrapped as schnorr); shapes.testkit.ts builds tagged owners. A
rename-only migration compiles clean here and derives garbage.
* signer callbacks: the v2 SDK requires (data) => Promise<Signature>;
every unshieldedKeystore.signData callback moved to signDataAsync
(sync signData made Effect's evaluate().then() throw at sign time).
* e2e: new waitForWalletSettlement gate wherever one facade does a
second prove+submit — node rc.4 rejects stale-DUST reuse with
1010 Custom error 170 (MalformedError::InvalidDustSpendProof).
Deliberately NOT the full-quiescence gate the effectstream e2e uses:
this repo finalizes offer txs that are never submitted on-chain
(they live in Celestia until a taker settles), so pending.all and
shielded pendingCoins stay non-zero by design; the gate is
DUST-subtree quiescence + strict-complete progress.
* api tests: /v1/offers rows are MIP-0006 payloads carrying offerId
(the offer hash), not DB ids — resolve ids by hash while offers are
live; a byte-identical re-submit of an archived offer answers
409 DUPLICATE_OFFER (dedup probes offer_file_history first, by
design) so the negative test accepts that or 400 NULLIFIER_SPENT.
* external-stack mode: new start.external.ts + packages/node/
preflight-external.ts run ONLY the kernel's own processes (pglite,
contract deploy, mint, sync, batcher) against externally-operated
Midnight + Celestia infrastructure. The dev orchestrator cannot be
pointed at an external stack: launchMidnight is unconditional AND
declares stopProcessAtPort [9944, 8088, 6300], which would kill the
external stack's listeners. Celestia auth arrives by sourcing
/celestia/auth/celestia.env when present.
* indexer URLs move to explicit /api/v4 (v3 was an alias).
Verified: bun run test green — 62 tests, 0 failures, full Phase A + B
(deploy, mint, dual-chain sync, batcher settle, multi-token, unshielded-
only, negative paths) with zero error-170 submissions; packages/validator
93/93 including the crypto ladder against the regenerated fixture;
marker-dedup 8/8.
Two failures, both v9-migration fallout the e2e could not see:
* @effectstream/db 0.200.1's startPglite close() DESTROYS live sockets
(0.103.1 closed politely). The db test files deliberately never send a
client Terminate (PGlite WASM throws on it), so every file's teardown
now fired an unhandled 'error' on its still-open pg client — which bun
attributes to the NEXT file, cascading "Connection terminated
unexpectedly" across the whole run and eating ~20 tests' registration.
Fix: swallow the expected teardown 'error' on every client right after
connect() (13 files). CI's exact invocation now passes locally:
313 tests, 0 fail, 0 errors (was 286/4 with 14 errors).
* docs playground: midnight-js 5 moved FetchZkConfigProvider's custom
fetch from a positional argument into { fetchFunc } (and the v5
provider verifies artifacts against the compactc integrity manifest by
default). Same fix as the frontend template. docs tsc -b is clean.
@effectstream/*@0.200.2 (monorepo PR #882, "fix(db): make PGlite force-close opt-in") restores startPglite().close()'s graceful default: live client sockets are no longer destroyed unless force is requested. That was the regression fe61612 worked around by swallowing the expected teardown 'error' on all 13 db-backed test clients. Bump every @effectstream pin 0.200.1 -> 0.200.2 and DELETE all 13 swallow handlers — running without them is the verification. CI's exact invocation: 313 tests, 0 failures, 0 errors.
…FERS
An offer whose maker is a CONTRACT CALL (e.g. an AA-Manager open swap) can
never pass wellFormed against the blank reference state: contract-call proof
verification reads the contract's verifier keys FROM STATE, and a blank state
has no contracts - every such offer dies with PROOF_INVALID
('call to non-existant contract'), measured live 2026-08-26 with a proven
selector-6 open-swap offer from the demo stack (which then SETTLED on-chain,
ledger-exact, via the stock balanceFinalizedTransaction taker path - the offer
class is genuine and settleable; only ingestion refused it).
verifyOfferCrypto now takes contractMakerRetry (default off): strict wellFormed
runs FIRST, and only the exact missing-contract failure class widens into one
retry with verifyContractProofs=false - native zswap proofs and signatures stay
verified; the contract-call proof is verified by the node itself at settlement.
Threaded from ALLOW_CONTRACT_MAKER_OFFERS in the node (POST /v1/offers and the
STM ingestion ladder) and the batcher's pre-fee gate. Default remains
fail-closed: the relaxed lane admits ingestion-time spam of unprovable
contract-call offers, bounded by liveness checks, size caps and the limiter.
Tests: validator 93/0, node api+marker-dedup 25/0.
The DA write path authenticates; the health probe did not. Against an auth-enabled Celestia node the header.NetworkHead probe 401s, the tip reads null forever, and /v1/health/sync reports 'syncing' on a fully-caught-up kernel — starving any consumer that gates on currentness (measured live 2026-08-26 on the demo stack: a book-mirror client blocked at its readiness gate). Token comes from the same CELESTIA_AUTH_TOKEN the batcher config reads; absent (skip-auth deployments) the header is simply omitted.
…stack The demo stack runs the kernel through `bunx orchestrator start start.external.ts`, which puts six processes in one container and re-deploys the offer-files contract on every recreate. Splitting that into one container per concern (demo-infra Phase 11) needs two things from this repo; both are additive and neither changes how `start.dev.ts` / `start.external.ts` behave. 1. packages/node/pglite.dev.ts — PGLite as a STANDALONE process. PGLite is a real Postgres-wire server the node dials over TCP, not an in-process library, so `main.dev.ts` cannot simply absorb it. Until now the only way to start it was the orchestrator's `launchPglite()` helper, which means a container that wants the store had to run the orchestrator too. This calls `startPglite()` directly rather than spawning the CLI — the upstream module guards its CLI behind `import.meta.main` and exports the function — so the kernel container is two real processes (this and main.dev.ts), not a wrapper babysitting a child. It closes the store on SIGTERM so a normal `docker stop` is the graceful path. Port precedence matches `launchPglite()`: --port, then DB_PORT, then 5432. 2. config.dev.ts / main.grand-b.ts — honour MIDNIGHT_CONTRACT_ADDRESS. When the deploy is its own one-shot container, the address exists before the sync node starts and has to be injectable. `api.ts` and `config.preview.ts` already read this variable; `config.dev.ts` and `main.grand-b.ts` did not — they dereferenced `midnightContract!` with a non-null assertion, so a missing artifact surfaced as a bare TypeError that says nothing about the cause. Both now resolve env-first, file second, and throw a message naming both ways to fix it. The two files are kept byte-identical in this region on purpose: grand-b-config-drift.test.ts asserts they do not diverge, and it caught this change when only one side had been edited. Verified: node + validator + batcher + database suites 264/0 (the drift test among them); pglite.dev.ts starts, accepts a TCP connection on a chosen port, and exits 0 on SIGTERM.
It was added yesterday for exactly one consumer: the split kernel container, which ran an embedded PGLite beside the sync node because the orchestrator was the only way to start one. The demo stack now runs a single shared PostgreSQL for every component (demo-infra T11.4), so the kernel container selects the real server with PGLITE=false plus DB_HOST/DB_PORT/DB_USER/DB_PW/DB_NAME — all of which @effectstream/db already supported — and starts one process. Nothing in this repo referenced the file. Note that `PGLITE` defaults to TRUE in @effectstream/utils, so a container that wants the real server must say so explicitly; the demo's entrypoint refuses to boot otherwise rather than silently falling back to an embedded store. PGLite is still used here for local, non-Docker development (start.dev.ts / start.external.ts / start.mainnet.ts via the orchestrator's launchPglite) and as the in-process database for the packages/database tests. Neither runs in a demo container. Tests: node + database suites unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
BREAKING — migrates the whole workspace from the ledger-v8 line (
@midnight-ntwrk/ledger-v8@8.1.0, midnight-js 4.1.1, wallet-SDK v1,@effectstream/*@0.103.1) to the node-2.x / ledger-v9 line:@effectstream/*@0.200.1,@effectstream/mip-zswap-offer@0.4.0-v9.0(dist-tagledger-v9),@midnightntwrk/ledger-v9@1.0.0-rc.3, midnight-js5.0.0-beta.6, wallet-sdk2.0.0-beta.2, compact-runtime0.18.0-rc.1/ compactc0.33.0-rc.2. Targets midnight-node2.0.0-rc.4, indexer4.4.0-rc.1(explicit/api/v4), proof-server9.0.0-rc.5.Why breaking
@midnightntwrk, no hyphen).transaction[v9](signature[v1],…)→transaction[v12](signature[v2],…): v8-era blobs and fixtures no longer deserialize. The committed validator fixture is regenerated against a live rc.4 stack.{tag: 'schnorr'|'ecdsa', value});UtxoSpend.ownerand the signer-callback contract changed shape.Changes of substance (beyond the 31 import-site rename)
derive.tspasses the taggedSignatureVerifyingKeytoaddressFromKey(a rename-only migration compiles clean and derives garbage);shapes.testkit.tsbuilds tagged owners.(data) => Promise<Signature>; everyunshieldedKeystore.signDatacallback moved tosignDataAsync.waitForWalletSettlementwherever one facade does a second prove+submit: rc.4 rejects stale-DUST reuse with1010 Custom error: 170(MalformedError::InvalidDustSpendProof). The gate is deliberately DUST-subtree-only: this repo finalizes offer transactions that are never submitted on-chain (they live in Celestia until a taker settles), so a full-quiescence gate deadlocks./v1/offersrows carryofferId(the offer hash), not DB ids; ids resolved by hash while offers are live. A byte-identical re-submit of an archived offer answers409 DUPLICATE_OFFER(dedup probes history first, by design), accepted alongside400 NULLIFIER_SPENT.start.external.ts+packages/node/preflight-external.tsrun only the kernel's own processes against externally-operated Midnight + Celestia infrastructure (e.g. theacedward/midnight-2-offerscompose stack). The dev orchestrator cannot do this:launchMidnightis unconditional and declaresstopProcessAtPort [9944, 8088, 6300], which kills the external stack's listeners. Celestia auth arrives by sourcing/celestia/auth/celestia.env.Verification
bun run test: 62 tests, 0 failures — full Phase A + B (deploy, mint, dual-chain sync, batcher settle, multi-token, unshielded-only, negative paths), zero error-170 submissions.bun test packages/validator/: 93/93, including the crypto ladder against the regenerated fixture.ledger-v8in the store; exactly oneledger-v9@1.0.0-rc.3and oneonchain-runtime-v4@4.0.0-rc.3(single-WASM rule).