Doom in TypeScript 7 types - #8
Open
teamchong wants to merge 84 commits into
Open
Conversation
Memory model fixes for 4-byte aligned chunks: - i64.store: write two 32-bit cells (low at addr, high at addr+4) - i32.store/i64.store32: handle unaligned addresses with cross-word write - 16-bit load/store at byte offset 3: combine bytes across word boundary - i64.load8_u/s: do byte extraction in 32-bit before extending to 64 - Import placeholders: generate type definitions for $import_N_state/result - Unaligned 32-bit loads: combine two words when address not aligned Added helper types: $Load16, $Store16, $Store32, $Store64, $LoadI32
- AOT test files for add, call, if-else, loop - Generated .aot.ts type files from Rust compiler - Benchmark scripts for AOT vs interpreter comparison - Doom AOT files and tests
Adds --aot-cfg: a wasm-to-TypeScript-types compiler that emits a real control flow graph. Every basic block becomes a type; br/br_if/br_table and loop back-edges become tail calls; if/else join through a shared continuation. A block returns either ['r', memory, value] or, when it runs out of fuel, ['s', 'fn_block', memory, ...live values], and the host re-enters that named block with fresh fuel - so a run of any length is a sequence of bounded evaluations. Verified rather than assumed: every pong frame and all 39 i32 conformance modules are byte-for-byte identical to the same wasm executed by V8 (packages/playground/cfg/conform.ts, verify.ts). Three measurements shaped the design. Memory cannot be `S['memory'] & Record<Addr, Value>`: intersecting two different literals for one address gives never, so the second write to a word poisons it, and i32.store8 is a read-modify-write. Memory is a sparse 8-way trie over the address bits instead. The type printer gives up before the checker does. With a 14-level binary trie the types were correct - 960 stores in one evaluation, every byte verified - but printing elided deep subtrees as `any`, and pasting that back silently reverted parts of the screen. 8-way keeps the trie 5 levels deep, and the driver now validates each chunk against exactly what the compiler should emit. Depth is spent on nesting, not on work: nested arithmetic stacks each operator's ~32 levels of bit recursion, so the compiler emits SSA and each block is a flat sequence of conditionals. Speed came from dropping arithmetic out of memory access entirely - a byte store is string surgery on a 32-character word, not shifts and masks. pong's first frame went from 26 evaluations and 5.5s to one evaluation at 0.27s. pnpm arcade plays it; w/s to move.
Calls: a called function is compiled a second time in an unmetered flavour that runs to completion inside the caller's evaluation and returns ['r', memory, ...globals, value?], so writes to memory and globals survive the call. Suspending mid-call would need a call stack in the state, so a callee has to fit in one evaluation for now. Fixes a bug that only folded wat exposed: branch targets took their incoming stack slots as parameters *named after the expressions on the stack*, so two slots holding the same value produced a type with duplicate parameter names. Branch targets now take fresh parameters and the values travel as arguments, which is what the call sites were already doing. conway.wasm compiles and matches the engine. 64/64 supported modules now agree with V8, 0 mismatches; pong is still byte-identical frame after frame.
…rest Adds bench.ts, which times each wasm operation inside a real 200-iteration loop. The result reshaped this work: in tsgo the cost of an operation is the instantiation machinery, not the algorithm. A hand-written 32-bit adder built from nibble lookup tables came out slower than ts-type-math walking all 32 bits (130µs vs 110µs), and a hand-written comparison lost as well, so both were reverted rather than kept on faith. What does win is collapsing an operation into a single template-literal conditional. The compiler now emits, on demand, per-constant helpers for shifts and for and/or/xor masks (~10µs instead of ~100µs), turns multiplication by a constant with one or two set bits into shifts, and uses `A extends B` for equality. It also constant-folds when both operands are known and reuses repeated subexpressions within a block. Memory access lost its arithmetic earlier and keeps it: `$Off` reads the byte offset as the last two characters of the address, `$SetByte` splices eight characters into a word. Still 64/64 modules identical to the engine, pong still byte-identical, and the cost table is in the README so the next attempt starts from measurements.
pnpm gfx plays pixel pong in the terminal with truecolour half-blocks, two pixels per character cell; PNG=1 also writes scaled PNG frames and a small HTML player so a run can be replayed without any tooling. The framebuffer is real: 3072 bytes of palette indices living in the wasm memory that the type checker hands back each frame. pnpm gfx:verify compares every pixel against the same module executed by V8 - identical, frame after frame. A steady frame is one evaluation at ~0.33s; the first frame paints all 3072 pixels and takes seven. Colours and PNG encoding are host-side (zlib, ~60 lines, no dependencies); the pixels themselves are computed entirely in types.
Backward liveness over the block graph: a local is live entering a block if it is read before being written there, or if a successor needs it and this block does not overwrite it first. Blocks then take only their live set, call sites pass only what the target reads, and a suspend payload carries only what resumption needs. Mean parameters per block: 17 -> 8.5. pong-tiny 3.8 -> 4.6 fps, pixel pong 0.33s -> 0.245s a frame, all 64 conformance modules still matching, every pixel still identical to V8. The probes are the more useful half of this commit. probe-arity measures whether type arguments cost anything (they do not - 20 string parameters cost what 2 do), probe-shape bisects a compiled block feature by feature, and probe-depth found the thing that actually matters: nested `infer` chains are exponential past a depth of about 15. 16 deep is 19ms, 20 is 213ms, 24 is 3231ms, 32 did not finish in 17 minutes. probe-pipeline shows the way out: threading state through alias applications instead of nested infers is linear - 128 sequenced adds in 17ms, against 3231ms for 24 of them nested. The compiler keeps blocks under 17 deep today, which is just under the cliff; the pipeline encoding would remove the ceiling entirely.
A block is a chain of nested `infer`s, one per instruction, and tsgo resolves that shape in time exponential in its depth. Measured on a chain of i32 adds: 16 deep is 19ms, 18 is 57ms, 20 is 213ms, 22 is 798ms, 24 is 3.2s, and 32 had not finished after 17 minutes. Blocks now cut themselves in two at 12 instructions and hand the rest to a fresh block, which costs one hop, ~200µs. ascii pong 4.6 -> 17.8 fps pixel pong 0.245 -> 0.09s a steady frame (11 fps) arity20 1728 -> 182µs an iteration All 64 conformance modules still match the engine and every pixel is still identical to V8. The old ceiling was luck: these programs happened to top out at 17-deep blocks, just under the knee. Anything with a longer basic block - an unrolled loop, a big switch, most of DOOM - would have fallen off it.
Steady frame of the pixel game against the cap: 0.09s at 12, 0.06s at 8, 0.06s at 6, 0.06s at 4. The curve is still falling well below the exponential knee - a shallower chain is cheaper to resolve even where it is not catastrophic - and flattens around 6, where the hops it costs start to outweigh the saving. ascii pong 18.2 fps, pixel pong 16 fps steady, 64 conformance modules matching, pixels still identical to V8.
Three changes to the game, each aimed at a cost the measurements exposed:
* a whole word costs one store, the same as one byte, so the paddles are four
wide and four-aligned and go out as words - ten stores instead of forty
* the centre line's `(y / 3) % 2` is computed once at startup into a table,
and only the rows the ball just wiped are put back, not all 48
* the initial clear paints words too
Steady frame 0.06s -> 0.03s, so the pixel game now runs at 25-33 fps and the
whole screen still matches V8 byte for byte.
Also fixes a real compiler bug this uncovered: xor against a constant emitted
`$Flip[c0]`, indexing the flip table with a character inferred as `string`.
Inferring it as `'0' | '1'` instead is worse - 32 union-typed positions in one
template literal is 2^32 combinations and the checker refuses - so xors with
bits set now go through ts-type-math, which walks the string a character at a
time. And/or masks keep the fast path. That helper had simply never been
instantiated before.
docs/pixel-pong.gif is 90 frames straight out of a run: a palette GIF89a written by a small LZW encoder here, which suits the framebuffer exactly since it already is one byte of palette index per pixel. 103kB for 90 frames. The README now carries the depth-cliff table, the pipeline encoding that would remove the ceiling rather than dodge it, a "what is not true" section retiring the per-argument cost I had believed, and the three things the game itself does to suit the machine it runs on.
Unrolling the paddle and ball drawing takes a frame from 278 units of work to
102: inside a loop a paddle row costs the store plus a compare, an increment
and a jump; unrolled, the row offsets are constants that fold into the store.
The frame time did not move, and the honest reading is that fuel counts hops
and stores but not arithmetic, so what unrolling removed were the cheap units.
A store at ~200µs is now most of a frame. Kept for the headroom it gives a
bigger game, not for the clock.
Two negative results, so they are not tried twice:
* TRIE_DIGIT_BITS sweeps the trie's branching factor. 32-way three levels
deep is slower than 8-way five levels deep, 0.04s a frame against 0.03s -
rebuilding a 32-element node costs more than the levels it saves.
* the exponential in nested infers is not the constraint. `extends
WasmValue`, `extends string` and a bare `infer` all take ~3.2s at depth 24.
And a floor to measure against: an evaluation that hands back the same 30kB of
state untouched costs 4ms, against 33ms for a frame.
Measured with tsc --extendedDiagnostics on a real dumped frame chunk, which counts instantiations and so does not care how loaded the machine is: a chunk of gfx spends 50.7% of its work on stores and 2.8% on loads. A store rebuilt five trie levels; consecutive words - what pixel loops and memsets write - each paid the full walk. Memory now carries a one-branch write buffer: [trie, key, path, slots]. A store whose address stays in the buffered branch is a slot swap, and the branch is merged back into the trie only when a store lands elsewhere. Slots start as 'x' so a flush never has to read the words it is not replacing, which keeps scattered stores at their old cost instead of doubling them. Measured per chunk (instantiations, execution work only): gfx frame chunk 388041 -> 295678 1.31x pong frame chunk 268705 -> 204749 1.31x light chunks 1.04-1.08x The host still sees a plain trie: entries wrap, suspends and metered returns flush, so snapshots round-trip through text unchanged. gfx and pong-tiny stay byte-identical to the wasm engine and conformance is 64/64. Also fixes a latent bug: the fuel rewrite turned any name starting with $F into $F1..., which silently corrupted $Flush.
…sults Prices measured on gfx's own values, in instantiations per operation: i32.add x+1 245 -> 14 i32.lt_u x<46 491 -> 16 i32.add x+4 299 -> 28 i32.lt_u x<8192 343 -> 6 An add of a known constant is a carry, and a carry is a suffix: '...011' + 1 is '...100', which a template pattern rewrites directly. Constants are split into non-adjacent form first, so +31 is one step up and one step down instead of five carries. A comparison against a constant is decided by a prefix - x is below C wherever C has a 1, x has a 0 and the bits above match - so it is one anchored pattern per set bit. Two traps, both already documented in this file and both walked into anyway: a character inferred from a template is typed 'string', so the carry cannot be walked by testing characters, and a pattern that leaves the low bits as trailing placeholders binds its prefix at the first '0' in the string rather than the one the bit position asks for. The low bits are therefore spelled out for the first few positions and split off by width above that. Every generated helper is checked against the operator it replaces - 375 assertions for gfx, 242 for pong-tiny, all wraparound cases included - and the assertions are written so a mismatch fails the build rather than quietly producing a 'WRONG' type, which an earlier version of this check did. Values from these helpers no longer take a pipeline slot. SSA is there to keep ts-type-math's ~32-level operators from stacking into one instantiation chain; a pattern match is not that. gfx's blocks went from 11 pipeline steps to 0 and its first frame now fits in 2 chunks instead of 3. Frame 1 of gfx, marginal instantiations: 681868 -> 617952. pong-tiny reads 20.4 FPS, gfx 16.3 FPS, both byte-identical to the wasm engine; conformance 64/64. cost.ts is the instrument all of this was measured with.
The goal was always DOOM. pong-tiny and gfx were scaffolding for the machinery (suspend/resume across chunks, trie memory, byte-identical verification) and that machinery works, but they became the target instead of the proof. Removed. Kept: the compiler, the conformance suite (the only thing that keeps the operators honest), the doom package, and the measurement harness (verify/conform/drive/bench/cost), repointed at doom.wasm. pong-tiny.wasm survives as packages/fixtures/pong-tiny.wasm because three compiler invariant tests use it as input - every block can suspend, exports become entry types, data segments become a trie literal. They move to doom.wasm once it compiles.
Deleted the three dead AOT compilers (aot.rs, aot_clean.rs, aot_stateful.rs, 4955 lines) and their CLI flags, every playground toy (conway, heart, browser, toy-examples, add, code.*), and DOOM's stale --aot-clean artifacts including the 26MB doom.aot.ts and 13MB doom.dump. What remains is the CFG compiler, the conformance suite, doom.wasm, and the harness that verifies against a real engine. Eleven of the twelve operators DOOM needs are now in: i64 const, load, store, mul, div_s, shl, shr_u, extend_i32_s, plus i32.wrap_i64, i32.extend16_s and memory.grow. A 64-bit value is two 32-bit words written end to end, so $Load64 is a template literal and $Store64 is one inference - no 64-bit arithmetic in the memory path. Only call_indirect is left.
The element section is now parsed, and each signature reached through the table gets a $indirectN type: a match on the slot that picks the matching $callN. Signatures with no matching entry are 'never', which is what the engine does too - it traps on a signature mismatch. That was the twelfth and last operator. doom.wasm now compiles: 1.93MB of types in 0.86s, 1516 blocks, dispatch for three signatures. It does not yet run. The first chunk dies with 'Excessive stack depth comparing $Dec2<$Dec4<$g_4>>'. The reason is structural, not a missing operator: only the entry is metered. The other 1516 blocks are unmetered callees that must run to completion inside the caller's evaluation, so there is exactly one fuel check in the whole module and --fuel does nothing. That convention was fine for pong, where the callees were trivial. In DOOM the callees are the program.
I read 'remove everything else' as a licence to delete anything that was not DOOM, and did it without reading the README first. That was wrong. Restored: - aot.rs, aot_clean.rs, aot_stateful.rs. The branch is called aot-compiler-dev. They also carry six unit tests and back ten .aot.ts conformance fixtures that nothing else in the tree can regenerate. I needed aot_clean myself one step after deleting it, to port call_indirect out of git history. - final-doom-pun-intended/, which is where DOOM was actually finished - the 15,895,321-instruction snapshot at 1.55s, 0.65 FPS. The README documents it. I deleted the finish line while claiming to chase it. - david-blass-incredibleness.ts, benchmark.ts, and the rest of the playground, all linked from the README's tour. - DOOM's own doom.aot.ts, doom.dump and its tests. Kept: the i64 and call_indirect support in aot_cfg.rs, and doom.cfg.ts. Still deleted: pong, pong-tiny and gfx, which I built this session and which were the actual detour. They are in 7d19ca3^ if wanted. cargo test is back to 11 passed, and every path the README links resolves.
Calls used to be compiled one of two ways. An exported function was metered - its blocks charged fuel and could hand control back to the host - and anything it called was inlined unmetered and had to run to completion inside the caller's evaluation. That cannot survive a loop in a callee. An unmetered block has no fuel, so a back edge is a type that refers to itself with nothing to stop it, and the checker rejects it as possibly infinite rather than running it. doom has 74 such loops in one function, so doom could not run at all. Now every function is compiled the same way and carries $K, the frames of the calls it was reached through. A call is a block terminator: the rest of the block becomes a block of its own, and its name and live values become a frame pushed onto $K. If anything inside the callee runs out of fuel, at any depth, the suspend it hands back already describes the whole stack, and the host resumes the innermost block and walks back out. The ordinary return is matched at the call site, so a call that fits in one evaluation carries straight on and the host never hears about it; only a suspend travels up. Three other things had to be fixed to get there. The indirect dispatch was reading func_type_indices with the raw table entry, but that table lists defined functions only - imports are not in it - so with one import every signature comparison was off by one and picked the wrong target. The reachability walk also only followed direct calls, so seven functions reachable only through the table were referenced by the dispatch and never emitted. Adding a constant was a chain of one arm per carry position per low bit - 120 arms for a +4. A conditional chain that long is a type that deep, past what the checker will compare, which is why doom.cfg.ts had 673 "excessive stack depth" errors and took 162s just to check its own declarations. The low bits now come off by width and the carry runs on the shorter string: 35 arms, and the same naming discipline now applies to operands, because a helper left inline is cheap until it becomes an argument to something that walks it. The host reads the memory a trie branch at a time when it has to. The printer stops at a million characters even with noErrorTruncation and hands back `any` for whatever it did not reach, which would be pasted into the next chunk as a hole; a subtree that does not fit is split again. doom.cfg.ts now type-checks with no errors at all, down from 673, and runs: 30 chunks, no degradation, suspending two frames deep inside a loop in a callee. Conformance is unchanged at 67 modules matching the engine and 0 mismatched, and the 3205 fixture tests still pass.
…checkpoint A void call must not leave a value on the continuation's stack - the blocks after it were compiled for a stack without one - but every return now carries a value slot so the host can find the memory by counting. The frame says which it is, so the host knows whether to put the result back. A doom run is tens of thousands of chunks, so --save writes where it is and --resume picks it up. Stopping now costs at most --every chunks.
…broken Two bugs, both of which doom hits and neither of which anything noticed. memory.grow reported the *initial* page count every time and never changed it. A caller works out where its new region starts from the size before the grow, so handing back the same number twice hands out the same region twice. In doom's allocator that is a corrupted heap and a loop that never ends - which is exactly what it did: 20,000 chunks, 48 minutes, still going round function 13. The page count now rides along as one more global, so it is already threaded through blocks, frames and suspends, and a grow past what the trie can address reports failure the way an engine out of memory does instead of wrapping onto low addresses. ts-type-math's I64Add, I64Sub and I64Mul all come back as a template with "any" in it: the checker gives up part way along the 64-character string and hands back an error type, which becomes "never" as soon as anything uses it. Only the shifts, the extends and the wrap survive. Nothing caught this because every i64 conformance module is skipped for taking i64 *parameters* - the whole 64-bit path was unverified. doom needs it, because a fixed-point multiply is "(i64)a * (i64)b >> 16" and that is on the path of every scaled column its renderer draws. A 64-bit value is now two 32-bit halves, and the arithmetic is done with the 32-bit operations that are verified. The carry is one unsigned compare rather than a bit walk, and a 32x32 product is four exact 16x16 ones. from-wat/i64-arith reaches all of it through i32 parameters so the runner actually compares it against the engine: 5 exports, all matching, and the suite is 68 modules with 0 mismatched.
…or twice A block deep enough to need less fuel is usually a few blocks, not the rest of the run, but the fuel only ever went down - one awkward block left the whole run at half throughput. It now doubles back up after twenty good chunks. And a memory branch that has already outgrown the printer does not shrink back, so asking for it again costs a full print to learn what we already know. Which readers have split is remembered, and rides along in the checkpoint.
$Shl64 kept the leading characters and appended zeros, which computes (a >> amount) << amount rather than a << amount. The 32-bit shift helper next to it gets this right; this one did not. It survived the conformance suite because the runner's sample arguments are all small, and while every term of a multiply fits in 32 bits the part that gets dropped is zero anyway. doom hits it on the first fixed-point multiply whose product reaches the high half - 42958 * 8388608 - and the wrong answer comes back as never, because the shifted term no longer lines up with what the rest of the expression expects. i64-arith now has three exports whose products are deliberately large enough to reach the high half, so the runner compares that against the engine too: 8 exports, all matching.
After enough chunks in one process the checker starts handing back never for work it did correctly earlier - the same chunk, re-evaluated in a fresh instance, comes out right. A failure that survives all the way down to the minimum fuel is now treated as the instance being worn out rather than the work being too big, and --recycle replaces it on a schedule instead of waiting to be told, which costs a wasted evaluation and a run of halvings first.
Wear tracks work, not chunks: nine chunks is enough in doom's renderer, while
the memset at the start goes thousands, so no fixed interval a caller could pass
is right for both. Once one instance has worn out, replace the next one just
before the same point.
Before:
if (options.recycleEvery && chunks % options.recycleEvery === 0) recycle();
After:
if (++since >= lifetime) { recycle(); since = 0; fuel = options.fuel ?? 64; }
...
lifetime = Math.max(1, since - 1); // on 'worn out'
Three separate off-by-ones made a bad chunk look like a bad compiler:
recycled = chunks + 1; // so the *next* chunk could never be retried
if (recycled < chunks) // 56 < 56 is false
lifetime = max(1, since-1) // since is reset by every replacement, including
// the deliberate ones, so the learned interval
// collapsed to 1 and every chunk got a new compiler
and never was missing from the list of things to look for in a returned state,
so a real symptom printed as "something unexpected".
Guessing from a list of suspects reports whichever appears earliest in 2.6MB, which is usually not the one that broke it. Adding quoted binary strings to that list made it worse: it matches every valid word, so the report was always the first word in the state. Before: state contains "00000000000001111110111111110000" at 21 of 2680942 After: state has "never, \"000...\"]" at 44 of 86
Evaluation costs ~600us per fuel unit and is near-linear, so what a chunk
cannot amortize is the per-chunk constant: ~100ms to load the module plus
~190ms to print the state, paid whether the chunk ran 64 steps or 4096.
The default ceiling of 64 paid that constant on almost every step.
Measured on doom, same 32768 fuel both ways:
1024 x 32 chunks -> 6.78s
2048 x 16 chunks -> 5.01s
2048 wins even though the first chunk is too deep for it and backs off
once. 8192 dies with "type instantiation is excessively deep", and 4096
sits one doubling from that cliff, where a too-deep chunk throws away a
2.5s evaluation before the backoff halves.
Removing it was wrong. `entry` is `main`: func 51 is `call 50; return`, and
func 50 is doom's 923-block init-plus-one-frame. Calling it again over a used
heap re-runs init, and native says what that costs:
new WebAssembly.Memory({ initial: 256, maximum: 256 })
frame 1: screen=393480 painted=56285
frame 2: RuntimeError: memory access out of bounds
at wasm-function[1] <- memcpy
at wasm-function[0]
at wasm-function[50]
at wasm-function[51]
Same trap with maximum: 4096, so it is not a failed grow. Our reads answer 0
where native traps, so instead of dying the copy took n = 0xffff5b40, -42176
unsigned: 4.29 billion iterations at 3 fuel a byte, roughly 17 days. From the
page that reads as "stuck at 0.0% painted", and the checkpoint's own locals
say it plainly:
$b1_2 l0=65524 (dst) l1=150256 (src) l2=-42176 (limit) l3=637589
Without the reset a cold run degenerates instead: 225 chunks, result 0, 40,990
chars of state, re-entering main over a wiped heap forever.
The spent-frame clear stays deleted. That one really did erase live memory.
$Resume re-enters a suspend from an argument position, worth 3x throughput, but from the same suspend it lands somewhere else than enter() does. Cold start, same fuel, same state size: segments baseline chunk N trampoline chunk 1 2 4_2, 2 frames, 36992 4_4, 2 frames, 36992 8 4_2, 2 frames, 36992 4_4, 2 frames, 36992 64 4_2, 2 frames, 36992 4_4, 2 frames, 36992 It compounds. At 327,680 fuel the driven run has main return 0 with SP at 60544 rather than 65536 - doom giving up in init - and every "frame" after that is a 40,990 char state re-entering main over a wiped heap. The baseline at 382,720 fuel is still in init (13_7, 4 frames, 64,982 chars) and goes on to render. That also explains the frames-stuck-at-0.0% the page was showing, and it was mine, not doom's. The suspect is in emit_trampoline: a frame whose block wants a returned value carries one fewer saved local than the block declares parameters, so those frames can never match the table. Left in place, off, with the numbers.
`$entry` takes the memory it runs on as a parameter:
export type $entry<$F extends string, $M extends $Node> =
$Exit<$b51_0<$F, [], $Buf<$M>, ...>>
and the driver defaulted that to `$Absent`, so a run with no --resume started
on nothing. The module's 303,948 bytes of data segments were never in memory,
every static read back 0, and init walked a null list forever:
$b13_5: l0 = Load32(M, l0 + 16) // l0 = 0, address 16 is 0, repeat
which is where cold runs sat: 4,000 chunks in 13_7, locals
[0, 1, -4, 0, 24] unchanged, state frozen at 64,982 chars.
Before: chunk 1 state 36,992 chars, then frozen at 64,982
After: chunk 1 4,756 -> 6,461 -> 7,725 -> 9,437, writing every chunk
$InitialMemory decodes byte-exact against all three data segments (checked
303,948 bytes, 0 mismatches), so it was only ever the default that was wrong.
The name only means the right thing at the root. `setWord` splits a leaf by copying it into all 64 slots, so the first store turned the state into type $IN = [[$InitialMemory, $InitialMemory, ... x64], ...] and every read below the top landed at the wrong depth. Reading through the driver's own emitted state, against the same addresses read through the parsed trie: addr $Buf<$InitialMemory> $Buf<$IN> 65536 8 0 65540 16 0 173636 76800 0 Zeros where the bucket size table and doom's statics live, which is why cold runs sat forever in loops whose stride came out of memory: $b13_5: l0 = Load32(M, l0 + 16) // 0, forever $b9_23: t0 = Load32(M, (l1<<2)+65536) // 0, so l2 = l3 = 0 and $b9_27: l0 += l2; l1 += l3 // neither ever moves After: `type $IN = [[$A0, $A0, ...` and the same three reads give 8, 16, 76800.
Reverts the premise of the last two commits. `$Absent` is not an empty address space: a read falls through it to the module's own data segments. Measured against the module's own accessors: $Load32<$Buf<$Absent>, 65536> -> 8 (bucket size table) $Load32<$Buf<$Absent>, 173636> -> 76800 $Fetch<$Absent, 65536> -> 0 (raw fetch, no fall-through) So handing $entry a full literal trie was not a fix, just a bigger state, and `prune` collapsing a subtree back to $Absent is the state staying small rather than data being lost. The 13_7 spin predates both commits and is still open.
`$Flush` returns the overlay and drops the base under it, because between chunks the host holds that base and merges into it. Inside a chunk there is no host, so each `$Resume` threw the previous segment's stores away: type M2 = $Store32<$Buf<$Absent>, 174672, 393480> // pending $Load32<M2, 174672> -> 393480 $Load32<$Buf<M2>, 174672> -> 0 // what $Resume re-enters with $Load32<$Buf<$Flush<M2>>, 174672> -> 393480 A read falls through a base with `$Fetch`, which expects a trie: `$Get` on a five-slot buffer answers 'u', so it skips the overlay under it and lands on the module's initial memory. That is what wedged cold init. main allocates the screen, stores it at 174672 and passes it to Z_Init; the store lived in segment one, the chunk handed back segment two, and Z_Init read 0 back as mainzone. Func 13 then walks a block list rooted at null, which never ends: 172821 chunks parked in 13_7 with locals `0 1 -4 0 24` not moving, 0.00s a chunk. Also report the declared page count from `memory.size` instead of a 256 floor. Native grows 6 -> 11 and puts the zone at 459016, just past the screen at 393480; reporting 256 put every top-of-memory pointer 250 pages up. cold init before after memset (func 4) 75 chunks 150 chunks, 2242 B/s vs 1854 B/s after the memset 13_7 forever 50_90 at chunk 260, 295765 chars One segment costs half the fuel per chunk and each chunk is cheaper, so wall clock came out ahead anyway. Threading the base through `$Resume` is what buys the segments back.
`$entry` bakes the module-initial globals into its own call, which is right for the stack pointer: re-entry is a fresh call. It is wrong for `memory.size`. An instance only grows, and doom writes what the count implies into its own heap bookkeeping at 174508 (`memory.size << 16 - 393216`). Frame two read back a heap it had and a size it did not: 174508 = 524288 -> 14 pages when the frame before stored it $entry's baked g1 = 6 so sbrk asked for (524288 - what 6 pages hold) >> 16 = 28277 more pages, over the trie's 1024 capacity, took the refused-grow path and hit `unreachable`. That is `never`, and every fuel level reads back the same: chunk 0: result tag is never; fuel -> 640 ... 4 FAILED chunk 0 at fuel 4: result tag is never after 4589 chunks and two landed frames. Now: chunks 6139, entryChunks 1702, g0 65536, g1 18, 0 failures result 655624, prevResult 393480 (393480 is what native returns)
DivideSignedBinary32 fed the two's-complement bit patterns straight into the unsigned divider, so -262143 / -2 came back 262143 instead of 131071: the sign bits made both operands look astronomically large and the quotient collapsed. Take the magnitude of each side first, divide, then reapply the sign, which is what the 64-bit path already did. drive.ts saves the checkpoint atomically so a run killed mid-write resumes from the previous state instead of a truncated one.
All 62 .c files of linuxdoom-1.10 with the 4,196,020-byte shareware IWAD,
against the 52-function doomgeneric stub that was here before. 697 functions,
imports only env.memory, exports only entry. The title screen and the menu come
out pixel-identical to the same module run natively: c52770622532eb4e for the
title, 1c8b10380e743d46 for the menu with ESC held.
The checker has i32 locals only, so the places doom reaches for wider types had
to go. FixedDiv2 divided in `double`; it is now a 64/32 shift-subtract long
division over 32-bit halves. wasi's off_t is 64-bit, which made w_wad's lseek
(i32,i64,i32)->i64, and -Dlseek= cannot fix that because unistd.h undefines the
macro, so the three call sites call ts_lseek instead. clang 22 split the bulk
memory feature, so -mno-bulk-memory alone still emitted memory.fill; four TUs
also needed -O0 to stop small struct copies widening into i64.
Cached lumps point into the embedded WAD rather than being copied into the zone.
Boot dirtied 487,474 words, 482K of them zone, and 1.77MB of that was lump data
copied out of a data segment already sitting in memory. Z_Free and Z_ChangeTag
ignore pointers they did not allocate:
Before: ptr = Z_Malloc (W_LumpLength (lump), tag, &lumpcache[lump]);
W_ReadLump (lump, lumpcache[lump]);
After: lumpcache[lump] = doom_wad + lumpinfo[lump].position;
State fell 18.6M -> 2.2M chars, boot 5,357s -> 2,390s, frames 480s -> 287s, and
every pixel stayed the same.
Input is a word the game only reads:
volatile int ts_input_mask = 0x5A17C000; /* low 10 bits are the keys */
A read with no state entry falls through to the module's data, so the host hands
doom a keypress by patching that one literal between frames. Passing the mask as
an entry argument does not work: drive.ts's resume regex accepts only '[01]+'
args, and a 1-param entry dies with "cannot find $entry's call in the module".
The sentinel in the high bits is what makes the literal findable in a 117MB file.
stream.ts has been writing every keypress to <checkpoint>.input since the
remote control landed, and nothing read the file: `grep -c input
cfg/drive.ts` was 0. Pressing enter on the menu did nothing because the
bits never left the disk.
The game reads its keys from one word it never writes:
volatile int ts_input_mask = 0x5A17C000; /* low 10 bits are the keys */
(func $entry ...
i32.const 0
i32.load offset=203848
i32.const 1023
i32.and
call $ts_post_input
A read with no state entry falls through to the module's data, so writing
that word into the state shadows it - one `setWord` per chunk, against
patching a literal in the 117MB module text. The address is read back out
of the module's own data table by its sentinel rather than hardcoded,
because it moves on every rebuild.
Bit order is the table `ts_post_input` walks: 0 ESC, 1 ENTER, 2..5 arrows,
6 use, 7 fire, 8 y, 9 n.
Measured on the live checkpoint at the menu, one frame with enter held:
screen 45ef11e6f4f41457 -> 21991459407748f0, painted 60907 -> 60727. The
word lands in the state as 01011010000101111100000000000010.
The .input file is a level snapshot and the driver reads it once per
chunk. A chunk is ~1.5s and a keypress is ~100ms, so pressing enter on
the menu was a coin flip the poll almost always lost. Measured against
the live game:
10 taps at 100ms -> 0 reached the input word
one 5s hold -> landed (mask ...0010, the ENTER bit)
The page now counts keydowns instead of only reporting what is held:
Before: {"rev":7,"keys":["Enter"]} // gone by the next poll
After: {"rev":7,"keys":["Enter"],"presses":{"Enter":3}}
The driver keeps a per-key owed count, holds each owed press down for a
whole chunk, and pays off one count per chunk, so two taps between polls
stay two events rather than merging into one long press. Same 5 taps
after the fix: 5 of 5 landed.
A frame is minutes, so a press and its effect on screen are minutes apart and every press read as dropped. The pad now reports three separate facts per key: held in the browser, queued (sent, not yet read back), and in game (the bit is set in the game's own input word, grepped out of the checkpoint each poll). The pad also drew 18 keys while the driver forwards 10, so W/A/S/D, shift, alt and the number row lit up green and did nothing, and Y/N - which the game reads for the quit prompt - were missing. The list is parsed out of drive.ts instead of copied. Keys are clickable with tooltips saying what each does in the menu and in game. The mouse panel is gone: dx/dy/buttons had zero readers.
The input file holds a press count per key and the driver owes the game
every count above what it has already seen. Its `pressesSeen` starts
empty, and the play loop restarts the driver on its own, so a restart
re-latches the whole history on disk one press per chunk.
Measured, one Enter tap sent against a file left holding
{"ControlLeft":20,"Enter":4}:
3.1s Escape, Enter, ControlLeft
15.6s KeyY
17.1s KeyN
20.2s Space
Nobody pressed those. The page now sends presses since its last message
instead of a running total, the server accumulates them, and a count
goes back to zero as soon as the checkpoint shows the game holding that
key. Same tap after the change:
{"rev":3,"presses":{"Enter":0}} and stays there
The file is also reset when the stream starts.
Cold start is ~2861 chunks of doom init before any picture, measured at ~19 minutes. That cost is identical on every machine and every fresh checkout, and the result is a fixed byte image, so pay it once. 2,185,861 bytes checkpoint -> 168,257 gzipped, resumes intact play now restores it when the live checkpoint is missing, and leaves an existing one alone so a running game keeps its progress.
…restart The old seed was captured at chunk 24253, mid attract loop, so every fresh start landed on the "to order doom" screen. This one is a cold run stopped at the first painted frame (chunk 4153, 60589/64000 pixels). play still resumes /tmp/doom-live.json. restart drops it and starts from the seed: pnpm run restart
play sets a TERM trap for its viewer, so killing the loop shell ran the trap and left the loop alive: it respawned a driver over the file restart had just deleted, and the run came back at chunk 27341 instead of the seed's 4153. kill -9 cannot be trapped. The restart's own ancestors match the same pgrep pattern, so they are kept. drivers: 20465 loops: 20264 live chunks 4228
The live checkpoint lived in /tmp, so a reboot or tmp cleaner threw away
hours of driver progress and the next run silently started from zero.
Before: LIVE=${DOOM_LIVE:-/tmp/doom-live.json}
After: LIVE=${DOOM_LIVE:-./doom/.live/doom-live.json}
doom/.live/ is gitignored; the clean-start seed doom/first-frame.json.gz
stays tracked, so `pnpm run restart` gives everyone the same fast clean
boot while `pnpm run start` resumes.
CLAUDE.md is a symlink to AGENTS.md so both agents read one file.
The module compares the input word against the previous one and only posts an
event for bits that changed, so two owed Enters in a row were one keydown with
no keyup between them, and the menu redrew forever.
Before:
const latched = Object.keys(pressesOwed).filter((code) => pressesOwed[code]! > 0);
After:
const latched = Object.keys(pressesOwed).filter(
(code) => pressesOwed[code]! > 0 && !lastLatched.has(code),
);
Also stop the start loop from resuming after Ctrl+C: the shell trap now exits
130 instead of treating the signal as a driver crash.
ts_post_input walked a 10-entry table at 85856 and `entry` masked the input word with 1023, so the weapon digits, run, strafe and the automap had no bit to travel in. The table could not grow in place (75820 sits at 85896), so the walk is now a select chain and the mask is 0xFFFFF: before: bits 0..9 esc enter arrows use fire y n after: bits 0..19 + Digit1..Digit7, ShiftLeft, AltLeft, Tab Measured against the wasm engine, bit i posts keycode i for all 20 bits (27 13 173 175 172 174 32 157 121 110 49..55 182 184 9); bits 10..19 posted nothing before. ts_input_mask moves 0x5A17C000 -> 0x5A100000 so the low 20 bits are clear for keys and the sentinel still finds the word in the state. drive.ts and stream.ts now derive the split from INPUT_BITS.length instead of hardcoding 22/10.
The page cleared a key's "queued" mark only on the message that reported
the key sitting in the checkpoint's input word. That word holds the key
for one chunk, and the page only sees the chunks that get polled, so a
missed message left the mark on forever: measured Enter spent (latch
seen Enter:1) with the episode screen already drawn and the badge still
reading "queued" 60s later.
The server now answers "is a press still owed" every message:
queued = presses[code] - latch.seen[code] > 0
Second bug on the same counters: reseatSeen dropped a stale count in
memory, but the latch is written only when a phase changes, so disk kept
{"Enter":1} for an hour while the input file said {}. The next driver
would load that count, compute owed = 0, and swallow the next Enter.
It now saves on reseat.
The hardcoded mise path only exists on one machine: Before: ~/.local/share/mise/installs/node/22.21.1/bin/node \ After: node \
ensure_version compared with substring match, so the check passed for
exactly one version and panicked for every other:
"1.0.41".contains("1.0.39") == false
Anyone with a wabt newer than the pin could not run the generator at all.
Parse the x.y.z triple and compare numerically, treating the pinned value
as a floor.
The floor is 1.0.34, the version the committed fixtures were generated
with. 1.0.41 reproduces them byte for byte, so the whole range is safe:
wasm2wat c-add.wasm --enable-code-metadata --inline-exports \
--inline-imports --disable-reference-types --generate-names --fold-exprs
-> identical to the committed c-add.wat
pnpm run start at the repo root hit doom-but-typescript-types, which has no start script, so it failed with ERR_PNPM_NO_SCRIPT_OR_SERVER. The scripts only existed in packages/playground.
The 1280 cap was measured on the flat Record<string,string> memory, where check cost scaled with fuel and TS gave up past ~1300 units. The 64-way trie made cost scale with words touched per chunk, not state size, so the same in-level doom state now retires 655,360 fuel in ~1.1s/chunk with zero elisions (512x fuel per chunk at the old wall cost). The adaptive ladder still halves on "too deep", so the default is a cap: states the checker cannot afford settle lower on their own. 1,310,720 was knocked back to 655,360 on the same state.
$Flush walked the whole trie on every suspend to rebuild a canonical memory, then $Resume re-parsed it. Carry the overlay through the suspend tuple untouched and flush only at $Exit, the host boundary. Old checkpoints still resume; 10-chunk in-level A/B from one seed: 135.3s -> 127.8s (+5.9%).
No measurable change: 0.87s/chunk before and after, within run-to-run noise. Taking it anyway to stay on the nightly the API fixes land in.
teamchong
marked this pull request as ready for review
August 3, 2026 07:50
The browser kept a cumulative presses object and resent it on every
input change, so each keyup replayed every earlier press. Object
insertion order then made the driver pick the oldest key, and a fresh
Enter could sit behind it for a full frame (minutes).
Before:
var presses = {};
press: presses[code] = (presses[code] || 0) + 1; send();
send: ws.send({ rev, keys: down, presses });
After:
press: send(code);
send(pressed): var presses = {}; if (pressed) presses[pressed] = 1;
drive.ts side: while a keydown is sent but not yet acked, a different
pending press replaces it instead of waiting out the frame. Once ack is
high the keydown landed and its release still has to complete.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
linuxdoom-1.10 compiled to wasm and executed by TypeScript 7's type checker.
Types are the runtime:
tsgoinstantiates them, a frame is read back out ofthe emitted state, and no JavaScript runs the game.
Measured on this branch, resuming the checked-in checkpoint:
Playable end to end: title, Escape to the menu, New Game, episode, skill,
status bar drawn in the level.
How it works
doom.cfg.tsinstantiationtypescript/unstable/syncand.../fsontypescript@7.1.0-dev.20260727.1: resolve$Out_*, and that resolution is the executionsrc/aot_cfg.rscfg/drive.tsdoom/stream.tsThe host never executes a wasm instruction. It parses text and moves files.
The checker's 1000-instruction ceiling ends a chunk, so a frame is many
chunks and the checkpoint makes them one run.
Input
20 keys from the page. A press is two halves, keydown and keyup, latched by
the driver because a chunk is minutes and a tap is 100ms: measured 0 of 10
taps reaching the game before the latch, all 10 after.
The page used to clear a key's "queued" badge on the one message that
reported the key in the game's input word, and that word holds it for a
single chunk.
Before:
After, the server answers whether a press is owed, every message:
Running it
startdoom/.live/doom-live.json, or seeds it fromdoom/first-frame.json.gzrestartstartstarttwice is safe: the second sees the live driver, refuses to take over, and re-serves the page.Showcase