64GPT — a tiny neural NPC dialogue brain running on a real Nintendo 64

A character-level GRU (~100K parameters, int8-quantized) that generates short NPC dialogue on real N64 hardware, built with the Pyrite64 engine on libdragon. The demo is a dialogue box that streams AI-generated text; controller buttons cycle the conditioning (NPC=guard / Mood=angry / Event=stole_sword), A regenerates.

Method: the walking skeleton

The AI is never built in isolation and "ported at the end". A bootable ROM ships at every milestone — starting with a fake model that says one canned line — and each milestone swaps in the smallest real piece. The on-console proof is re-verified continuously:

• Inference is pure integer math and the model blob is parsed byte-by-byte, so output is bit-exact identical on the Mac, in the Ares emulator, and on the console.
• Every ROM boots with a self-test that replays committed golden vectors and prints SELFTEST PASS/FAIL on screen.
• Per-milestone gate: host tests green + Ares boot with SELFTEST PASS. Real-hardware (EverDrive-64) runs are occasional spot-checks, and required for v1.0.

Status

• M0 — repo skeleton, docs, verification loop: toolchain + Ares installed, stock example built headlessly and boots in Ares (docs/milestones/m0.md)
• M1 — ROM v0.1, the walking skeleton: canned-line "model" behind the final streaming API; parser + generation tests green; boots in Ares with SELFTEST PASS + streaming dialogue box (docs/milestones/m1.md)
• M2 — ROM v0.2: real GRU overfit on ONE line — first neural net on N64: int8 weights, integer-only inference, bit-identical to the Python trainer (docs/milestones/m2.md)
• M3 — ROM v0.3: conditioning on a dozen hand-written lines — prompt priming (NPC=… MOOD=… EV=…|) selects which line the GRU speaks; 12/12 bit-exact on host and in the ROM self-test (docs/milestones/m3.md)
• M4 — ROM v0.9: full generated corpus + temperature/top-k sampling — 1.5MB generated corpus, H=128 (~68K params) trained with a val split; seeded xorshift32 sampler bit-exact from trainer to silicon; every regenerate speaks a fresh in-character line (docs/milestones/m4.md)
• M5 — ROM v1.0-rc: performance pass — 16.6ms → 9.8ms per char (-O3 core + int32 accumulators, bit-exactness preserved); raw 102 chars/sec, sustained 60 at a held 60 VPS — 2× the ≥30 gate (docs/milestones/m5.md, docs/07-performance.md)
• M6 — ROM v1.0: running on a real N64 via EverDrive
• M6.1 — ROM v1.1: RSP-powered inference — the hot matvec runs on the graphics coprocessor, bit-exact (boot self-test + on-screen CPU-vs-RSP cross-check), 10.3ms → 4.8ms per char with the CPU freed; engine cap raised to H=256 for the M7-scale model (docs/milestones/m6.1.md)
• M7 — ROM v1.2: first living NPC, Selena — Event Bus + Shared World State + NPC Database + Context Builder replace hardcoded demo conditioning; H=256 (~266K params) trained on a schema-conditioned, 120-combo corpus with prefix-loss masking and a combo-level holdout split; identity-conditioning proven at scale (per-axis divergence table, identity 0.94 ≥ mood 0.92); int8-vs-float agreement 99.25%; SELFTEST PASS in Ares (docs/milestones/m7.md)
• M8 — ROM v1.3: archetypes, the portability substrate — manifest meta-schema (docs/08-manifest-schema.md) separates mechanism from per-game content; archetype/instance system (personality-range + deterministic xorshift32 jitter) proven on guard, 4 seeded instances sharing Selena's model; within-archetype divergence 0.94 vs. mood baseline 0.97; SELFTEST PASS in Ares, START cycles NPC live in the demo (docs/milestones/m8.md)
• M9 — ROM v1.4: compositional conditioning — M8's opaque N:<id> identity tag replaced with reusable descriptive features (P:<person> D:<descriptor> OCC:<occupation> R:<tier> M:<mood> C:<context> EV:<event>|); RSP matvec kernel generalized to H=320 (~394K params, first non-power-of-2 hidden size, 320B DMEM headroom); curated 3-character cast (Bram/Fergus/Kragan) trained via template-grammar corpus generation after a freeform-LLM-corpus attempt measurably garbled at this model's scale; val loss 0.0985 (beats M8's 0.1026 despite +50% params), int8-vs-float agreement 0.9838; SELFTEST PASS + XCHK PASS together on the real trained model for the first time; all three new characters confirmed live via START cycling on real hardware. Known open gap: generated text coherence remains inconsistent, confirmed live on real hardware — not silently claimed solved (docs/milestones/m9.md, docs/plan.md Known follow-ups). M9.2 (docs/milestones/ m9.2.md): evaluated four ideas from an external N64 project (Legend of Elya) — RSP tiling and float math primitives rejected on the merits (redundant/less-precise, and violates this project's hard no-floats constraint respectively), a narrow Kragan catchphrase bank adopted and A/B tested. Production retrain + ROM build + Ares boot: SELFTEST PASS, RSP ON, XCHK PASS, but the coherence gap persists on real hardware at full production scale — the mitigation didn't close it, recorded honestly rather than oversold
• M10 — ROM v1.5: procedural cast — a recurring villain (Shadewrath, full tier), a mid-tier talking boss (Korrath) bound into his service, 4 new town archetypes, and DungeonGenerator::npcsForLevel() deriving thin-tier NPC placement deterministically from a level seed (30 host-test checks, including a test that reloading a level reproduces identical dialogue text, not just seed equality). A real EEPROM save system (game/src/user/SaveData.h/.cpp, libdragon's eepfs) persists the villain's/boss's highest trust tier across a genuine Ares restart — verified live, not just compiled. Wired into the live demo: R generates a new dungeon level, START cycles its NPCs, Z returns to the fixed roster; verified on real hardware with the bad guy's persisted trust tier carrying into a freshly-generated encounter. SELFTEST + XCHK PASS for 27 goldens, RSP ON. Known open gap, carried forward honestly: Shadewrath's/Korrath's generated-text coherence remains inconsistent even after a density-raising retrain, which surfaced a new cross-trust-tier content-bleed failure mode rather than resolving the original one (docs/milestones/m10.md, docs/plan.md Known follow-ups)
• M11 — ROM v1.6: town cast + Elowen — the gossip mechanism (a player-caused event reaches nearby town NPCs' conditioning secondhand, verified: a hand-authored gossip line reproduced verbatim mid-generation), 2 more town archetypes (merchant/ healer), and Elowen, a rescued elf princess found in the dungeon (mid tier, dungeon-only, her own persisted trust tier doubling as the in-game rescue event). SELFTEST + XCHK PASS for 36 goldens, RSP ON. Honest negative result, shipped anyway: this milestone's quality push (a shared lore bank meant to fix Shadewrath's/Korrath's standing coherence gap) was combined with the new content into one retrain — agreement dropped 0.9771→0.9536 and garbling spread beyond Shadewrath/Korrath to previously-clean characters including Selena, evidence the combined-retrain approach (not the lore-bank idea itself) was the mistake. Recorded plainly rather than reworked until it looked like a win (docs/milestones/m11.md has the full record and a retrospective correction to this project's own prior "one retrain covers all of it" advice). M11.1 retries with proper single-variable isolation
• M11.1 — ROM v1.7: full genericization + isolated lore-bank retrain. The last old-scheme holdouts (Selena, Shadewrath, Korrath, Elowen, and guard's 4 fixed instances) all migrated onto NpcService's compositional scheme — ContextBuilder/N:<id> deleted entirely (zero callers left after the migration), not just deprecated. Three new conditioning axes: AUD: (audience, gates private-register content), BOND: (relationship type, orthogonal to R:'s closeness), SPECIES: (human/elf/dwarf/beast/shade). Then ran the controlled experiment M11's combined retrain made impossible — baseline vs. treatment (Ravendale-lore bank on/off), RNG-identical otherwise, against a bar pre-registered before either result was seen. Second honest negative result, properly isolated this time: the lore bank does not clear the bar (invented-word count 69→67 of a 26-golden probe, needed ≤34) — agreement and divergence both held fine, the coherence gap simply didn't move. Shipped the winning baseline configuration: val loss 0.1015, agreement 0.9508. SELFTEST + XCHK PASS for 36 goldens, RSP ON. Full record: docs/milestones/m11.1.md
• M12 — ROM v1.8: capacity scaling — does a bigger model fix coherence? NGPT_GRU_MAX_HIDDEN bumped 320→1024 (explicit sign-off, core/ngpt.h is a frozen interface) via a K-chunked RSP kernel ported from the hardware-verified spike/rsp-matvec-ktile branch onto current main, plus an int64 overflow-mitigation on the CPU reference path. Trained the exact M11.1 corpus/seed at H=1024 (only capacity changed) via MPS. Third honest negative result: val loss landed worse than the H=320 baseline (0.1051 vs. 0.1015), and the garbling itself didn't move — agreement and divergence held fine, but that's not the thing that mattered. All three levers this project has tried for the standing coherence problem (raw pair count, shared structural content, model capacity) are now tried and found insufficient. Also found and fixed a real, previously-latent bug in the golden-recording pipeline (generate_sampled()'s max_len=256 default silently truncating goldens past 256 chars, present since M11.1). SELFTEST + XCHK PASS for 36 goldens, RSP ON, ~5 ch/s (an explicit, disclosed tradeoff, not a regression). Full record: docs/milestones/m12.md
• M12.1 — ROM v1.9: the coherence rescue — a controlled decoding experiment on M12's own model found the garbling was never a capacity problem: int8 quantization alone tripled invented words (5→17 greedy on a 12-prompt probe), on top of a 75:1 per-character training-data skew (Selena 36,000 pairs vs. Elowen 480) and an unconditional sampler forcing 144 off-argmax draws per 12 lines — and val loss (teacher-forced, prefix-masked, float) was blind to all three. Fixed: quantization-aware fine-tuning (straight-through estimator onto the exact int8 export grid — the shipped int8 model now BEATS the old pre-quantization float model, 0.1002 vs. 0.1015) and corpus rebalancing (worst-case ratio 75:1 → 6:1). The old 0.95 agreement gate — which FATALed the first, excellent QAT run — was retired as a quality bar (demoted to a 0.95 sanity floor) in favor of gating on a new shipped-config coherence probe + greedy int8-vs-float parity. Back on H=320 (capacity exonerated by M12; both kernels kept as a build-time toggle, NGPT_RSP_H, instead of overwriting one in git history). Coherence probe: 2.60 → 0.50 invented words/line across 48 sampled lines. Two more phases closed this out: an integer min-p sampler gate (ngpt_set_minp, arXiv 2407.01082) cut the probe's invented rate a further 47% (0.354 → 0.188), then a lexicon-trie decode guard (ngpt_set_trie_guard) made invented words structurally unreachable — every sampler candidate is masked to real corpus-word continuations, backed by a new blob format version that leaves every pre-M12.1 vector file untouched. Result: 36-golden invented-word total 91 → 26 → 0 (a hard guarantee, not a probe sample). SELFTEST PASS, RSP ON, XCHK PASS, 44 ch/s restored (M5-era bar) throughout. Full plan: docs/ideas-coherence-rescue-plan.md; record: docs/milestones/ m12.1.md.
• M12.2–M12.6 — five follow-on conditioning-mechanism experiments chasing the standing coherence gap on Guard/Korrath/Elowen, none shipping a new ROM (v1.9 unchanged throughout — every result below is a host-only training experiment, trainer/m12_*.py): richer per-character voice content (M12.2, 1.65 inv/line, worse than M12.1's ≤1.00 gate); per-step D:/M: attribute conditioning (M12.3, 2.33, worse still); the same mechanism with the redundant prefix copy ablated (M12.4, 1.44, the best of the five, still short of the gate); FiLM hidden-state modulation (M12.5, 2.15, worse than M12.4); densifying Korrath's/Elowen's thinnest phrase banks (M12.6, 1.65, no better than M12.2). Four honest negatives and one real-but-insufficient improvement, all measured against the same pre-registered ≤1.00 bar — full tables in each file: docs/milestones/m12.2.md · m12.3.md · m12.4.md · m12.5.md · m12.6.md.
• M13 — LLM-as-judge phrase-bank distillation (in progress, no ROM shipped). Mechanism 4 (splice judge-approved fragments into Guard/Korrath's banks) ran its full pre-registered protocol — FINAL VERDICT: FAIL. Both the approved arm (2.4375 inv/line) and the rejected arm (2.75) landed worse than an untouched baseline (1.50/1.81, noise floor 0.3125) — the judge's relative ranking was correct (approved beat rejected) but the underlying premise ("judge-approved content improves real training outcomes") didn't hold. A real, unexpected negative, not a bug — full protocol, numbers, and plausible-cause analysis: docs/milestones/ m13.md.
• M14 — the portability release: capacity-monitoring metric (trainer/ngpt_trainer/capacity_monitor.py, real baseline numbers — Selena 0.1356 / Shadewrath 0.1763 held-out float val loss), capacity split-trigger decided and calibrated against real data (5%, float-loss-only), the manifest-update skill (validate → retrain → gate → ship-or-refuse, manifest_update.py + manifest_update_cli.py), and the cross-genre portability proof — a sci-fi engineer archetype ported through the unmodified toolkit with zero mechanism-code changes (mechanism confirmed; coherence at production density explicitly out of scope for this lightweight proof). A real hardware demo of the engineer archetype is in progress (booting it in Ares as the first end-to-end, not just host-side Python, verification of the portability mechanism) — this line updates once that lands. Full record: docs/milestones/m14.md.

Quickstart

Host tests (any OS, needs cmake + a C++20 compiler):

cmake -B build tests && cmake --build build
ctest --test-dir build --output-on-failure

Regenerate the model blob + golden vectors (stdlib-only Python 3):

python3 trainer/make_canned_blob.py

Build the ROM (macOS, after the one-time toolchain install — docs/01-toolchain-and-pyrite64.md):

# ./pyrite64 is a local wrapper that execs the app binary by its REAL path —
# running it via a plain symlink breaks the app's resource lookup and the
# build fails with an empty Makefile (docs/01-toolchain-and-pyrite64.md).
./pyrite64 --cli --cmd build "$PWD/game/project.p64proj"   # → game/64gpt.z64
ares game/64gpt.z64

Repo map

dir	contents
core/	the inference engine — portable C-style C++, zero libdragon deps, compiled unchanged into host tests and the ROM
tests/	host test suite (CMake/CTest, ASan/UBSan); tests/vectors/ holds committed goldens
trainer/	Python tooling: blob export now; corpus/train/quantize/ref-impl from M2
game/	the Pyrite64 project (see game/README.md)
docs/	concept guides + per-milestone notes
versions/	built .z64 ROMs, one per milestone (EverDrive-ready; see its README)

Docs

Written for a good software engineer who has never done game dev or embedded ML — start at 00 and read in order, or jump in:

• 00 — N64 primer
• 01 — Toolchain & Pyrite64
• 02 — Pyrite64 scripting: DialogueDemo explained
• 03 — The NGPT model blob format
• 04 — Fixed-point inference
• 05 — A GRU on a napkin
• 06 — The training pipeline
• 07 — Performance
• ADR 0001 — Isolate decision logic into portable C++, host-test it
• ADR 0002 — No Debug/Release build type; a source switch gates the boot self-test instead
• Milestone notes: m0 · m1 · m2 · m3 · m4 · m5 · m6 · m6.1 · m7 · m8 · m8.1 · m9 · m9.1 · m9.2 · m10 · m11 · m11.1 · m12 · m12.1 · m12.2 · m12.3 · m12.4 · m12.5 · m12.6 · m13 · m14

Pinned versions

what	version	why
pyrite64-mac	v0.4.0 (release-asset app at ~/GitHub/Pyrite64-v0.4.0, source clone at ~/GitHub/pyrite64-mac)	early-dev engine, breaking changes; upgrade deliberately
MIPS toolchain	GCC 14.4.0 in ~/pyrite64-sdk	installed by the Toolchain Manager
Ares emulator	v147+	hardware-accurate; installed by the Toolchain Manager
Python	3.12 via uv python pin 3.12 (from M2)	PyTorch wheel availability