What Must Be Learned, and What Must Be Exact - Discrete Execution Boundary Intermediate Language

Training a model to reason over a semantic token ABI — findings from the Krystal experiments (K0 → K4, Curriculum v0 → v0.4.5).

---

Abstract

We study a small but complete question: when a neural network reasons over a semantic token intermediate representation, which parts of reasoning should be learned, and which should be exact machine operations? We train tiny from-scratch LFM2 models on procedurally generated latent worlds encoded as Krystal ABI tokens, with a runtime ("Oracle") that owns identity, arithmetic, mutation and history. Five findings hold across the arc of experiments:

1. Canonical, dense IR is a large sample-efficiency lever — semantic tokens beat controlled English + ByteLevel BPE by ~40 points on compositional holdouts at matched token budgets, and the cost concentrates in token locality, not in the bit layout or in BPE itself.
2. Depth does not extrapolate in a single forward pass — but moving depth into the number of local calls of a learned operator under an external scheduler reaches 100% to depth 16.
3. Learned embeddings should not be asked to be identifiers — equality and associative lookup over unseen symbols fail; identity must be a separate, exact channel.4. A learned "intent compiler" plus an exact runtime handles persistent state — property, motion and transfer stages reach 100% on all three seeds; the sole residual is associative quantity-pointer selection, which is real and seed-dependent (97.9% ± 3.1 over 13 seeds, 6/13 at 100%).
4. The learned part of the stack is tiny — compilers hold at ~126K parameters; the honest whole-system floor is set by the quantity pointer stages at ≈400K, not the 300K a three-seed screen suggested.

Stage	Learned	Exact (runtime)	Result (3 seeds)
K0 follow	relational composition	—	98.6% (1–2 hop), 65% (3-hop)
K1	5 local operators	traversal, state	100% to depth 16
K2c	typed payload functions	identity, lookup	100% on unseen bank
K3/K4	small artifacts	—	≈ cache, no edge
v0.3 all-learned	5 competencies	COPY/EQ_TOKEN/arithmetic	94.1% train / 79.1% holdout
v0.4 M1b	routing, retrieval, plans	exact lowering	100% canonical (audit: shortcuts)
v0.4.5 S0–S9	intent compiler	state, ALU, history	see §7

---

1. The question

Large language models interleave two things that are rarely separated: semantic interpretation (what does this mean?) and exact execution (what follows from it, precisely?). The Krystal project asks whether a tiny model, trained from scratch on a deliberately small semantic vocabulary, can do the first while a deterministic runtime does the second — and, just as important, where the boundary between them should sit.

The setup is deliberately minimal. Procedural generators emit latent worlds (entities, relations, quantities, events). A fixed 4096-token, 12-bit ABI vocabulary encodes those worlds as canonical prefix tapes. A ~1.35M-parameter LFM2 model is trained with ordinary next-token / head prediction. Nothing is pretrained, no language data is used, and every number in this article is a controlled-split holdout on worlds the model never saw.

2. Setup: model, ABI, and the runtime boundary

• Model. transformers.Lfm2ForCausalLM: hidden 128, 4 layers (conv/attention/conv/attention), FFN 384, vocab 4096, ~1.35M params, random init, full training (no LoRA). The v0.4.5 stages add small task heads on the same body.
• ABI. 12-bit token space with class ranges (0x0xx control, 0x2xx operations, 0xExx dynamic entity/QREF slots). Specification: docs/KRYSTAL_ABI_V0.md.
• Generators. Deterministic, seeded latent worlds with controlled semantic splits; 20 competencies in the v0 curriculum (docs/KRYSTAL_CURRICULUM_V0.md), code in curriculum_generators/.
• Runtime. Identity resolution, exact arithmetic, copy, equality, mutation validity, snapshot construction and versioned history are exact operations.

Figure 1 — System Architecture. Learned side (left): what the model is allowed to learn — routing, typed plan compilation, and a retrieval pointer. Exact side (right): what the runtime owns — identity, arithmetic, mutation validity, versioned history. The v0.4+ contract (docs/KRYSTAL_CURRICULUM_V04.md, docs/KRYSTAL_CURRICULUM_V045.md) makes this split explicit in every record: brain_view vs. gold typed plan vs. runtime-recomputed state.

---

3. Representation: what the interface costs

The first block of experiments (K0, docs/RESULTS.md) asks how the form of the tokens determines what a tiny model can learn.

Compositional beats atomic, decisively. On a relational follow task, semantic facts (E03 REL_LINK E17) generalize to unseen entity pairs at 99.95% / 96.0% (1/2-hop), while atomic per-pair tokens collapse to 1.26% / 1.37% there. The ABI bit layout itself is inert: a fixed random permutation of token IDs behaves identically, because a plain embedding lookup cannot see the class structure.

Real tokenization costs a lot — and mostly not because of BPE. At matched token budgets (~7.84M), semantic IR keeps 98.7% on 1–2 hop while controlled English + ByteLevel BPE reaches 57.1% (and 41.6% vs 16.9% with distractors); NL never reaches the 90% threshold even with 2× tokens. The ablation ladder A–H localizes the cost:

Figure 2a — Ablation ladder A–H (same worlds, same model; A→B is an isomorphic rename, C→D is word tokens vs BPE; single-seed screening except B, C, G, H = 2 seeds; Direct 1-hop is 100% in A–G, 99.3% in H):

Profile	Representation	1–2 hop	3-hop	4–6 distractor	3-hop + distractor	Tokens
A	E03 REL_LINK E17 (semantic IR)	99.95%	60.94%	95.07%	53.56%	2.881M
B	entity03 links entity17 (symbolic surface)	99.90%	62.67%	95.12%	54.54%	2.881M
C	canonical English, word tokens	71.14%	56.35%	24.02%	23.10%	6.820M
D	identical text, ByteLevel BPE	67.19%	53.12%	21.48%	18.99%	8.619M
E	+ forward paraphrases	66.99%	38.82%	20.70%	19.48%	9.930M
F	+ inverted word order	56.20%	37.79%	19.29%	15.43%	10.250M
G	+ disjoint relation lexicalization	72.07%	44.63%	26.15%	16.99%	9.388M
H	+ entity aliases and filler	50.93%	30.27%	17.87%	13.06%	10.015M

The B→C cliff (99.90% → 71.14%; distractors 95.12% → 24.02%, tokens 2.37×) is the dominant effect. BPE alone is cheap (C→D ≈ −4 pp); a second cliff (G→H) is mostly entity aliases (82% of the drop). Crucially, it is not merely total length: with equal token budgets a symbolic form with is linked to (B2) stays at 91.3% while a same-length form with filler tokens between operands (B-long) drops to 56.9%. Locality of operands and operators is the mechanism.

The convolutional kernel is a hard locality limit. Prefix facts of the form OP X^d ARG1 X^d ARG2 collapse exactly at separator distance d = kernel_L − 1, and growing the kernel by one rescues exactly the failed distance:

Figure 2b — Kernel-distance cliff (2-hop, 4–6 distractors; 2 seeds; below ≈17% is the random-sink floor):

Kernel L	Last distance in window (d ≤ L−2)	Accuracy	First distance outside (d = L−1)	Accuracy
L3	d1	86.74 ± 1.69%	d2	10.45 ± 0.62%
L4	d2	95.58 ± 0.24%	d3	22.02 ± 5.87%
L5	d3	96.41 ± 0.93%	d4	30.35 ± 12.53%

This is causal evidence that local operator–argument binding in the conv blocks drives the assay.

Numbers should be a typed projection, not brain atoms. A lossy four-band interval projection plus an exact ALU behind a typed call reaches 99.68% end-to-end while exact integers stay invisible to the model; 4 bands is the best granularity for this model (docs/QUANTITY_SEMANTICS_RESULTS.md).

Depth does not extrapolate in one forward pass. Recursive AST execution is ~chance at unseen depth 3+, full-state rewriting is 0% at depth 3, and explicit brackets collapse. Deep computation must be staged — which is exactly what the next section does.

---

4. Local executors: learned operators, external scheduler (K1–K4)

• K1 (Oracle reduction). If a runtime picks the redex and manages state, a model that has learned five local operators applies them perfectly 100% to depth 16 (46,991 local decisions on depth-16 trees), vs. ~chance for whole-AST execution. Depth becomes the number of local calls. (docs/K1_RESULTS.md)
• K2 (open symbols). Pure equality does not transfer to unseen embeddings: exact repetition transfers (100%), rejection of distinct pairs does not (~30%, chance-like), and dynamic BIND … QUERY bank lookup is not learned at all. But with Oracle-resolved typed payloads, the executor is 100% on a fully unseen concept bank. Learned embeddings should not be identifiers. (docs/K2_RESULTS.md)
• K3 / K4 (learned artifacts vs. caches). A fresh tiny model absorbs a finite arbitrary function in 0.6–10 s of GPU training — but a packed lookup table is ~4.5 KB while a checkpoint is 5.4 MB, and on a 16.8M-call space (0.195% seen) the model's 38.96% heldout is tied with a Hamming nearest cache and beaten by a pairwise count cache (41.17%) that is 14–41× smaller. The honest bar for a "learned tool": beat strong non-neural interpolation at matched corpus and storage. (docs/K3_RESULTS.md, docs/K4_RESULTS.md)

---

5. The curriculum line: retrieval, equality, and diversity (v0 → v0.3)

With six generator competencies and one shared body, the v0.2 baseline (docs/CURRICULUM_BASELINE_RESULTS.md) shows only counting is robust (all-six holdout 61.2%); identity sits at its 50% floor by design, and arithmetic/comparison show unstable transfer. Surgical diagnostics localize the failures precisely:

• The ontology ladder is fine. Property/value vocabulary reaches 100%, including unseen recombinations (docs/PROPERTY_LADDER_RESULTS.md).
• The failure is non-local content-addressable retrieval over dynamic keys. Local GET is 100%; distance is harmless when the value is unique; per-scene permutation of handle→position bindings breaks it, and opaque BIND key value lookup sits at exactly 25% chance (docs/PROPERTY_RETRIEVAL_RESULTS.md).
• A hardware bias fixes addressing. ABI slot-forwarding attention gives a monotonic dose-response: R1 goes from ~40% to 100% at alpha ≥ 2, opaque R7 from 25% to 100% (docs/ADDRESS_ATTENTION_RESULTS.md).
• Equality must be exact. With addressing repaired, a learned hidden-state comparator stays at chance even with the gold payload; swapping in exact token equality closes full mixed R6 to 100% (docs/MIXED_EXECUTION_RESULTS.md).
• RETRIEVE needs no labels. The pointer reaches 100% exact source selection from copied answer tokens alone (docs/RETRIEVAL_SUPERVISION_RESULTS.md).

With the hybrid property contract frozen, the v0.3 all-learned body reaches 94.1% train / 79.1% balanced holdout (80.4%): properties and comparison at 100%, subtraction +68.6 pp in joint — but per-label, joint counting has NUM_5 at 0%. Headline numbers hide label failures; balanced, per-label metrics are required gates (docs/CURRICULUM_V03_BASELINE_RESULTS.md). Gradient cosine between tasks is a weak-to-moderate transfer diagnostic (≈ +0.47); transfer itself is highly asymmetric and not additive (docs/PAIRWISE_TRANSFER_RESULTS.md).

---

6. v0.4: the machine-record contract — and its shortcut

The v0.4 contract (docs/KRYSTAL_CURRICULUM_V04.md) makes every record a machine artifact: brain_view + gold typed plan + runtime-recomputed state.

• M0/M1 migrate the quantity and property assays losslessly (99.64% / 100% end-to-end).
• M1b runs all five datapaths in one shared body to 100% canonical — and the post-hoc audit destroys it: permuting only already-seen QREF IDs drops ADD to 43.1%, SUB to 55.0%, comparison to 0%; fresh legal slots and distractors degrade everything further (docs/CURRICULUM_V04_DIVERSITY_AUDIT_RESULTS.md). Canonical repetition let the model encode argument role in token identity.
• Procedural diversity is the fix, causally. Randomizing role assignment (C1) restores 100% on canonical and forced-swap views; random legal-pool allocation (C2) generalizes to fresh slots; a clean/augmented mixture (C3M) reaches 90–92% on the hardest distractor profile at 500 steps and 96.8–100% at 2000. Coupling augmentation axes creates new shortcuts — they must vary independently (docs/CURRICULUM_V04_QREF_RANDOMIZATION_RESULTS.md).

---

7. v0.4.5: persistent state, three seeds (S0–S9)

The latest line adds persistent state on top of the corrected foundation (docs/KRYSTAL_CURRICULUM_V045.md). Every stage follows the same boundary: learned plan → exact mutation → learned consumer on the exact snapshot. All numbers below are controlled-test holdouts at the final checkpoint (500 steps; 250 for S8), model seeds 1, 2 and 3 (same corpus; the probes S4–S7 use same-seed frozen checkpoints and fresh trajectory corpora).

Variance audit (n=13 for S2, crossed design for S6/S7). Because the three-seed spread on the quantity stages was wide, S2 was retrained on seeds 4–13 and the S6/S7 probes were run in a crossed init × corpus design (frozen seed-1/2/3 checkpoints across fresh corpus seeds). Two results: the S2 residual is a real seed-dependent floor (below), and the S6/S7 spread is ~90% frozen-S2 init variance — varying only the corpus moves depth-16 survival by ≤3.5 pts (S7) / ≤1.5 pts (S6), while the init dimension spans 22 pts (S7, all three inits at one corpus) and ~15 pts total (S6). The "product of conditioned local reliabilities" prediction still matches observed survival to <2 pts in every cell.

Stage	Learned	Exact	Seed 1	Seed 2	Seed 3
S0 SET_PROPERTY	plan (op/entity/property/value)	mutation, snapshot	100%	100%	100%
S1 MOVE	old-edge binding	invalidate + commit	100%	100%	100%
S2 CHANGE_QUANTITY	op, entity, delta, current QREF	ADD/SUB_INT	96.61%	98.31%	100%
S3 TRANSFER	both QREFs, amount	atomic coupled mutation	100%	100%	100%
S4 MOVE rollout (d1–16)	frozen S1 calls	snapshot chain	100%	100%	100%
S5 mixed dispatch (rollout)	frozen S0–S3	dispatch, state	95.5%	99.5%	100%
S6 random re-entry (d16)	frozen S0–S3	legal-event sampling	85.0%	93.0%	100%
S7 versioned QREFs (d16)	current binding	COW, stale/history	78.0%	90.5%	100%
S8 temporal selectors	view selection	snapshot, lifetime	100%	100%	100%
S9 mixed history (e2e)	anchor + datapath	frame projection	97.44%	99.68%	100%

Reading this table:

• S0, S1, S3, S4, S8 are seed-robust at 100%. Property and motion transitions, the atomic two-entity transfer, the homogeneous rollout, and temporal view selection hold on every seed. Per-stage reports: docs/CURRICULUM_V045_S{0,1,3,4,8}_RESULTS.md.
• S2 is the single bottleneck, and it is a real seed-dependent floor, not a lucky-seed artifact. The exact path (opcode, entity, delta) is 100%; the learned selection of the live persistent QREF by (entity, carrier) over 13 seeds is 97.91% ± 3.09, range 91.5–100%, 6/13 seeds at 100% — mass at 100% plus a tail down to ~91.5% (seeds 5 and 10 sit at 91.5%), so roughly half of initializations leave a 1.7–8.5% residual. Failure-mode isolation on all 13 frozen checkpoints shows the residual is exactly one wrong token — the current-QREF pointer lands on another entity's ref (never the delta; entity/delta/opcode always correct) — and a shortcut audit rules out a data shortcut: the ref marginal is uniform over the 32-token pool, and frozen checkpoints transfer at in-band accuracy to fresh corpus seeds and to a never-seen QREF pool (E60–E7F), so binding is slot-identity-agnostic. A 1000-step retest is a slow-convergence basin, not a hard ceiling: seeds 5/10 improve 91.5 → 99.15 / 94.07%, but neither reaches 100% (docs/CURRICULUM_V045_S2_RESULTS.md).
• Rollout survival is accumulation of local error, not new interaction failure. S5/S6/S7 whole-rollout accuracy across seeds (95.5–100%, and depth-16 survival 78–100%) tracks the product of conditioned local reliabilities to <2 pts on every seed and every corpus; every first divergence is CHANGE_QUANTITY compile/current-ref selection. The wide three-seed spreads are ~90% init variance of the frozen S2 checkpoint, not trajectory noise. Seed 3 is fully exact at every depth (docs/CURRICULUM_V045_S{5,6,7}_RESULTS.md).
• History is an external machine facility. The S9 compiler (anchor, argument, datapath, plan, bounded frame) is 100% on all three seeds; the only residual is the inherited S2 quantity comparison (92.31 / 99.04 / 100%), identical on the gold and model-selected frames (docs/CURRICULUM_V045_S9_RESULTS.md).

How small can the stack get? A model-size sweep (docs/CURRICULUM_V045_SIZE_SWEEP_RESULTS.md) shows the compilers hold at 100% down to 125,700 params (~10.7× smaller) and the S8 temporal selector survives at 88K. The quantity pointer stages (S2/S3) set the honest whole-system floor higher than the three-seed screen suggested: at ~300K (32/4/384), 2 of 9 S3 seeds collapse (50.98%, 90.20%) and 1 of 9 S2 seeds fails (84.75%) — so ≈300K gives roughly 80–85% seed reliability, not the 3/3 robustness the published drill-down implied, and a ≈400K-size stage is the defensible floor for high-reliability claims:

Figure 3a — Compiler stages S8/S9 by size (plan accuracy):

hidden/layers/FFN	Params	S8 temporal selector	S9 mixed-history plan
128/4/384 (baseline)	1,345,536	100%	100%
96/4/384	966,144	100%	100%
64/4/384	615,424	100%	100%
48/4/384	450,816	100%	100%
32/4/384	293,376	100%	100%
32/4/192	219,648	100%	100%
24/4/192	162,048	100%	100%
24/4/96	134,400	100%	100%
24/3/96	125,700	100% (3/3 seeds)	100% (3/3 seeds)
24/2/96	116,364	100%	99.36% (1/3 seeds)
16/4/96	87,808	100%	99.04% (1/3 seeds)

Figure 3b — Whole-stack drill-down (plan accuracy; boundary sizes run on 9 seeds for S2/S3): S0/S1 hold 100% at 135K; S2/S3 collapse at 135K (77% / 47% on a seed) and at ~300K still fail on 1/9 (S2: 84.75%) and 2/9 (S3: 50.98%, 90.20%) seeds. The first size with no observed failure in nine seeds was not found at 300K — the honest floor is ≈400K. Crumble order: temporal anchor argument first, then quantity pointer reads.

---

8. Takeaways

• The boundary is the result. Every success came from a clean split (Figure 1): the model learns which semantic view/op/argument; the runtime executes exact identity, arithmetic, mutation, lifetime and history. Learned embeddings should not be asked to be identifiers (K2, diversity audit).
• Canonical, dense IR is a sample-efficiency lever, not magic. The bit layout is inert; token locality and per-token information density drive the B→C cliff; BPE and aliases add real but separable costs.
• No depth in a single forward pass. Recursive execution, full-state rewrite and AST depth fail to extrapolate; moving depth into number of local calls (K1, S4–S9) works to depth 16 and beyond.
• Procedural diversity is a correctness requirement. Without it, models encode role in token identity; with independent randomization of slot allocation/layout/distractors, the same capacity generalizes.
• Learned artifacts must beat caches. K3/K4 set the bar: compare against strong non-neural interpolation at matched corpus and storage.
• The whole stack is tiny. ~1.35M params, seconds-to-minutes of GPU training per stage, compilers down to ~126K — but the quantity-pointer stages set an honest floor at ≈400K once seed variance is counted (2/9 S3 failures at 300K).

9. Open questions

• The S2 quantity-pointer floor is characterized (13 seeds, mass at 100% plus a tail to 91.5%; the 1000-step retest lifts the low-basin seeds to 94–99% but not always 100%, and the residual is a neighboring-entity QREF pick under candidate competition, not a data shortcut). Open: what makes the pointer head peak on a neighbor at high candidate counts (entity_count=5 carries 3× its share) — an attention/capacity explanation vs. a better curriculum — and ~5 seeds at the ≈400K floor size to confirm reliability.
• Mixed-event training of one shared body (S5/S6 currently compose frozen artifacts) and history-depth extrapolation (S8/S9 are interpolation).
• Replication of the v0.4 diversity baselines (C3F) across seeds.
• Scaling the payload bank (more properties/relations/values), then real NL + aliases as a frontend that compiles to the ABI.

10. Reproducibility

• Reproduction commands: bottom of every docs/*_RESULTS.md; data generators and tests: curriculum_generators/; v0.4 / v0.4.5 code: curriculum_v04/, curriculum_v045/; raw machine reports: out/.
• Environment note (NixOS): run inside the dev shell (direnv / nix develop) or set LD_LIBRARY_PATH=/run/opengl-driver/lib so torch can reach the NVIDIA driver; use the finetune venv (chomato-monorepo/packages/finetune/.venv) for the published GPU runs.
• All S0–S9 numbers above are model seeds 1–3 on the baseline 128/4/384 model; S2 was additionally run on seeds 4–13, the S6/S7 probes were crossed over corpus seeds, and the size-sweep boundary cells (S2/S3 at 24/3/96 and 32/4/384) were extended to 6–9 seeds. New runs: out/curriculum-v045-s2-seed{4..13}/, out/curriculum-v045-s{6,7}-corr*/, out/size-sweep-drilldown/s{2,3}-h{24l3i96,32l4i384}-seed*/.