Correct false and stale claims across the READMEs (plus research-loop hardening) - #5
Merged
Conversation
…, crash-survivable rungs Three long-standing blockers on the unattended loop, all execution-verified (411 passed / 1 skipped, green *while* a trainer is live — the case that matters). 1. loop_state schema fork (the last cron blocker). default_state() emitted a 7-key shape that diverged from the §C5 pinned 12-key one, so a fail-open recovery silently DROPPED the recovery-critical train_pid/ckpt_path/ resume_cmd instead of returning them as None — the recovery chain would KeyError exactly when it was needed. Now emits the full live schema; the dead fork keys (last_marker, iteration) and the unused advance(marker=) arg are gone. Three new tests pin the shape, the fail-open keys, and their survival across record_resume(). 2. sentinel thermal kill-switch. The box was hard-locking mid-run under sustained load (a ~16-18h 420M rung could never finish). watch now samples the GPU die temp, the hottest ACPI SoC zone, and — the definitive, self-calibrating signal — the GPU's own hw/sw thermal-slowdown flag, and SIGTERMs at >=90C or an active throttle after 3 consecutive samples (debounced against nvidia-smi blips). Fail-open throughout: an unreadable sensor never fabricates heat. The kill JSON gains trigger/gpu_temp_c/ soc_temp_c/gpu_throttling; heartbeats carry the temps. 3. Crash-survivable rungs. train_ablation.py checkpoints full training state (model + both optimizers + step + python/numpy/torch/cuda RNG, atomic write + fsync before rename) every --resume_every steps and auto-resumes, failing closed to a fresh start on an unreadable checkpoint; a run with no resume file is numerically identical to before. run_ladder.sh loops passes until every cell has its .done marker (a thermally-killed cell is re-attempted, resuming from its checkpoint, rather than the driver exiting after one pass), behind a bounded cool-down gate and an escalating hot-spell backoff, and smoke is now sentinel-watched like the cells. Also fixes a test-isolation bug found while running the gate beside the live HybridSSM trainer: test_boot_resume's _run() helper never set the BOOT_TRAINER_RE hook that boot_resume.sh documents "for hermetic tests only", so pgrep matched the REAL host trainer and three recovery-guard decisions degraded to already-running. The suite was therefore green-by-accident on an idle box and red during training — i.e. the recovery guards went unverified precisely when recovery matters. The later tests in that file already scoped their own pattern; the early helper was missed. Added a regression test that spawns a trainer-shaped process and asserts the decision is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… first pretrain arm in flight A hybrid attention-SSM LM written from a blank file in JAX/Flax — the standing "next model in JAX" request, and the first build here with no bit-exact oracle (novel design → the gate is a numerical cross-check at ~1e-2, §C14). Architecture (ARCHITECTURE.md): d=768, 24 layers in a 1:1 [full-attn, efficient] interleave, GQA 12/4 head_dim 64 + RoPE, Mamba-2-style selective SSM via associative_scan on the efficient layers, SwiGLU, RMSNorm, tied embedding over the Qwen3 151,936 vocab, chunked CE. 189.1M non-embed / 305.8M total. The mixer type / attention fraction / NoPE toggles are what make it a study rather than a model: the headline object is an emergence-speed curve, not a single number. Guards (§C1) verified in-code, not asserted: jax_safe_env imports before jax (train_hybrid.py:10 vs :15), and chunked_cross_entropy (model.py:136) does a streaming max + sumexp over vocab chunks so the (N, 151936) logit matrix is never materialized — the exact shape that hard-crashed the box in June. Arm ssm_base_s0 is training on real data (FineWeb-Edu sample-10BT via the Qwen3 tokcache, 170,034,304 train + 300,000 val tokens, seed 0 — the ~168M rung of the brief's token-budget ladder, and BPB-comparable to the 596M study). It survived a real incident: the sentinel killed it at step 580 when the SSM scan + chunked CE under autodiff held ~61 GB (pool 81.3% >= the 0.80 line, no thermal component); nn.remat on the decoder block (model.py:129) plus batch 8->4 took allocation to 16.6 GB, and it resumed from the step-400 checkpoint. That was a manual recovery behind a config change — the "not safe to auto-resume at the same config" path — so no auto-resume budget was consumed. c5_evidence.json is included and is honest about its own provenance: it was RECONSTRUCTED after the launch, because the 2026-07-19 launch created the ledger run entry without ever writing it, so §C5's "evidence recorded BEFORE launch" was not met for this run. Every item carries a src tag — log / derived / attestation / not-captured — rather than a uniform claim of compliance. Two items are weak and say so: the §C5.0 smoke numbers survive only as prose in BUILD_STATUS.md with no captured log, and the pre-launch concurrency check left no evidence at all. Recorded, not buried: verify.py last ran 2026-07-19 12:52 and model.py changed at 22:27 to add nn.remat, so the verify gate has NOT been re-run against the model that is actually training. nn.remat is semantically identity and the loss curve is continuous across the resume, but that is corroboration, not verification — re-run verify.py and capture verify.log before this arm is scored (GPU work, so it waits for the arm to finish per the one-job rule). The arm as launched also deviates from ARCHITECTURE.md: seq 2048 not 4096, and plain AdamW not Muon, since the JAX Muon port is unwritten — which makes AdamW @ seq 2048 the ladder's baseline recipe. No result is claimed. n=1 seed, no comparand, no iso-FLOP match: this is a baseline datum, capped at directional at absolute best. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The blanket `research/` ignore made git backup depend on WHEN a file was written: anything committed before that line was added stayed tracked, and everything written after it had no backup anywhere. That silently left ~70 load-bearing, unit-tested files with a single copy on one disk — including posttrain_losses.py (the SFT/DPO/GRPO loss math), scaling_ladder.py, the CCE kernel trio, kernel_oracle.py, data_decontam.py, boot_resume.sh and thermal_log.py (the whole crash/thermal recovery chain), and 11 of the 34 test modules. Their tracked siblings (ledger.py, sentinel tests, eval_stats.py, CI) were safe purely by having been written earlier. It also silently overrode research/.gitignore, whose own stated policy is to "keep the durable record tracked; exclude only transient/runtime files" — that file has been dead letter for as long as the blanket rule existed, because git never descends into an excluded directory to read it. Now: `research/**` with directories re-included so git descends, `.py`/`.sh` re-included as durable source, and harness-search's machine-GENERATED candidate files (archive/, targets/*/candidates/) re-excluded since they are search output, not authored source. 38 source files + research/recovery/.gitignore land here — 39 new paths, 392 K, no blobs; `git status` stays instant (602 files under research/). Deliberately still local-only, unchanged by this commit: ledger.json and loop_state.json (churning state, where a stale tracked copy is worse than none — durable backup is research/backup_ledger.sh), and the generated artifacts under digests/, pulse/, radar/, provenance/, ledger/runs/ and ledger/backups/. Whether the durable RECORD (run detail md, briefs, digests) should also be tracked is a separate call and is left open — research/.gitignore's original intent says yes, the branch-switch incident that destroyed a ledger.json says be careful. Caveat worth knowing: these paths are tracked on this branch only, so a `git checkout main` will remove them from the working tree until this branch is merged or checked back out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…erges away The section claimed the four 420M rungs were "running". They finished 2026-07-12: all ten cells carry .done markers, ladder.done is set, and score_ladder.py wrote verdict.json. Replaced with the actual result, every number traced to that file. The gap (AdamW − NorMuon, BPB) shrinks with budget on both corpora — wikitext-2 +0.474 [+0.443,+0.505] at 42M, +0.126 [+0.089,+0.163] at 168M, +0.073 [−0.038,+0.184] at 420M; code_py +0.502 → +0.176 → +0.192 — with OLS slopes over log10(tokens) of −0.416 (r² 0.92) and −0.328 (r² 0.81). Verdict CONVERGES on both, ledger verdict directional. Kept the honesty that the raw shape hides: the fitted slope is resolved but the edge at the top rung is NOT (edge_resolved=false — at n=2 the 420M CI holds both "converged" and "still ahead"), the code_py gap does not even shrink monotonically (+0.176 → +0.192; only the fit is negative), and the run carries an inherited confound that pushes the same direction — both learning rates were tuned at the 42M horizon and never re-tuned per budget, so some of the fade could be a mis-tuned-LR artifact rather than convergence. Stated what it would take to earn more: a 3rd 420M seed, an 840M rung, a per-horizon LR check. Also closed the loop on the thermal note: a 2026-07-09 hot spell had the ladder stuck at 4/10 cells with no net progress, and after the crash-survival work the four 420M rungs finished 2026-07-10→12 *through* repeated thermal kills. The recovery chain is load-bearing, not theoretical. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…hat hid it
`--type pretrain-ablation` is not a run type — it is an OBJECTIVE. argparse
rejected the call with exit 2 on every scoring pass, so the scaling-persistence
ladder was scored and never landed in the ledger. The entry that exists was put
there by hand afterwards, which is why it carried started=2026-07-13 AFTER
ended=2026-07-12.
The typo is the small half. The reason a broken ledger write survived a
multi-day ladder is that the call passed `check=False` inside a `try/except
Exception`: a non-zero exit raises nothing, so nothing was caught, nothing was
printed, and the script still returned 0. A failure that cannot be seen is worse
than a crash.
sync_ledger() now:
* passes `--type scaling-fit` (a token-budget ladder IS a scaling-fit run),
keeping `--objective pretrain-ablation` where it belongs;
* captures the exit code and prints an unmissable banner naming the exit
status, the full quoted argv and the CLI's own error, then returns False;
* is idempotent — add-run REJECTS a duplicate run_id (exit 2), so a re-scored
ladder falls back to update-run instead of losing the write;
* records real dates and a terminal state: `started` parsed from the first
dated START line in run_ladder.log, `ended` from the ladder.done mtime, and
status=done (add-run defaults to `launched`), instead of letting both dates
default to today.
Verified against a sandbox copy of the ledger, all three paths: add-run on a
fresh ledger creates type=scaling-fit with started(2026-07-05) <= ended
(2026-07-12); a second call falls through to update-run and is idempotent; and
an injected bad flag produces the loud banner and False rather than silence. The
old argv was re-run against ledger.py to confirm it is still rejected — the fix
was necessary, not cosmetic.
Regression guard in test_ledger.py: a repo-wide lint asserting every literal
--type handed to ledger.py is in RUN_TYPES, plus three tests that keep the lint
honest — it must not be vacuous, it must flag the original broken argv shape
(proven on a synthetic caller, not by reverting the real one), and it must not
flag the fixed shape. Its one blind spot is documented in the docstring: a
caller that assembles the CLI path piecewise would not be scanned.
Also corrected the live ledger entry's inverted dates via ledger.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ells to run continuously
Three things had to be right before queuing 5.9 GPU-days, and two of them were wrong.
1. RESUME INFLATED THE BUDGET. train_hybrid.py looped
`range(start_step, start_step + steps)`, so a resumed cell repeated the whole
budget from its resume point — the pilot arm resumed at 400 and ran 21,156
steps (173.3M tok vs a declared 170.0M, +1.93%), and ran the optax cosine
schedule past its end. Harmless at n=1; fatal for a ladder, because only the
arms that happen to crash would get extra compute, in proportion to where
they crashed, with nothing in the logs saying so. Now `range(start_step,
steps)`: total steps are independent of the resume point.
2. TWO ARMS WOULD HAVE BEEN SILENT DUPLICATES. The first draft queued
mixer="full" and mixer="none", but model.py dispatches
`if is_full(i) -> attention; elif mixer=="swa128" -> SWA; else -> SSM`, so
both fall into the else branch and are byte-identical to the base arm. That
was ~33 GPU-h of duplicates. The all-attention control is expressed as
attn_every=1 instead. An all-SSM arm turns out to be unreachable at all
(is_full(i) = i % attn_every == 0 makes layer 0 full attention for every
attn_every >= 1) and is documented as such rather than faked.
3. THE ARMS WERE NOT COMPARABLE. Non-embed params differ by 10-20% across arms
(151.0M to 208.2M), so an iso-token ladder would have violated §C18's 5%
tolerance and could never have been called a result. Token budgets are now
matched on TOTAL train FLOPs, measured from the real built configs
(6N + 12*L_full*H*Dh*T + 12*L_swa*H*Dh*w): worst mismatch 0.17%. The
quadratic-attention term turns out to cancel the param difference for
attn1to3 and fullattn, while swa128 needs 1.129x tokens; rungs were chosen so
even that arm fits inside the 170.03M cache, so no arm wraps into a second
epoch.
The queue is 15 cells — 5 single-variable arms x 3 rungs (42M/85M/150M
base-equivalent), n=1 scout, ordered CHEAP RUNG FIRST so the 42M rung yields a
complete 5-arm comparison in ~22 h and partial completion is still a result.
Seeds 1-2 are appended later only for arms that separate (§C17).
run_arch_ladder.sh is the continuous driver: .done markers make it idempotent,
each cell gets its own sentinel on the real python pid, a cool-down gate keeps
it off a hot box, and it loops passes until every cell is done with an
escalating hot-spell backoff and a completion gate. Relaunching it is always
safe, so recovery needs no human decision.
Its §C4.5 guard also had to be fixed after the first launch attempt deferred all
15 cells: a bare `pgrep -f train_*.py` matches any process merely MENTIONING a
trainer, including the monitoring command watching it. sentinel's own preflight
reported trainers=none on the same second. The guard now requires argv[0] to be
a python interpreter, unit-tested against a shell decoy, a real python trainer,
and a clean box.
§C5.0 smoke: all 5 arm configs through the exact script, exit 0, 8.79-8.94 ->
0.0016-0.0054, ckpt round-trip max|Δ|=0.00e+00 — the only evidence attn_every=1
works, since verify.py's toggle sweep only covered attn_every in {2,4}.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The audit's #3 risk: ledger.json (the truth store) and the runs/*.md + briefs + digests (its human-readable trail) were git-untracked, living on one disk — and a branch switch already destroyed a ledger.json once (recovered from a git blob). Source .py/.sh became tracked yesterday; the record did not. Per the 2026-07-22 decision (batch 7 Q1), the record is now tracked too: ledger.json + research/ledger/runs/**/*.md + research/briefs + research/digests — 16 files, ~160K of text. Git history becomes the off-box backup; the rule is to commit the record before any branch switch (the hazard that bit us before). Kept deliberately local: loop_state.json (churns every loop wake — a stale tracked copy is worse than none), the single-generation ledger.json.bak, the dated backups/ snapshots, and the generated non-record artifacts (pulse/radar/provenance, datasets, checkpoints, logs). Verified by dry-run that all of those stay ignored. Also force-adds today's audit deliverables (LOOP_AUDIT + LOOP_UPGRADE_PLAN) so the 28-decision analysis survives. Note: 7 of 24 runs have a detail_md on disk; the other 17 dangling paths are a separate Tier-2 backfill item. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ctions The audit found the upload artifact stale: arxiv_package.tar.gz and the .bbl were 2026-07-07, while the sections + PDF were rebuilt 2026-07-20 — so the packaged tarball no longer matched the paper, and state.json still claimed phase 3 while Phase-6 artifacts (SUBMISSION.md, ledger papers[]=packaged) already existed. Per the 2026-07-22 decision (batch 7 Q3): rebuild the package; the human still does the actual arXiv upload (§C16, nothing auto-submits). - Rebuilt PDF + .bbl with tectonic from the current sources: 34 pages, 26 refs. - Verified arXiv-safe by a CLEAN-ROOM compile (extract the tarball, build with the shipped .bbl as arXiv AutoTeX does): rc=0, 34 pages, 0 unresolved [?] citations, References section renders. - Repacked arxiv_package.tar.gz (36 files) per SUBMISSION.md's recipe. - Fixed state.json to phase 6 (packaged — awaiting human submission), updated 2026-07-22, with the rebuild note. - Scrub check: the paper sections carry no leaked LLM meta-commentary (the meta-commentary the audit flagged is in the strategy docs, a separate item). Remaining is HUMAN-only: the arXiv upload using the metadata block in SUBMISSION.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…osite results) The audit's finding #5: "directional" meant two opposite things in the ledger — a genuine null (SFT-masking and GRPO found nothing) AND a big SIGNIFICANT effect capped only by a missing HARD item or n<3 (data-mix +0.59 code BPB, significant). A ledger reader couldn't tell "found nothing" from "found something big, one gate short." For a rigor factory whose product IS trustworthy negatives, that ambiguity is the core defect (batch 1 Q4). ledger.py VERDICTS now distinguishes: win — HARD-complete, significant, single-variable + iso-FLOP (§C18, unchanged) promising — a real measured (usually significant) effect, capped below win by a missing HARD item / n<3 / unresolved confound. NOT a never_repeat loss. null — measured; no effect beyond the noise floor. A first-class negative result, NOT a never_repeat loss. loss — worse than baseline; auto never_repeat (unchanged). inconclusive— could not be measured/interpreted (crash, undecidable confound, no comparand). directional — kept valid but DEPRECATED; new runs use null|promising. Relabeled only the crystal-clear historical cases (verifiable-accuracy — read each verdict.json, no guessing): - grpo-phase2 -> null ("PREDICTED NULL CONFIRMED" in its verdict.json) - sft-3seed -> null (masking does not beat the iso-FLOP control) - scaling -> promising (significant +0.474/+0.126 early, converges, n=2 cap) - hybrid pilot -> inconclusive (n=1, no comparand) 5 runs stay 'directional' pending a careful per-run read (imu1-deconfound, arch-subdrill, the two data runs whose verdict.json says "recipe-level" yet whose README records big significant effects — that contradiction must be resolved before relabeling — and cce). Regression test added; 42 ledger tests still pass. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… the silent no-op) The audit found run_arch_ladder.sh's completion hook was `[ -f score_arch_ladder.py ] && python score_arch_ladder.py` — but the scorer never existed, so a completed ladder produced NO comparable numbers and nothing said so. - score_arch_ladder.py REUSES the proven JAX suite functions from the pilot's eval_suite_jax.py (load_model / ppl / load_corpora), patching the per-arm config per cell — it does NOT re-implement scoring (the 6-copies-of-score_cohort anti-pattern the audit flagged). It reports cross-arm val PPL (the valid cross-arm metric: shared tokenizer/corpora/windows) + the emergence-speed curve (ppl vs log10 tokens per arm, gap-vs-base per rung). It deliberately does NOT stamp a hand-rolled BPB — cross-study BPB comes from the single consolidated eval-harness (upgrade-plan item 6), not a copy here. - --smoke runs the pure aggregation/curve math on CPU (no GPU, no model load); PASSES. GPU end-to-end validation is pending the first rung gap (§C4.5: score only when no trainer is live). - The driver hook now logs LOUDLY on a missing or failing scorer instead of silently passing — "ladder COMPLETE but UNSCORED" is a visible state, not silence. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…m-v3 status
Item 7 (batch 6 Q4): self-floor noise floors on undertrained checkpoints are
15-26% of PPL (measured on the HybridSSM pilot), making the standing baseline
useless for future significance calls. eval_stats.noise_floor(target, reference=None)
prefers a stable reference checkpoint's subsample spread over the target's wobbly
self-floor, and RETURNS the basis ("reference"|"self") so a wide self-floor is
never silently read as a real significance bar. Same max-min primitive as the
legacy self-floor, so it stays comparable to prior self-floored numbers. Tested.
On item 6 (one real eval-harness): the audit's "6 hand-rolled copies re-implement
the suite" is partly overstated — the canonical PRIMITIVES already exist and are
tested (eval_metrics.bits_per_byte, eval_stats.subsample_noise_floor /
seed_delta_significant), and the score_cohort copies import them. What actually
duplicates is the thin per-model windowed-CE score LOOP, which is legitimate
experiment-isolation glue. So the consolidation is smaller than framed; the real
integrity gap was the noise-floor basis (fixed above) and the suite stamp.
Also corrected a stale, contradictory line in the eval-harness suite.md status
table (local-only, under .claude/): text-lm-v3 was marked "flips ACTIVE on the
first GPU eval run / no v3 number yet", but the v3 downstream battery already
scored 25 checkpoints 2026-06-24. Flagged the deeper governance call (flip the v3
core, or keep v3 downstream-only) as a deliberate TODO rather than deciding it
unilaterally, and documented that BPB is the §C10 headline reported under v2.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…m-6 framing Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ed work The audit found next-best "actively wrong today": techniques that finished their runs were stuck at status=briefed, so the loop's selector kept offering finished work as runnable. Fixed the clear cases (verifiable from run_ids + run statuses): - normuon-optimizer briefed -> done (produced the win + scaling study) - vibethinker-small-reasoning briefed -> done (5/5 runs done) - midtrain-anneal-premium-mix briefed -> done (1/1 run done) - hybrid-attention-rethink briefed -> running (owns the live arch ladder) zeta-dual-whitening correctly stays briefed (0 runs — declined at the 2026-07-14 interactive gate, genuinely awaiting launch). next-best now returns only zeta-dual-whitening as the top runnable candidate, instead of a pile of finished techniques. Technique histogram is now honest: briefed=1, candidate=14, done=4, proposal=1, running=1. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…_env guards The audit found the newest, most box-crash-critical code had ZERO tests, and batch 7 Q2 chose "test the killers before the features." These are the code paths that prevent (or cause) a whole-box hard-lock, so they get tested first. Thermal-kill path (sentinel.py, added because the GB10 hard-locks from heat): - gpu_thermal() parses the nvidia-smi temp + hw/sw throttle CSV, and MUST fail open — a dead/absent nvidia-smi returns (None, False), never a fabricated high temp that would SIGTERM a healthy trainer. Tested incl. the exact "Not Active" != "Active" substring trap. - hottest_soc_c() picks the max ACPI zone and fails open on unreadable zones. - Constants guard: >=2 consecutive over-limit samples before a kill (a single nvidia-smi blip must not fire), WARN < KILL, ceiling set high on purpose. L1 memory guards (safe_cuda, jax_safe_env) — new test_guards.py, pure CPU (the validation/env logic runs before any torch/jax call, so no GPU needed): - safe_cuda.guard() rejects an unsafe fraction (<=0 or >0.95) BEFORE touching torch — the check that stops a 0.99/typo'd 1.5 from handing the whole unified pool to one process; default stays the §C1 0.85; import composes PYTORCH_CUDA_ALLOC_CONF. - jax_safe_env sets PREALLOCATE=false + MEM_FRACTION=0.5 on import, and REFUSES (RuntimeError) if PREALLOCATE=true is already set — the ~90GB startup grab that would crash the box. Full gate green: 433 passed, 1 skipped, with the trainer live. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… fuzzy title)
The audit found check-dup was exact-slug-string only — it depended on three separate
LLM invocations minting IDENTICAL slugs, so the same paper could enter the ledger
twice under different slugs (and a real technique already bypassed the eligibility
window). Duplicate briefs/runs cost real GPU (batch 4 Q4).
check-dup now, in addition to the exact-slug + never_repeat check:
- extracts the arXiv id from --source-url (version-suffix stripped, so v1/v2 dedup)
and flags a match against any existing technique's source_url;
- fuzzy-matches --title by normalized token Jaccard (>= 0.85), so punctuation/case/
spacing differences collapse.
Deterministic, no external fuzzy lib. Verified against the LIVE ledger: a re-entry of
zeta-dual-whitening (arxiv 2606.14187) under a new slug is caught by both id and
title, while a genuinely new candidate stays NEW.
New helpers arxiv_id() / norm_title_tokens() / title_jaccard() + check-dup gains
--source-url / --title. Tests added; ledger suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The audit found c5_evidence.json schema-free and drifting: the flagship win has NO c5 file, one 3-seed cohort has none, and the key naming varies so nothing could machine-check §C5 compliance (batch 3 Q3). This turns "smoke: pass" from prose into a checkable gate — validate_c5() reports which §C5 items (smoke/budget/probe/eta/ resume/sentinel/guards) are missing, so a launcher (or the adopted-run protocol for manual launches) can REFUSE to spawn on incomplete evidence. Accepts both historical namings (numbered c5_0_smoke and flat smoke) — the point is that the evidence exists in a machine-readable FIELD, not the key name; an item buried in free prose reads as missing, which is the correct signal. Verified on disk: both live HybridSSM c5 files PASS 7/7; the drifted scaling-persistence outlier FAILs informatively. CLI (exit 0/1/2) + importable validate_c5(); 6 tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The hand-launched HybridSSM arch ladder died: sentinel thermally killed 2 of
the 85M cells (2026-07-23 14:16/14:21Z) and the box then hard-locked at 15:24
BST. The safety system worked; the driver's cool-down gate and a batch of
"done" loop-hardening items did not. Recon (2 workflows, adversarially
cross-checked) found the real defects; this lands the fixes, each diff reviewed.
Ladder / safety:
- run_arch_ladder.sh cool_down: 3-min sustained-cool dwell, DEFER on the bounded
fall-through (engages the hot-spell backoff), honour the return value at the
call site, COOL_C 70->58 (= sentinel KILL 90 - measured +31C load transient - 1).
- score_arch_ladder.py resolved checkpoints under the wrong dir -> scored 0/7
cells and exited 0 (a silent no-op). Fixed the path, made it fail-loud, added
a pure-path smoke check.
- sentinel.py writes the kill marker BEFORE the SIGTERM->SIGKILL grace loop, so a
caller that reaps the watcher on trainer exit no longer loses it (+ ordering test).
- train_hybrid.py checkpoints/restores the PRNG key so a resumed cell continues
its exact data-window stream (round-trip verified bit-exact).
Loop integrity:
- eval_completeness: fix a disallowed-sole-signal floor BYPASS (fired only at
len(present)==1, so an n=1 val-PPL headline + any 2nd item reached promising/win);
now floors to inconclusive whenever no admissible signal remains. Verdict cap aligned.
- loop_state.py: parent-dir fsync + advisory lock + mode preservation (ledger.py
parity) + register() setter for the flat in-flight fields.
- ledger.py: unknown-run-key hygiene warning; cross-lane provenance keys allowlisted.
ledger.json: 17 dangling detail_md nulled, 33 stray eval keys moved into metrics{}.
- adopt_run.py (adopt + reconcile) + calibration_pairs.py (read-only join, n=0 today),
both with tests. Standalone for now; wiring into their callers is the follow-up.
- Docs reframed to "rigor factory / propose-only GPU"; S8 eval correctly called a
bounded preflight-gated GPU read, not a training launch.
Reconciled the dead run through the sanctioned CLI only: arch-ladder + orphan s1
runs -> crashed; technique hybrid-attention-rethink queued->running (it was wrongly
surfaced as a fresh next-best launch); loop_state in-flight pointer cleared;
sentinel_kill marker archived. sentinel liveness now exits 0. Full suite 537
passed / 1 skipped; ledger fsck clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Builds the launch-ready package for upgrade-plan #9 — the flagship "does NorMuon's win persist or converge with budget?" question — WITHOUT launching (GPU spend stays human-triggered). - run_ladder_scale_ext.sh: extension driver adding the 420M 3rd seed (takes the top rung from n=2 -> n=3 paired seeds, earning a real across-seed CI) + a 840M seed pair (a higher-budget trend point). Ships the FIXED cool-down gate ported from the arch-ladder recovery (sustained-cool dwell + DEFER + honoured return, COOL_C 70->58) so it cannot re-trigger the thermal thrash that preceded today's hard-lock. Reuses train_ablation.py + the CORE .done markers (done cells skipped). - c5_evidence_scale_ext.json: structured §C5 evidence, PASSES c5_validate 7/7. ETA/probe from the real measured 420M cell (6,685 tok/s, 51.5 GB, 17.5 h/cell). - §C5.0 smoke run + verified (exit 0, model built, loss moved, checkpoint saved). Carries an explicit thermal precondition: the box hard-locked today under a lighter load and the durable firmware fix is still pending — recommend the 420M pair first, firmware/kdump fix before the 840M rung. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…d behind firmware fix) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ty pgrep The inline check deferred every cell forever (an empty `pgrep | while` exits 0, so it read 'trainer alive' on an idle box). Replaced with the proper argv[0]-is-python function form from run_arch_ladder.sh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ked launches) cool_down compares to hottest_c = max(GPU die, ALL ACPI SoC zones); the SoC zones idle at ~64-68C, so COOL_C=58 could never be satisfied and deferred every cell forever. The +31C transient that argued for 58 was measured on a heat-soaked box mid-cooldown, not a cool one — which is exactly what the dwell rejects. 72C is above the reachable idle floor, 10C under WARN(82), 18C under KILL(90); proven CORE ran at 70C. Fixes both run_ladder_scale_ext.sh and run_arch_ladder.sh (same A-lane value). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…min) The authorized 420M pair launched and trained, but sentinel thermal-killed the first cell at SoC 90-91C after ~28 min (pool only 64% — pure heat). Same thrash pattern that preceded today's 15:24 hard-lock, on a heavier load. Stopped it for a thermal decision; user chose firmware/kdump fix first. - resume_every 200->100 so forward progress survives each ~28-min thermal kill. - kdump enablement scripted separately (scratchpad/enable_kdump_gb10.sh): root cause is USE_KDUMP=0 -> crashkernel=1G-:0M (0 reserved) in the kdump-tools grub drop-in. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ard kill User wants to run on-box without the reboot. New governor watches hottest_c and SIGSTOPs the trainer at >=85C (GPU idles, box cools, ZERO lost progress), rechecks every 3 min, SIGCONTs below 75C. Keeps the box out of the 90C zone that preceded today's hard-lock while making continuous progress; sentinel stays the memory + 90C last-resort guard, checkpoints (resume_every 100) the hard-lock backstop. SIGSTOP/SIGCONT decision logic unit-tested (pause->T, resume->R, never left stopped). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…vershot to 90) Live 2026-07-23 test: 596M + concurrent GPU work heats the box ~21C/min, so an 85C/30s governor overshot to 90-91C (the hard-lock line) before pausing. 80C/10s catches it at ~82-83C — a real margin below sentinel's 90C kill. Validated live that SIGSTOP cools 90->66C in 3 min and sentinel did NOT kill (governor got there first). ~25% duty at this thermal load; checkpoints (resume_persist_*.pt) preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… idle) User keeps heavy concurrent load permanently; the die barely cools, so the 3-min cooldown recheck left the trainer idle ~85-90% even after the die was cool. Switch to a tight 78-80C band with 5s rechecks: resume the instant it dips to 78C instead of waiting 3 min. Holds the die just under WARN(82), 10C under the 90C kill, and runs as much as the box's cooling allows (~3x duty). Safety unchanged (PAUSE_C + sentinel backstop). Overrides the earlier 3-min spec, which was the idle cause. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Measured the 80/78 retune: 50% duty (great) but peaked 88C (2C from the hard-lock line) due to ~8C thermal-inertia overshoot. Pulled to 76/72 + 3s run-sampling -> peak ~82-84C (~7C margin), still ~40% duty since the die cools fast to ~50C idle. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…val) Traces each number a HuggingFace model card would carry back to the dataset id, config, split, seq len, stride, tokenizer and results file that produced it, plus Reproduce / Training-details / checkpoint-inventory / loader-API sections. Produced by an 8-dimension parallel file audit with an adversarial verification pass per dimension; three load-bearing findings were re-verified by hand. Three findings contradict what the repo currently asserts: - Qwen3-0.6B/README.md:35-37 claims all four Qwen3 perplexities use "the identical 300k-token FineWeb-Edu val slice". They do not. 13.40 and 46.31 come from tokcache_133072000_300000.pt (hardcoded at eval_original_vs_repro.py:22); 28.65 and 23.52 come from tokcache_1191478400_300000.pt. So 28.65/13.40 and 23.52/13.40 are cross-slice ratios, not like-for-like gaps. - The -0.474 bpb NorMuon result is still advertised as a "significant win" in four places, but the scaling-persistence ladder closed 2026-07-28 with trend_verdict=CONVERGES / ledger_verdict=null: the gap converges away with budget (0.474 -> 0.126 -> 0.072). It holds at a 42M-token budget only. - The "max error 0.0 / bit-exact" claim is CPU-fp32-only on a single 5-token prompt. The repo's own GPU per-layer delta is 1.95e-03, which trips its own 1e-3 gate. 13.40 is confirmed as our own measurement of the released Qwen3-0.6B-Base on this box (2026-06-09), not a figure copied from the tech report; only the "36T tok" label is borrowed. Also records the live environment stamp (python 3.12.11, torch 2.11.0+cu130, CUDA 13.0, cuDNN 91900, GB10 driver 580.142) since the repo pins versions in one place only and bit-exactness is version-sensitive, and the honest gap list: no commit hash stamped in any results file, no dataset revision pinned for fineweb-edu, and zero determinism flags repo-wide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MODEL_CARD_FACTS.md is the synthesis; this is everything under it, unabridged: 160 extracted facts (value, evidence path, verbatim source quote, self-assessed confidence, caveat) each paired with the ruling the adversarial refute pass returned, plus all 57 gaps the repo cannot answer. Nothing is dropped. Verifier entries with no 1:1 extracted fact are reproduced per dimension under "Additional verifier findings", so all 166 verdicts appear: 157 attached to a fact, 9 standing alone. The refute pass overturned 9 facts outright and qualified 38 more, so 47 of 166 checks caught something that would have been misleading if published as first extracted -- which is why the raw pass is worth keeping alongside the synthesis. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every replacement number was re-derived from the artifact on disk, then checked by an adversarial pass whose instructions were to refute it. Three rounds were needed: the first two introduced errors of their own, which are also fixed here. Comparability (the load-bearing one): - Qwen3-0.6B/README.md claimed all four Qwen3 perplexities use "the identical 300k-token FineWeb-Edu val slice ... every row is directly comparable". False. 13.40 and the Phase-A sweep sit on tokcache_133072000_300000.pt (hardcoded at eval_original_vs_repro.py:22); 28.65 / 23.52 / 29.54 sit on tokcache_1191478400_300000.pt. Replaced with a per-cache table. The derived 2.14x and 1.76x gaps are relabelled cross-cache wherever they appear (Qwen3 README, root README, both plots READMEs, both build READMEs); 46.31/13.40 = 3.46x is same-cache and kept. NorMuon: - The -0.474 bpb result was advertised as a "significant win" in four places with no mention of the ladder that nulls its persistence. Now scoped to its 42M budget everywhere, with a new "Scaling persistence" section carrying the full n=3 table. The ledger verdict `null` is attributed to the ladder run, not to 2026-06-16_qwen3_normuon-vs-adamw (which is `win`). - The root README's ladder block was stale in five specifics (n=2, +0.073 [-0.038,+0.184], "not significant at the top", code +0.192, slope -0.328). Current: n=3 at every rung, +0.072 [+0.055,+0.088], all six rungs significant, code +0.177, slope -0.342 (r2 0.84). - "Falls within the noise floor" is now stated as what it is: the OLS-fitted edge at the top rung (wikitext 0.0297 vs 0.0368 floor), not the measured gap. On code the fitted edge 0.1255 vs 0.0463 is still resolved, so the corpora are no longer described with one blanket claim. Parity: - "bit-exact / max error 0.0" now says fp32-on-CPU, 5-token prompt, everywhere it appears. For SmolLM2 the GPU numbers are given (final-logits 4.72e-05, per-layer 1.95e-03 at layer 14 - which exceeds the repo's own 1e-3 gate); for Qwen3 it is stated that no GPU parity check exists. Arithmetic and citations: - NorMuon wall-clock "~30% more" -> +43.9% (-30.5% throughput, 5,172 vs 7,444). - Faithful build: 7,480 tok/s was the step-100 reading, final is 7,444. - Modernized build: 1,191,478,400 was the token-cache size; trained budget is 1,189,478,400 (18,150 x 65,536). - SmolLM2 demo loss 6.321 -> 6.288 (min 6.039 @ step 140); "agree to 6 decimal places" -> 5; wikitext double-count ratio 1.97x; tokenization sourced to results.ipynb + POST_DATA.md, not summary.json (which has no such key). - CI lower bound 0.444 -> 0.443 at three sites. - params 596,049,920 no longer attributed to verify.json, which has no such field. Stale status: - Data arm, mid-training and the 3-seed SFT were all done but still listed as running/planned. The n=1 VibeThinker SFT number is marked superseded by 2026-06-27_qwen3-0.6b_sft-3seed and relabelled in-loop, not held-out. - "MC accuracy is near-chance (no-signal)" was wrong: only WinoGrande is at chance; ARC-easy and HellaSwag carry signal:true above their 0.25 floor. - tinystories_summary.md documents an EARLIER run (3.7893 / 137.3 min / 12,150 tok/s) than the committed artifacts (3.7900 / 116.1 min / 14,356 tok/s); it is now labelled as such rather than having one number swapped. - PLOTS_INDEX plot count 74/72 -> 79/79. Known remaining gap: the committed overview figures still print "Published Qwen3-0.6B-Base = 13.40" and annotate 2.21x. Both are wrong, both live in the PNG/PDF and make_overview_plots.py rather than in a README, so they are disclosed in the captions and left for a regeneration pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…e count The 1.316 figure was the (23000,24000] bucket; the true last bucket (24000,24414] over 414 rows is 1.3138, matching tinystories_summary.md. The 198-line row cited wc -l model.py for a file that does not exist in this tree (only model_full.py does), so the row is removed rather than guessed.
1.316 was the (23000,24000] bucket. The final bucket (24000,24414] is 1.3138, which is what tinystories_summary.md already reports.
The architecture is in model_full.py. Keeping the recorded param count, dropping the unrunnable command.
No train_flops artifact exists for any Phase-B run, so the C18 5% iso-FLOP gate was never evaluated. 1.00043 is the +0.077% parameter-count ratio used as a proxy. The three sites that presented it as a measured FLOP ratio now say so; references to the gate as a protocol are left alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The default branch currently contradicts the published Hugging Face model cards. This brings
mainup to date.Why this matters
mainis 35 commits behindharden-research-loop. Anyone landing on the repository today readsQwen3-0.6B/README.mdwith the claim that the faithful baseline (28.65) and the released Qwen3-0.6B-Base (13.40) were scored on an "identical 300k-token FineWeb-Edu val slice". They were not: the val caches differ (sha18ad9e246b0bf63bdvsad3513719d0f81e4), so the 2.14x and 1.76x ratios derived from that pair are cross-slice and have been retracted. Only 46.31 / 13.40 = 3.456x is same-slice.What the correction pass fixes
nullwc -l model.pymodel.pyexists in this treetrain_flopsartifact exists, so the C18 5% gate was never evaluatedDeliberately not changed
warmup 50in the NorMuon-vs-AdamW ledger entry andRESULT.mdis correct. Those files document the 640-step / 42M-token ablation cells, which ran a cosine schedule with 50 warmup steps. The 900-step warmup belongs to the 18,150-step Phase-B runs. The two describe different experiments, and reconciling them would introduce an error rather than remove one.The generated
c5_evidence.jsonfiles still contain1.00043. The value is right; only the prose describing it as a FLOP measurement was wrong. Editing a machine-written evidence artifact to match prose would be worse than the prose.Verification
Every figure above was checked against the source artifact it cites rather than against a summary. The three Hugging Face model cards were rewritten from the same evidence and now agree with this branch:
https://huggingface.co/bishnoiyash/Qwen3-0.6B-reproduction
https://huggingface.co/bishnoiyash/Qwen3-0.6B-imu1
https://huggingface.co/bishnoiyash/SmolLM2-135M-reproduction