Conversation
Resolves the core PoC blocker: CommunicationAdapter was producing cross-bucket cosine = 1.0000 (constant output for any input), causing circular prose. Cross-bucket cos now 0.5676 (target ≤ 0.85) with coherent, differentiated codebase-adjacent prose. Training changes (Interlat paper prescription): - Revert input-dependent queries in CommunicationAdapter (architectural change backfired — cross-bucket cos 1.0→0.96, within-bucket collapse) - Add L_xsep (cross-bucket cosine separation proxy, λ=0.1) to adapter Phase 1 training in _train_basic; reduce LAMBDA_DIV 0.5→0.2 - Re-enable _SEP_LAMBDA=0.1, _ALIGN_LAMBDA=0.05 in LoRAReceiverTrainer - Switch lora_trainer Phase 2 from margin_separation_loss → separation_loss - Decouple training: 5 epochs adapter-only, 1 epoch LoRA-only (prevents catastrophic forgetting from lora_r=16 × 131 samples × 3 epochs) Unit test fixes (4 failures → 570/570 passing): - test_embedding_service: mock accepts **kwargs (EmbeddingService now passes device="cpu" to SentenceTransformer) - test_latent_strategy: decode() embed_tokens call count 2→3 (bop + eop + unconditional assistant cue at latent_strategy.py:332) - test_lora_trainer: perturb wrong_embed by randn*0.5 so JSD(p,p)≠0 after switching from margin_separation_loss to separation_loss - config.py: base_model default Instruct→Base (Qwen/Qwen2.5-0.5B) — receiver must use the Base model per V2 architecture Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Remove duplicate entries from .gitignore (.libucks/, __pycache__/,
*.egg-info/, .venv/ each appeared twice).
Git hooks installed via `libucks install-hooks`:
.git/hooks/post-commit, post-checkout, post-rewrite
→ fire `libucks hook <event>` on every commit to keep buckets
current via StartupRecovery diff-and-replay.
Socket round-trip verified: hook sends {"event":"post-commit"} JSON
over Unix socket; listener receives correctly. Integration tests for
the full commit→DiffExtractor→StartupRecovery→Librarian chain: 5/5.
Registry last_indexed_head set to HEAD so MCP server restart will
not replay the full commit history from April.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Mismatch caused _sync_setup() to crash on load_state_dict shape mismatch, leaving server.sock uncreated and _load_error set on every startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
DeepMind cache-aug architecture (2412.17747) as a 5th eval path on the
fresh uncontaminated repo echoswarm:
- libucks/cache_augmentation/{kv_extract,bucket_kv_cache,coprocessor,
fusion,decode}.py — per-bucket KV cache, 202M coprocessor, 101M
cross-bucket fusion, augmented-cache decode
- cache_aug_trainer.py + train-cache-aug/generate-qa CLI commands
- Cold Stop entropy-gate decode (Soft Thinking §3.3): cache_aug
1/25 -> 12/25, cos 0.126 -> 0.561, beats hybrid 9/25 on echoswarm
- query-aware Librarian, novel bucket service, chunk retriever
Verdict accepted as the cache-aug headline; decode-fairness, no-verbatim
ablation, and libugry cross-val remain open limitations (see
docs/phase-4c-log.md 4-C.6-SALVAGE / 4-C.7).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Added cache_aug_no_verbatim + cache_aug_greedy_nogate paths (same 3B bundle) and ran the echoswarm fairness eval: - Cold Stop is a red herring: greedy_nogate 12/25 == gated 12/25 (the 1->12 jump was model.generate -> manual greedy loop, not the gate) - latent channel is inert: no_verbatim 2/25 < no_context 3/25; all grounding comes from verbatim - cache_aug 12 vs hybrid 11 = +1 (within noise); hybrid cos 0.626 > 0.561 Verdict corrected to 4-D-C future-work / negative result. Phase 4-A hybrid stays the production path and writeup headline. Log 4-C.7 revised. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fresh 'Cartridge Memory' track supersedes V1/V2/Phase-4 docs.
- Archive to docs/archive/: IMPLEMENTATION_PLAN, V2_IMPLEMENTATION_PLAN,
V1 QUICKSTART, INTERLAT_LITE_MANUAL (stale Phase-12 runbook),
LATENT_RAG_ARCHITECTURE (superseded latent-LoRA design record),
and docs/phase-4c-{plan,log} (closed negative-result phase)
- Promote QUICKSTART_V2 -> QUICKSTART (single current quickstart)
- Consolidate 15 arXiv paper markdowns into docs/papers/ (deduped 2412.06769)
- Seed docs/cartridges-plan.md + docs/cartridges-log.md
- Repoint CLAUDE.md sections 1 & 3 and ARCHITECTURE.md tree to the new plan/log
Root .md files: 11 -> 6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trainable per-bucket KV-prefix (Cartridges recipe, arXiv 2506.06266): per-layer (1, n_kv_heads, P, head_dim) K/V, warm-start from kv_extract, grad-preserving to_dynamic_cache, safetensors save/load. 6/6 contract tests green. First building block of the CM-A context-distillation path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- losses.distillation_loss: KL(full-context teacher || cartridge student) - self_study: scaled per-bucket synthetic query gen (model + template fallback) - CartridgeTrainer: teacher greedy-gen w/ logit capture, cartridge-prefix student forward, KL(+CE) distill into prefix only; MPS grad guards mirrored from cache_aug_trainer; generate_answer for latent-alone decode - KVPrefixCartridge.for_model constructor - smoke (0.5B): base frozen, cartridge moves, KL 5.48->4.02->3.56 monotonic Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Context distillation works as a mechanism (KL 0.916->0.517, latent-alone grounding 1/8->2/8, coherent on-topic answers) but at P=64 / 128 templated queries / 2 epochs the latent carries structure not precise identifiers — the Phase-4A two-channel decomposition reproduced under a proper objective. Gate needed >=3/8. Levers to try: fact-probing queries, larger P, more epochs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fact-probing self-study queries (vs generic templates) + P=128 + 4 epochs:
latent-alone grounding 2/8 -> 4/8, KL 0.749 -> 0.219. Cartridge now carries
specific facts ("80%", garble, STRANDED) that v1 fabricated. Confirms query
coverage was the bottleneck, not a fundamental ceiling.
Two trainer bugs fixed:
- per-step torch.mps.empty_cache() (fragmentation hang at ~step 180)
- precompute teacher answers once + single teacher-forced forward per step
(was regenerating deterministic greedy every epoch; ~2h -> ~30min)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- cm_distill_buckets.py: resumable per-bucket distillation of all fixture- routed echoswarm buckets (skip-if-exists); saves <bucket>.cartridge.safetensors - cm_eval_cartridge.py: latent-alone grounding across all 25 fixtures -> CM-A.2 gate number (cartridge >= 8/25 vs no_context 3/25) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A bucket whose first chunk exceeds max_chars previously yielded "" because the overflow branch dropped the block entirely. Truncate to the remaining budget instead, so verbatim is never empty. Fixes latent distillation on large-header buckets. Adds unit test; also tightens teacher-precompute log cadence 40->10 for finer overnight progress. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…rainer - distill_bucket(checkpoint_path=...): save cartridge after every epoch; batch script writes <bid>.cartridge.ckpt.safetensors, removed on success - faulthandler watchdog re-armed per step: a >5min stall dumps all thread stacks every 5min instead of hanging silently (2 prior silent MPS wedges) - torch.mps.synchronize() every 20 steps to bound Metal queue depth - 2 orchestration unit tests (stubbed heavy paths); suite 592 green Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Records the reproduced CM-A.2 result: latent-alone grounding 7/25 vs gate >=8/25, bit-identical across two evals (stable, not variance). Includes the fe7ded0d redistill history (r1-r6) and the finding that PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 is causally necessary for any distill/eval run on this machine. - docs/cartridges-log.md: CM-A.2 entry + TOC line - tests/eval/results/cm/echoswarm_cartridge_A2.json (eval r2, new cartridge) - tests/eval/results/cm/echoswarm_cartridge_A2_r1_fail7.json (eval r1, preserved) - .gitignore: cm_*.log run logs, .claude/scheduled_tasks.lock - untrack docs/.DS_Store (already gitignored) Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Declutter (25k lines removed, no behaviour change; unit suite 592 passed /
1 skipped, identical to pre-clean baseline):
- untrack vendored mps_bitsandbytes-0.7.0/ + tarball. Both importers
(model_manager.py:236, lora_trainer.py:36) guard with try/except ImportError
and fall back to unquantized; every eval config sets quantization = "none",
so no reported number depends on it. Now gitignored.
- untrack .claude/worktrees/{gallant-bardeen,keen-curran} — orphaned gitlinks
(mode 160000, no .gitmodules) that rendered as broken folders on GitHub.
- delete main.py — 12-line click stub registering no commands; the real entry
point is libucks/_cli.py via [project.scripts].
- archive POCSTRAT.md ("Phase B is complete", stale by 5 phases) and PITCH.md
(duplicated two README H2s, described LoRA rank 4 and a Watchdog that
`libucks serve` no longer starts).
- QUICKSTART.md -> docs/RUNBOOK.md, keeping the deep runbook detail (daemon
mode, stdout/stderr split, Claude Desktop wiring) rather than dropping it.
- pyproject.toml: drop the bogus bitsandbytes>=0.43 darwin/arm64 requirement
(upstream has no MPS backend) and document why.
- ARCHITECTURE.md: repo tree pointed at the deleted main.py.
README: replace the "PoC Status" table with an honest phase-by-phase results
section carrying BOTH negative results (4-C latent-alone 2/25; CM-A.2 7/25),
the correct per-repo model story (3B receiver produced the 19.5/30 headline;
repos without config.toml silently fall back to 0.5B), and the cross-model
baseline caveat.
Add scripts/run_eval.py: reproduces the headline with --runs N, sets
PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 before any torch import, and prints/warns
about the resolved receiver so a 0.5B-default run can't be mistaken for the
3B headline configuration.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… 7/25 -> 10/25, GATE PASS _grounding_score did plain case-insensitive substring matching, so a correct answer stated in a different surface form scored as wrong. Three of CM-A.2's 18 failures were correct answers: echoswarm_01 expected "80%" model said "relay probability ... is 0.8" echoswarm_02 expected "2" model said "at least two different sources" echoswarm_03 expected "1" model said "randomly changing one character" Re-scoring every stored answer under the corrected metric: cartridge (CM-A.2) 7/25 -> 10/25 (+3) gate >= 8/25: FAIL -> PASS cache_aug_no_verbatim 2/25 -> 3/25 (+1) hybrid, text_clean, no_context, latent, cache_aug: unchanged The old metric reproduces the logged 7/25 exactly, so the re-score is trustworthy. Baselines move by at most +1, so this is not general inflation — their failures are vague or wrong answers, not correct facts in another format. New: - libucks/eval_metrics.py — keyword_variants/keyword_hit/grounding_score. The literal keyword keeps plain substring matching so no previously-passing fixture can start failing and historical numbers stay comparable; ADDED variants (percent<->decimal, number word<->digit) match on word boundaries only, because keyword "1" generates variant "one" which is a substring of "money". - tests/unit/test_eval_metrics.py — 19 tests written first, watched fail with ModuleNotFoundError. Carries the four real CM-A.2 cases as regressions: 01/02 must flip to grounded, 06/10 must stay failures. - scripts/cm_rescore_grounding.py — re-scores stored answers under both metrics. Modified: - tests/eval/test_latent_vs_baseline.py:451 delegates to the shared function. - docs/cartridges-log.md — CM-B.0a entry; CM-A.2 marked superseded. Still owed, and NOT claimed here: the cartridge ran on 3B while hybrid / text_clean / no_context ran on 0.5B (echoswarm has no config.toml), so 10/25 vs hybrid 11/25 is cross-model and must not be quoted as a comparison. Phase 4-C stays negative — cache_aug_no_verbatim re-scores to 3/25, exactly the no_context floor. Unit suite 611 passed / 1 skipped (592 pre-existing + 19 new). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t-probing ones
scripts/cm_distill_buckets.py:96 called
generate_self_study_queries(verbatim, N_QUERIES, model=None)
With model=None, generate_self_study_queries skips _model_queries entirely and
falls through to generic templates (self_study.py:178-185). CM-A.1 had already
established that templated queries were the bottleneck — 2/8 templated vs 4/8
fact-probing, logged as "query coverage was the bottleneck" — and
cm_proof_single_bucket.py:126 passes a real generator. The batch script that
scaled that finding to all ten buckets reverted the exact variable the finding
was about, while its log entry recorded "fact-probing self-study".
Corroborating: CM-A.1-retry ran KL 0.749 -> 0.219; CM-A.2 ran 3.690 -> 2.685, a
~5x higher starting error, which is what vague template questions produce.
Changes:
- Load Qwen2.5-0.5B-Instruct as the query generator and pass it through.
- Two-phase structure: generate ALL queries first with only the 0.5B resident,
free it, then load the 3B. The two models are never co-resident — on 16 GB
that is the difference between running and wedging.
- PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 set before the torch import (proven
causally necessary in CM-A.2: r4 wedged without it, a clean probe passed with
it in 62 s).
- --buckets to restrict to specific bucket id prefixes, --force to re-distill
over an existing cartridge, --dry-run to resolve the work list without
loading any model.
- Resume/skip logic moved ahead of model loading so a fully-satisfied run costs
no model load at all.
- Docstring records the detached launch form; a non-detached run dies with its
parent shell (this killed CM-A.2 r5 mid-precompute).
Verified: --dry-run resolves the three CM-B.0b spot-check buckets
(40615ba9, bc6b90e2, fe7ded0d), ~4.1 KB verbatim each.
Not yet run — distillation is the next step.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…borts on MPS
The CM-B.0b query-gen fix crashed on first launch:
MPSNDArray.mm:761: failed assertion
[MPSTemporaryNDArray initWithDevice:descriptor:] Error: total bytes of NDArray > 2**32
This is a hard Metal assertion that aborts the process, not a Python exception,
so it cannot be caught, retried, or worked around in the loop.
Diagnosed rather than guessed. Ruled out:
- logits size — 0.5-0.7 GB at the real prompt length, far under the 4 GB ceiling
- dtype — bf16 crashes identically to fp32
- prompt length — still crashes at ~780 tokens
- the sampling path — greedy, plain sampling, top-k and top-p ALL crash
Each config was run in its own process (the abort kills the interpreter):
greedy CRASHED sample_topk CRASHED
sample_notopp CRASHED sample_topp CRASHED
cpu OK, 9.5 s
So the fault is transformers' generate() on MPS at this prompt length, full
stop. Query generation now runs on CPU: ~10 s per call, a few minutes per
bucket, and it removes all contention with the MPS allocator.
Distillation is unaffected and stays on MPS — CartridgeTrainer._teacher_generate
uses a manual decode loop rather than generate(), which is why CM-A.2 ran at
all. Eval is unaffected for the same reason plus much shorter prompts.
Also added: abort if any bucket's query generation comes back empty, so the run
fails loudly instead of silently falling back to templates — the exact
regression CM-B.0b exists to fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scripts/cm_eval_cartridge.py:40 had a private _grounded() doing plain substring matching — the exact metric CM-B.0a proved under-counts by 3 fixtures. Had the Stage 0b eval run against it, the result would have been scored under the old metric and been incomparable to the 10/25 established in CM-B.0a, silently undoing that work. Now delegates to libucks.eval_metrics.grounding_score, with a docstring warning against reintroducing a local copy. Also: - --buckets to evaluate only fixtures routing to given buckets, so the Stage 0b spot-check scores just its 13 fixtures. Subset runs print the correct bar (CM-A.2 was 2/13 old metric, 5/13 CM-B.0a metric) and explicitly suppress the >=8/25 gate verdict, which does not apply to a subset. - --tag so a subset run cannot overwrite the full-run results JSON. - PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 before the torch import; the eval had been relying on the caller to set it. - Results JSON records buckets_filter and metric:"cm-b.0a" so future readers know which scorer produced the number. - Header now flags that the quoted baselines are 0.5B while this script hardcodes 3B, i.e. cross-model and not a valid comparison. Eval generation is unaffected by the MPS generate() bug: it goes through CartridgeTrainer.generate_answer, a manual decode loop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…oks viable
docs/cm-b-plan.md: ports the approved CM-B plan into the repo so it survives the
session. Logging deliberately continues in docs/cartridges-log.md rather than a
separate cm-b-log.md, so the CM track (CM-0 -> CM-A -> CM-B) reads end to end.
Records Stage 0c (cross-model baseline) as an explicit TODO that blocks all
cartridge-vs-baseline comparisons.
scripts/cm_probe_slot_structure.py: CPU-only, no model load, safe to run while a
distill occupies the GPU. Answers the precondition for Stage 1's research bet —
slot-localized repair is only possible if the P prefix slots are differentiated
rather than collapsed.
Result on the two CM-A.2 backups (both nearly identical, so this is structure and
not noise):
slot_cos_mean eff_rank (of 128)
fe7ded0d k +0.550 74.0
v +0.141 96.9
bc6b90e2 k +0.556 68.6
v +0.135 86.7
V slots are strongly differentiated — mean pairwise cosine 0.14 and ~97 of 128
effective dimensions. K is less so (+0.55, ~74), which is expected given the
shared rotary/positional structure in keys; one layer does reach +0.994, i.e.
near-collapsed K.
So distillation did NOT collapse the prefix, and slot-localized repair is worth
building. Caveat recorded in the script: differentiation is necessary but not
sufficient — it does not show slot i corresponds to source position i. That test
needs to re-extract the bucket KV on the 3B and correlate, so it waits for the
GPU.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every bug found so far falls into three classes: silent defaults that flip semantics, duplicated logic that drifts, and manual cross-file invariants. Swept all three. 1. SILENT CORRUPTION IN KVPrefixCartridge.load (real bug, fixed) load() went straight to `param.data.copy_(flat[...])` with no validation. Tensor.copy_ BROADCASTS, verified empirically: copying a (1,1,P,D) source into a (1,2,P,D) destination SUCCEEDS and duplicates head 0 across both heads, silently corrupting the prefix. A file with more layers than the object silently loads only a prefix of them. This is live risk, not hypothetical: libucks runs two receiver geometries (3B = 36 layers/2 heads, 0.5B = 24 layers) and PREFIX_LEN is duplicated across scripts under a "MUST match" comment. save() has ALWAYS written n_layers/n_kv_heads/prefix_len/head_dim as safetensors metadata — load() simply never read it. It does now, with a per-tensor shape fallback for files lacking metadata, and errors that name both geometries and the file. 6 tests added, written first, 4 of which failed before the fix. Verified against real cartridges: a 3B file loads, and loading it into a 0.5B-shaped object is now refused loudly. 2. THIRD COPY OF THE STALE GROUNDING METRIC (fixed) scripts/cm_proof_single_bucket.py:52 still had the plain-substring scorer. Now delegates to libucks.eval_metrics. Its docstring records that the CM-A.1 numbers (2/8, 4/8) were produced under the old metric and are therefore lower bounds. 3. THE DEFAULT THAT CAUSED THE CM-A.2 REGRESSION (warned) cm_proof_single_bucket.py:45 defaults CM_MODEL_QUERIES=0, i.e. templated queries — the configuration CM-A.1 proved fails. Copying that default is how the batch distiller ended up on templates. Default kept so the original v1 run reproduces, but it now logs a loud warning naming the 2/8 vs 4/8 result. CHECKED AND CLEAN: - PREFIX_LEN (128) and RECEIVER_ID (3B) agree between cm_distill_buckets.py and cm_eval_cartridge.py, so the running job is unaffected. - Both _lr_lambda copies are correctly 1-indexed `(step + 1)`; the warmup off-by-one did not survive anywhere. - Both _collect_source_text copies carry the slice-on-overflow fix. NOTED, NOT FIXED (outside the experiment path): startup_recovery.py has several `except Exception: pass` paths returning None/[] that would mask update-pipeline failures. Unit suite 617 passed / 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…e bug _compute_coherence called embed_batch directly with no caching, once per bucket per health pass, every 300s forever. Measured live: the libucks-on-libucks server held a performance core at 58% CPU re-embedding all 159 buckets while the distill job it competed with got 8% (load 7.66 / 8 cores). ChunkRetriever already keys embeddings by (chunk_id, git_sha) and was already constructed in the bridge — HealthMonitor simply was never given one. The part that is not just wiring: the two components use different _read_chunk_content variants. ChunkRetriever is CONTENT-family and omits chunks whose source file is unreadable; HealthMonitor is GEOMETRY-family and embeds the file path instead, on purpose, so dead chunks do not collapse onto one degenerate vector. Reusing the cache naively would have changed what a dead chunk contributes to coherence and therefore changed split decisions. So: cached vectors for every chunk the cache knows, geometry path fallback for every chunk it does not. Coherence is bit-identical; the work is not. A cache failure now falls back to the old uncached path instead of returning None. Returning None disabled the coherence trigger while the size trigger kept working — "splitting quietly stopped happening", near-undiagnosable. Caught while wiring it: chunk_retriever is built inside _load_heavy's nested _sync_setup(), whose values reach the outer body only through its return tuple. Referencing it outside without extending that tuple is a NameError at `libucks serve` time and completely invisible to the suite, because _load_heavy loads real models and is never unit-tested. Added the missing return value, plus a static AST test that fails on ANY name built inside _sync_setup, used outside it, and not returned — the whole class, not just this instance. Also killed two stale `libucks serve` processes (libucks-self, and a click server running since Sunday); load 7.66 -> 5.53. Verified the running distill job is untouched by any of this: its transitive import closure is 13 modules and contains none of merging_service, mitosis, health_monitor, chunk_retriever, _cli or mcp_bridge; no server targets echoswarm; its buckets are unmodified since before the job started. 670 unit passed / 1 skipped; 22 model-free integration passed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d config Structural pass over libucks/ + scripts/ + tests/: import graph, runtime cycles, layer ordering, unreferenced functions, and every config key checked for an actual reader. The architecture itself is clean: no runtime import cycles, no layering violations, and every production module except _cli.py has a test. The one TYPE_CHECKING/runtime mismatch flagged (mitosis:171 calling Librarian) is a false positive — there is a deliberate function-local import at :169 to break the cycle. Detector refined to honour local imports; it then reports none. margin_separation_loss has NEVER been called, in the entire git history. Added by 03a9db5, imported by lora_trainer.py:33, invoked nowhere, and untested — test_lsep_loss.py covers only separation_loss. This matters more than a normal dead function: the loss actually in use is JSD, which has zero gradient once the two distributions coincide, i.e. exactly the sep=0.0000 collapse CLAUDE.md says to halt a run over. margin_separation_loss was written to escape that state. Documented in place, NOT wired in — it changes the LoRA objective and needs its own gate and retrain. BucketKVCache.invalidate() also has zero callers despite a docstring saying "call on mitosis / merge / removal". Staleness detection still works, but lazily: load() recompares the chunk signature and returns None. Corrected the CM-B plan, which claimed the cache is marked stale ON those events. Measured the consequence rather than assuming: 0 real orphaned cache files across echoswarm and libugry. Deliberately NOT wiring invalidate() in — eager deletion on split/merge is the throw-it-away behaviour Stage 2 exists to replace with repair. Four more inert config keys beyond compression_steps: PathsConfig's grammar_cache, log_file, pending_events and repo_cache are documented in ARCHITECTURE.md and read by no production code. No behaviour changes in this commit; docs and one docstring. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two things, both aimed at the same worry: how do we know there are not more bugs? Answer: stop hunting them by hand and make the hunt executable, then write the new code test-first. tests/unit/test_repo_invariants.py — the four manual sweeps, as tests. Re-runs every structural check on every commit: no runtime import cycles, no upward layer imports, no helper defined in two modules, no config key without a production reader, no TYPE_CHECKING name called at runtime, one and only one grounding-metric implementation, and PYTORCH_MPS_HIGH_WATERMARK_RATIO set before torch is imported in every script that touches MPS. Each carries an allowlist of known, accepted exceptions with reasons, so the tests are about CHANGE: they pass now and fail when something new appears. Verified the guards actually bite rather than merely passing: injected a duplicate encode_centroid into mitosis.py and an unread config key into config.py, confirmed three separate tests failed, then restored both. scripts/cm_make_edits.py — Stage 1, step 1. Generates the four controlled single-chunk edits (rename / constant / branch / delete) the Edit experiment needs, since real history cannot supply them: libugry has one commit and echoswarm's fourteen are almost all README touches. Pure Python, no torch, so it runs alongside the distill job. Properties the tests pin, each one a way the experiment could silently produce a meaningless number: an edit must actually change the file (a no-op yields a free "repair worked!" datapoint), must stay valid Python, must be confined to a small span, must revert byte-for-byte so ground truth and every repair method start from an identical tree, and must be deterministic given a seed. Three real defects found by running it against echoswarm rather than only the fixture. Target discovery walked tree.body only, so it saw four module-level functions and missed twelve methods — including _relay_messages and _move_agents, the exact identifiers the eval fixtures probe; rename and delete were simply unusable on the files this experiment needs. The branch planner would emit `if self is None: return None` inside methods. And delete could remove a class's only method (SyntaxError) or a 65-line function spanning several chunks, which stops testing single-chunk repair at all — now capped at MAX_DELETE_LINES=25, after which it selects _relay_messages (15 lines). End-to-end verified on a clone of echoswarm: all four types apply, commit, move HEAD, and leave the file parseable. 738 unit passed / 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage 1, step 3: the three ways to repair a stale cartridge instead of
re-distilling it (~7,200 s measured).
continue warm-start, retrain only on queries whose teacher answer
changed; every parameter still trainable. Honest baseline.
slots retrain only the prefix slots the edit touches. The research
bet — a cartridge has no index, so whether knowledge about one
chunk is localised at all is the open question.
lowrank freeze the cartridge, learn an additive correction. Fallback.
Deliberately no forward passes here. Deciding WHAT to retrain is separable
from running the training, is CPU-testable, and is where the experiment can
quietly produce a meaningless number — so it lives in its own module with the
invariants pinned. Orchestration goes in scripts/cm_edit_experiment.py.
Two ways this experiment could lie to us, both now blocked:
An edit that moves no teacher answer makes every method report a wonderful
cost ratio for doing nothing. changed_queries(require_change=True) raises
NoChangeDetected naming the staleness floor, and RepairResult.is_uninformative
flags any zero-query row so it cannot be read as a success. Comparison is
whitespace-normalised so decode jitter does not inflate the changed set and
with it the measured cost.
A "slot-localized" method that in fact updates every slot is continue-training
wearing a hat, and would make the headline claim false. SlotMask zeroes
gradients outside its selection, and a test drives a real SGD step to assert
frozen slots do not move. The docstring records that this is gradient masking,
not true freezing — momentum or weight decay can still nudge a zero-gradient
parameter, so the measurement needs plain SGD or weight_decay=0.
LowRankDelta zero-initialises A and randomises B so the product starts at
exactly zero: the wrapped cartridge is untouched until training begins.
Zero-initialising BOTH factors is the obvious way to guarantee that no-op and
is a trap — d(AB)/dA = B = 0 and d(AB)/dB = A = 0, so the delta is frozen at
zero forever and "the fallback didn't help" would be an initialisation
artifact. Verified by injecting the degenerate init: the no-op test still
PASSED and only test_becomes_non_zero_once_trained caught it, which is why
both exist.
Fixed while testing: slots_for_char_span promised in its docstring to widen
the range by a slot on each side and did not. Widening is now an explicit
`pad` argument defaulting to 0, since every extra slot weakens the localized
claim and that trade-off should be a recorded choice at the call site rather
than a hidden default. char_end is documented as exclusive.
Real-geometry check (36 layers, P=128, 2.36M params): lowrank r=4 is 32x
smaller; a single-chunk edit selects 10/128 slots (7.8%); gradient selection
recovers a planted 40-49 signal exactly.
778 unit passed / 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Stage 1, step 4. Per (edit type, seed): apply the edit, diff teacher answers to find what actually moved, build ground truth ONCE, run every repair method against it, revert. Model work goes through a TrialRunner protocol so the orchestration — matrix, ordering, resume, gate arithmetic — is CPU-testable with a fake. The gate is where a result gets declared and should not be exercised only by a multi-hour GPU job. Resolved an ambiguity in the plan's gate wording rather than leaving it to be settled after seeing the data. "Within 10% of full-re-distill KL" now means repaired_kl <= redistill_kl * 1.10, both being each cartridge's own distillation KL against the teacher, so they are like-for-like and their ratio is meaningful. KL(repaired || re-distilled) is still recorded as agreement_kl — the plan calls it primary and it carries no decode variance — but it is NOT the gate, because it has no natural scale to threshold against. Ordering is a cost decision: a full re-distill is ~7,200 s and dominates everything else, so the matrix groups all methods under one (edit type, seed) and ground truth is computed once per group. Interleaving by method would roughly triple the wall clock. A test asserts the grouping holds. Two ways the experiment could report a false success, both blocked. A group whose edit moves no teacher answer is skipped entirely rather than recorded, so a zero-query trial cannot sit in the results looking cheap and successful. And the gate refuses any trial whose staleness floor is within 25% of the re-distill: if the STALE cartridge already matches ground truth, the edit disturbed nothing and there was nothing to repair. That is the plan's pre-committed honest outcome, enforced rather than remembered. Edits are always reverted in a finally block. Without it a failed repair leaves the edit in the tree and every later group stacks on top of it, quietly invalidating the rest of the run. Resume is per-trial JSONL, fsynced, one line each, and a truncated final line is skipped with a warning instead of being fatal — verified by simulating a kill mid-write. Stage 0b already lost a bucket to a write-only checkpoint. Documented a gate property that changes experiment design, not just arithmetic: SEED_MAJORITY is a strict majority, so with 2 seeds BOTH must pass (1/2 = 0.5 is not > 0.5) while 3 seeds allow 2-of-3. Ground truth is ~16 h for 4 types x 2 seeds and ~24 h for 3, so the cheaper option is also the harsher gate. Both cases are pinned by tests. run_experiment was initially written untested; caught it, wrote the contract first, and rewrote it. The rewrite fixed a placeholder that passed an empty changed-query list to every repair. 818 unit passed / 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CM-B.0b re-distilled bc6b90e2/40615ba9/fe7ded0d with a 0.5B query generator and scored 1/13 against a 5/13 baseline — a regression, not a near miss. Six of thirteen answers are degenerate token loops. Root cause of the mistaken premise: `_TEMPLATES` has been the FACT-PROBING set since bdba4ae (2026-07-02), the commit whose message reads "Fact-probing self-study queries (vs generic templates): 2/8 -> 4/8". The 2/8 loser was the generic template set, deleted in that same commit. So `model=None` was the WINNING configuration, not the losing one. A comment added yesterday (12ca750) claimed the opposite and motivated this whole stage; CM-B.0b then displaced proven templates with 0.5B output and fell 5/8 -> 1/8. Diagnostics (zero compute): - The 4096-char verbatim cap discards 72% of 40615ba9 and 66% of fe7ded0d. Cross-bucket KL is not comparable while truncation ratios differ this much. - Eval ceiling is 20/25 (10/13 on the subset); five fixtures cannot be answered from the routed bucket's kept verbatim. Raising the cap fixes only two of them — the other three are routing defects. - Truncation is NOT what blocks 40615ba9: all six identifiers echoswarm_11 needs were inside the text query-gen read and the cartridge distilled, at KL 0.227, and the answer invented every key. "Structure, not identifiers" under the cleanest possible conditions. - KL is decoupled from grounding in both directions. Code: - generate_self_study_queries gains an optional `stats` out-dict. The function always tops up to n, so callers could not tell model from template queries; the `if not qs` guard in the distiller could never fire and gave false assurance while bc6b90e2 trained on 30% templates. - cm_distill_buckets: CM_MODEL_QUERIES / CM_MAX_ANSWER_TOKENS / CM_MAX_TEMPLATE_FRACTION levers, all defaulting to existing behaviour, plus a RECIPE banner — CM-A.2's log claimed a configuration it did not run. - distill_bucket gains opt-in `best_path`: CM-B.0b promoted bc6b90e2's epoch 3 (5.2284) when epoch 1 was 4.1816. Kept separate from the rolling checkpoint so `epochs_done` still describes the saved weights and resume cannot rewind. Opt-in because best-of-N selection is a protocol change. 818 -> 836 unit tests, 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Fifth bug sweep, aimed at this repo's demonstrated failure classes rather than a generic checklist. Two findings, both made executable. 1. NOTHING in the CM pipeline was seeded. cartridge_trainer shuffles the query order with the global `random` every epoch; neither cm_distill_buckets nor cm_eval_cartridge seeds anything; only scripts/diagnose_adapter.py seeds at all, anywhere in the repo. So every headline in this track — 2/8 vs 4/8, 7/25 vs 10/25, 5/8 vs 1/8 — is one sample against another with unmeasured variance and no error bar. This outranks any individual bug found so far: it means we cannot currently tell a real delta from noise. distill_bucket gains `seed`, using a dedicated random.Random so reproducibility does not depend on global RNG state another import could have advanced. cm_distill_buckets exposes CM_SEED and reports seed=... | UNSEEDED in the RECIPE banner. Unset = historical behaviour. 2. `PREFIX_LEN = 128` was declared in cm_distill_buckets and cm_eval_cartridge under reciprocal "MUST match" comments — while cartridge.py's own load() docstring called such a mismatch "a question of when, not if". A P sweep would have made it when. The geometry was already being written into every safetensors file and simply never read back. Adds KVPrefixCartridge.read_geometry(path), which raises rather than guess when metadata is absent. The eval now derives P per cartridge and logs when it differs from the default, so cartridges trained at different P are evaluable with no code edit. The distiller keeps PREFIX_LEN as the single source of truth, now CM_PREFIX_LEN-tunable. Invariants added to test_repo_invariants.py so neither class can return silently: no constant may be restated under a MUST-match comment, the eval must call read_geometry, distill_bucket must accept a seed, and the batch script must both expose and forward it. The first of these caught the distiller half of the PREFIX_LEN pair that the manual fix had missed. 836 -> 849 unit tests, 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… silently
Sweep via ruff (RUF006). asyncio holds only a WEAK reference to a running Task,
so a task nobody else references can be garbage-collected mid-execution — the
CPython docs say plainly to save a reference. libucks had six bare
asyncio.ensure_future() calls, none of them storing one.
The two riskiest are one-shot coroutines doing real work with many await points:
librarian.py:245 -> MitosisService.split() (bucket silently never splits)
query_orchestrator.py:52 -> stale-bucket reindex (staleness silently persists)
The other four are in mcp_bridge: the git-hook socket server (the entire update
pipeline), HealthMonitor.run and NovelBucketService.run — both named in
CLAUDE.md as core mechanisms — and _load_heavy. A loop parked in asyncio.sleep
is usually kept alive by its timer handle, so those are latent rather than
guaranteed; the failure mode when it does bite ("auto-mitosis just stopped, no
error anywhere") is close to undebuggable.
Second defect at the same sites: a bare ensure_future swallows exceptions until
GC, surfacing as asyncio's context-free "Task exception was never retrieved".
Adds libucks/background_tasks.spawn(coro, name=...), which keeps a strong
reference until completion, logs failures immediately with the task name and a
consequence field, and treats cancellation as normal shutdown rather than an
error. A test asserts none of the three modules reopens the bare form.
Also adds scripts/cm_seed_variance.sh: chains two seeded draws after the running
unseeded one, each distill -> eval -> archive, so this track finally gets an
error bar. A failed distill skips its eval rather than scoring the previous
draw's stale cartridge.
Triage note — ruff reported 99 findings; most were not bugs. All 7 F821
"undefined name" are string annotations with function-local imports, and all 3
zip-without-strict sites are provably equal-length (one is explicitly guarded by
a length check on the preceding line). Yield was one real class out of 99.
849 -> 858 unit tests, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…andidate
scripts/cm_variance_report.py turns repeated draws of one config into the error
bar this track has never had. It reports score spread AND per-fixture churn
separately, because they disagree: two fixtures swapping verdicts leaves the
headline N/8 unchanged while making every per-fixture claim irreproducible. With
2-3 draws it leads with the range rather than a stdev computed from three points,
and --claim-delta says outright whether a proposed delta sits inside the noise.
Two findings while validating it against existing results.
1. The eval is NOT bit-deterministic. echoswarm_cartridge_A2.json and
A2_r1_fail7.json are the same cartridge evaluated twice: verdicts identical
(7/25 both), but the ANSWER TEXT differs on echoswarm_10 and echoswarm_16.
The CM-A.2 log's "bit-identical across 2 evals" is wrong — the score was
identical, the text was not. Most likely MPS reduction ordering flipping an
argmax. Small, but it means the seed draws measure distill variance plus a
little eval noise, not distill variance alone.
2. The verbatim-slicing candidate for the CM-A.1-retry non-reproduction is real
but too small to be the cause. Post-fix verbatim ends mid-identifier
("if self.agent_") where pre-fix ended at a clean chunk boundary
("return None"). That does leak into query generation: _extract_identifiers
gains 'agent' and 'bool' and loses 'n_drop' and 'pop', and 5 of 200 template
queries end up anchored on 'bool' — "What is the exact numeric value,
probability, or threshold associated with bool?" is nonsense and its teacher
answer will be high-entropy. But 2.5% of queries cannot explain an 8x jump in
initial KL (0.749 -> 5.7629). By elimination this strengthens the remaining
candidate, unmeasured run-to-run variance, which the queued seed draws test.
858 -> 869 unit tests, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two prep items for after the seed draws land. 1. CM_VERBATIM_CHARS makes the slice-on-overflow question a one-command experiment. It also unifies three literals that had to agree by hand — the text collected from chunks, the teacher's context window, and the KV extraction length — since a mismatch means the teacher answers from text the cartridge never encoded. CM_EXTRACT_TOKENS derives from it by default. Verified: CM_VERBATIM_CHARS=3868 reproduces bc6b90e2's pre-30ee434 verbatim BYTE-FOR-BYTE (3898 chars, ending at the clean chunk boundary "return None" rather than mid-statement at "if self.agent_"). Found while verifying: _collect_source_text sums block lengths but joins with a 6-char separator, so it overshoots max_chars by (blocks-1)*6 — +36 at the 4096 default. Left as-is and documented, because changing it now would shift every verbatim length and break comparability with the runs in flight. Corrects an earlier figure of mine: pre-fix verbatim is 3898, not 3868; I had computed it without separators. 2. Sixteen new fixtures for bc6b90e2, taking it from 8 to 24 items. Motivation: four nominally-similar configs scored 1/8, 2/8, 4/8 and 5/8 on this bucket. A spread of 4 on an 8-item test means nothing in this track currently clears its own noise floor, and standard error falls as 1/sqrt(n), so more items beats more recipe tuning. Deliberately a SEPARATE file with a --fixtures flag, because appending would change the denominator of every logged number (7/25, 10/25, 5/13). The eval warns when a non-default set is used. Held to the standard the original 25 never were: test_fixture_reachability checks every fixture clears the 50% grounding bar from the KEPT verbatim, that at most two sit exactly on the boundary, that ids cannot collide with the main set, and that each question actually routes to bc6b90e2. That last check caught echoswarm_x15 routing to ff8a1e0c; the fixture was reworded, not the test relaxed. Five of the original 25 fail these same checks. 869 -> 883 unit tests, 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…(spread 1) Three seeded draws of the exact CM-A.1-retry recipe on bc6b90e2 (200 queries, max_answer_tokens=48, fact-probing templates, P=128, 4ep, last-epoch save): 2/8, 1/8, 1/8. mean 1.33, range 1-2, spread 1, stdev 0.58. First error bar this track has ever had. The pre-registered read was "spread >=3 means every finding collapses into noise". It is 1, so the findings are real — but they are not the findings we attributed them to. THE RECIPE IS THE PROBLEM. CM-A.2 ran Jul 7, four days AFTER the slice-on-overflow fix, so it had the identical verbatim, templates, P and epochs. It differs from today only in query count (120 vs 200) and answer budget (32 vs 48) — and scores 5/8. Per-fixture, today's config loses echoswarm_02, 03 and 04 in ALL THREE draws and gains nothing: a monotone loss, not sampling. The "proven" recipe is reproducibly worse than the configuration this stage set out to fix, and CM-A.2 — declared a failure, then re-scored to 10/25 GATE PASS — is the best result this project has produced. Ruled out as causes: verbatim slicing (CM-A.2 had the same post-fix text) and environment drift (torch 2.11.0 May 14, transformers 5.4.0 Mar 31, Qwen2.5-3B snapshot 3aab1f19 Apr 4 — all predate Jul 2). CM-B.0b's collapse was also not mainly the 0.5B generator: templates at 200/48 give 1/8 too. Two independent changes were each harmful. Three corrections recorded: * KL is NOT comparable across runs with different query sets, since it is measured against that run's teacher answers and the identifier list shifts. An earlier claim that "the 20x KL gap is too stable to be noise" was unsound; the score comparison, called weaker at the time, is what held up. * The eval is not bit-deterministic — same cartridge, two runs, identical verdicts (7/25) but different answer text on echoswarm_10 and _16. CM-A.2's "bit-identical across 2 evals" is wrong about the text. * _collect_source_text overshoots max_chars by (blocks-1)*6; pre-fix verbatim is 3898 chars, not the 3868 quoted earlier (that omitted separators). Hypothesis: at P=128 the prefix cannot absorb 200 queries' worth of signal and 48-token answers dilute it further, so more self-study makes the cartridge worse. CM-A.2's own recommendation to scale queries to 512-1000 is withdrawn. cm_seed_variance.sh generalised: config now comes from the environment (CM_TAG, CM_SEEDS, and the CM_* levers) instead of being hardcoded, with required-var guards, and each draw is scored on BOTH the 25-fixture set and the new 16-fixture extension set. Next: CM-B.0d, queued — CM-A.2's exact config across 3 seeds. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Primary evidence for the CM-B.0b-repro entry in docs/cartridges-log.md, which until now cited numbers nobody could verify. Matches existing practice for echoswarm_cartridge_A2.json / _r1_fail7.json / _B0b.json. s0_unseeded 2/8 s1 (seed=1) 1/8 s2 (seed=2) 1/8 Each carries per_question verdicts and answer text, so the per-fixture claim in the log — that echoswarm_02, 03 and 04 fail in ALL THREE draws where CM-A.2 passed — is checkable rather than asserted. They also preserve the raw answers behind the degenerate-output observations, and the `metric` field records that all three were scored under the CM-B.0a grounding metric, not the older substring one. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…random prefix
The number that makes CM-B.0d interpretable. Three arms on the SAME questions
with the SAME decoder, on the b0d seed-3 cartridge:
orig 8 ext 16
floor (no prefix) 0/8 4/16
random (untrained) 1/8 2/16
cartridge 1/8 5/16
cartridge - floor +1 +1
cartridge - random +0 +3
TWO FINDINGS, and the first is about fixtures added earlier today.
1. The extension set LEAKS. Its floor is 4/16 — the base 3B answers x09, x11,
x12 and x15 correctly with no memory at all, because questions like "what are
the five states an agent can be in?" invite guesses that happen to be right.
So CM-B.0d's 4.33/16 was mostly the model answering from the question. Against
a 4/16 floor the cartridge is +1, the same as the original set — not the
better-powered signal the set was built to provide. The leaky items need
replacing before that set is worth quoting.
2. On the trustworthy fixture set, distillation buys nothing. An untrained
random-init prefix of identical geometry scores the same 1/8 as the cartridge
that took ~90 min to distill. The +1 over floor comes from having 128 extra
attendable positions, not from learned content.
Implementation: generate_answer now accepts cartridge=None for the floor arm, so
floor and cartridge differ in exactly one respect — the presence of the prefix.
Tokenisation, query template, greedy selection, EOS handling and mask width are
shared by construction rather than by copying, which is what makes the delta
attributable. 8 tests pin that, including that attention width excludes
prefix_len on the floor and that both arms build identical prompt text.
Cross-check: cm_floor's cartridge arm independently reproduces
cm_eval_cartridge's numbers for the same cartridge (1/8 and 5/16), and
.cartridge.safetensors is byte-identical to b0d_s3.bak.
EVAL_MAX_NEW_TOKENS moved into eval_metrics.py and read by both eval scripts.
Answers generated with different token budgets are not comparable — a shorter
answer has fewer chances to emit the required keywords — and test_repo_invariants
caught me creating exactly the reciprocal-constant pattern it was written to
forbid. Fixed at the source rather than silenced.
Also commits the six CM-B.0d draw snapshots and both floor snapshots as evidence.
890 -> 898 unit tests, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…/8, +0.33/16 Three seeds of the floor harness across three arms settles it. ORIGINAL 8 FIXTURES floor random cartridge c-floor c-random seeds 1/2/3 mean 0.00 0.67 0.67 +0.67 +0.00 EXTENSION 16 FIXTURES seeds 1/2/3 mean 4.00 2.33 4.33 +0.33 +2.00 cartridge - floor is +0.67/8 and +0.33/16 — zero at a spread of +/-1. On the non-leaky set the base 3B scores 0.00/8 cold, and the cartridge, holding the bucket's entire source distilled into 2.36M parameters, scores 0.67/8. CM-B.0d also shows CM-A.2's config does not reproduce: 0/8, 1/8, 1/8 against its historical 5/8. Six modern draws across two configs all land at 0-2/8 against two historical single draws of 4/8 and 5/8, and no artifact from Jul 2 or Jul 7 survives to diagnose against. TWO CORRECTIONS TO WORK DONE THE SAME DAY: 1. cartridge - random is a misleading statistic, and I introduced it. It was meant to separate learned content from the mere presence of P extra attendable positions. But a random prefix is ACTIVELY HARMFUL — 2.33 vs a floor of 4.00 — so it is a negative baseline, not a neutral one. The +2.00 mean mostly measures the cartridge undoing the damage a random prefix does, ending back at roughly floor level. cartridge - floor is the only defensible statistic; the single-seed "+3" reported earlier must not be quoted as a positive signal. 2. The CM-B.0b-repro entry's conclusion that "200/48 is reproducibly worse than 120/32" was wrong. 120/32 measures 0.67 and 200/48 measures 1.33 — both configs land in the same place. A one-fixture difference was read as signal immediately after warning against exactly that. CONSEQUENCE: Stage 1 (Living Cartridges) is dead as designed. Its premise is cheap repair of a cartridge when its bucket changes; if the cartridge contributes ~0 over no memory, repair is repairing nothing and the relative claim is trivially true and uninteresting. Do not build the model-backed TrialRunner. SCOPE: this is evidence about context distillation at LAPTOP SCALE (Qwen2.5-3B, P=128, ~90 min/cartridge), not about the idea. It is decisive only about whether the technique is available to libucks. Tooling: cm_floor_all_seeds.sh runs the arms against every archived cartridge of a config, saving and restoring the in-place file under a trap so an interrupt cannot leave the wrong cartridge installed under the canonical name (verified byte-identical after the run). cm_distill_buckets gains CM_KEEP_BEST, which promotes the lowest-mean-KL epoch and logs loudly that such a run is not comparable with last-epoch runs. cm_b0f_levers.sh queues the two surviving objections. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…/12 at P=128
Records CM-B.0f and adds a leak filter that makes the extension set usable.
CM-B.0f, both arms on CM-A.2's config at seed 1, scored against all three floor
arms:
best-epoch (P=128) epochs 5.684/3.990/4.065/4.039, PROMOTED epoch 1
c-floor +2 (orig 8) and +2 (ext 16)
P=384 (last-epoch) epochs 3.625/3.342/3.319/4.784, shipped the WORST epoch
c-floor +3 (orig 8) and +5 (ext 16)
Initial KL fell 5.68 -> 3.63 from the capacity change alone, and the P=384 arm
shipped 4.784 when its epoch 2 was 3.319 (keep_best was off for that arm), so
there is headroom above these numbers.
P is a POSITIONAL BANDWIDTH constraint, not a parameter-count one. At P=128 the
cartridge holds 2.36M parameters for 4,132 characters — ~570 per character,
wildly overparameterised. What it lacks is places to look: the verbatim is ~1,009
token positions and attention can only reach P of them. That is also the cleanest
account of the recurring "structure, not identifiers" failure — structure
compresses, literal strings do not.
CORRECTION: the CM-B.0e verdict should read "inert AT P=128". It was right about
the configuration measured and wrong as a claim about the channel. The estimate
that these levers had a ~15-20% chance of overturning the negative was too
pessimistic. Stage 1's "dead as designed" call is SUSPENDED pending CM-B.0g.
LEAK FILTER. The extension set's floor is 4/16, so full-set scores mix cartridge
performance with the model's prior. cm_leak_filter.py excludes any fixture the
floor ever answered, deciding the set ONCE across all runs rather than per run —
per-run selection would move the denominator between arms, the same mistake as
changing the fixture file mid-track. On the resulting 12 items the floor is 0/12
in every run, and the picture sharpens rather than softens:
P=128 last-epoch (3 runs) 1, 2, 1 /12
P=128 + best-epoch 2 /12
P=384 6 /12 <- 3-4x anything at P=128
Also unchanged and still true: cartridge - random remains unquotable, because a
random prefix is actively harmful and therefore a negative baseline.
Interpretive limit recorded for the sweep: bc6b90e2's verbatim is ~1,009 tokens,
so P=384 is 2.6x compression and P=768 only 1.3x. As P approaches seq_len the
cartridge stops compressing the KV cache and starts reparameterising it, at which
point training-free cache pruning is the simpler answer. P=768 is an upper bound,
not a proposal.
898 -> 902 unit tests, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ction arms TWO PIECES, both prep for when the GPU frees up. 1. FIXTURE LEAK, root-caused and fixed. CM-B.0e measured the extension set's floor at 4/16. Three of those four leaks were pure keyword echo: x09 had 3 of 4 answer keywords sitting in its own question, x11 3 of 4, x12 2 of 3 — each enough to clear the >=50% bar by parroting the question, with no memory involved. x15 leaked differently, through generic type names (int, bool) that a model guesses for any signature question. All four replaced with facts whose keywords appear nowhere in the question (_seen_sources/repr/init, the WAITING-vs-INFORMED comments, the node_id / origin_node_id / route / route_index group, total_key_tokens). Two invariants now enforce the rules in test_fixture_reachability: echo alone must not reach the grounding threshold, and generic type names must not be load-bearing keywords. All 16 fixtures pass those plus the existing answerable-from-kept-verbatim and routes-to-bc6b90e2 checks. Stated honestly in both the test and the fixture file: the echo rule is NECESSARY BUT NOT SUFFICIENT. x15's failure mode is not statically detectable, so these replacements still need a floor re-measurement before being trusted. 2. cm_kv_prune.py — does distillation beat slicing the REAL cache? Distillation has to relearn identifiers from a few thousand supervised tokens; a slice of the real KV cache already contains their activations, which is exactly the "structure, not identifiers" failure mode. And with bc6b90e2's verbatim at ~1009 tokens, P=384 is only 2.6x compression and P=768 is 1.3x — as P approaches seq_len a learned prefix stops compressing the cache and starts reparameterising it, where gradient-free pruning (SnapKV/H2O/StreamingLLM) is the established approach. Four training-free arms at the trained cartridge's own P: kv_first, kv_last, kv_stride (uniform coverage including both endpoints), kv_norm (top-P by ||K||, no attention rollout needed). All loaded into a KVPrefixCartridge and generated through generate_answer, so the decode path is shared with floor and cartridge. kv_first is provably identical to init_from_extracted_kv — a test asserts it — which makes cartridge minus kv_first a direct measure of what the gradient steps actually add. That number has never been measured in this project. Selector tests pin that position ORDER is preserved: cache keys carry RoPE positional information, so returning indices out of order would hand the model a scrambled document while still producing a plausible score. 902 -> 934 unit tests, 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… prefix wins
THE RESULT. kv_first — the untrained first-P slice of the REAL extracted cache,
provably identical to init_from_extracted_kv — beats the distilled cartridge on 3
of 4 measurements:
floor kv_first kv_last kv_stride kv_norm cartridge
P=384 ext16 0 12/16 0 7 0 9/16
P=384 orig8 0 2/8 0 3 0 4/8
P=768 ext16 0 13/16 1 1 1 8/16
P=768 orig8 0 3/8 1 3 0 2/8
98 minutes of gradient descent leaves the cartridge WORSE than where it started.
And the positive result is the bigger one: 13/16 against a 0/16 floor from a single
forward pass, degrading gracefully to 12/16 at 2.6x compression.
This retro-explains the entire track. KL improves while grounding does not because
training overwrites real activations that already carried the answers. "Structure,
not identifiers" because the real cache HAS the identifiers and distillation washes
them out. More capacity does not help because it is more room to overfit. And
CM-A.1-retry never reproduced because it was likely never far from the warm start.
CM-B.0g set this up: P=768 reached the best KL ever recorded here (1.093) and
scored 2/8, worse than P=512's 4/8 on a 3x worse KL. Capacity was A constraint;
removing it exposed the real one, which is supervision coverage — 120 self-study
queries that the fixtures do not ask about.
MECHANISM: contiguity. kv_first >> kv_stride >> kv_norm ~ kv_last. The stride arm
at P=768 takes 768 of 1009 positions with irregular gaps, scrambling the offsets
RoPE encodes, and collapses to 1/16. kv_last drops the file opening where the enum
definitions most fixtures ask about live.
METHODOLOGICAL GAIN: the training-free arms have no seed and no optimiser, so their
only variance is MPS decode noise. They are reproducible in a way no cartridge here
has been, retiring the spread-of-3-to-4 problem behind every earlier number.
CORRECTIONS: "200 queries vs 120 makes no difference" was measured at P=128 where
capacity was binding — that test is void, not negative. And CM_SEED gives only
approximate reproducibility: identical config and seed produced epoch-0 KL of
3.6252 vs 4.8253, a 33% MPS swing, far larger at P=384 than the 1.2% at P=128.
CONSEQUENCE: Stage 1 (Living Cartridges) is closed, for a better reason than
failure. Its premise was that rebuilding a bucket's memory costs ~7,200 s, making
cheap repair a research question. If memory is a truncated real cache, rebuild is
one forward pass and the problem does not exist. Do not build the TrialRunner.
Adds cm_kv_sweep.py to map the compression/accuracy curve over
P in {32..768} — minutes per point instead of 98, and deterministic. It reuses
cm_kv_prune's tested selector rather than copying the slicing logic, since a second
copy is exactly how the two grounding scorers drifted apart.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… distillation
kv_first swept over P on both fixture sets, floor 0 on both.
16-ITEM SET (floor 0/16), monotone with a knee at 128->256, saturating by 384:
P compression score
32 31.5x 1/16
64 15.8x 3/16
128 7.9x 8/16 <- equals the distilled cartridge, free
256 3.9x 11/16
384 2.6x 12/16
512 2.0x 12/16
768 1.3x 13/16
distilled cartridge (P=768, ~98 min) 8/16
As retention of the full-cache result: 3.9x keeps 85%, 7.9x keeps 62%. P=768 buys
one fixture over P=256 for 3x the memory, so the useful operating point is 256-384.
8-ITEM SET: no curve at all — 2/8 at P=32 and 2/8 at P=384. A 32-position cache
cannot contain a 1009-token file, so that is not a weak signal, it is proof the
8-item set has NO RESOLVING POWER at n=8. It can neither corroborate nor refute
the curve.
WHICH FORCES THE CENTRAL CAVEAT: the clean curve comes entirely from 16 fixtures
I authored against this bucket's verbatim. They ask about directly-stated facts —
enum members, field names, comments — which is precisely what a raw cache slice
should support. The set I did NOT author is too coarse to check that work. So the
defensible claim is narrower than 13/16:
for DIRECT FACT RETRIEVAL, raw KV-cache truncation shows a clean
compression/accuracy curve saturating near 2-4x, and at 7.9x compression it
equals a 98-minute distillation.
Whether it holds for synthesis or multi-hop questions is untested, and the orig8
set — which contains those — cannot tell us.
The binding limitation is now EVALUATION, not mechanism: one bucket, one repo, 24
fixtures, 16 of them mine.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ad-clustering artifact
WHY. The bc6b90e2 extension set produced an apparently clean compression curve
for kv_first truncation. It was an artifact. The scores track the count of
fixtures whose keywords fall inside the first P tokens almost exactly:
P positional ceiling actual score
128 8/16 8/16
256 11/16 11/16
384 14/16 12/16
Those questions clustered in the head of the file, and kv_first cannot know
anything past P — it is TRUNCATION, not compression. A set that only asks about
the head cannot tell the two apart. "3.9x compression retains 85%" was wrong; the
correct statement is "11 of my 16 questions are about the first quarter of the
file."
What survives is the head-to-head: the distilled cartridge saw the WHOLE document
and still lost 8/16 to a slice that discards 76% of it. That does not depend on
keyword position.
THE NEW SET. 26 fixtures for 95c8e099 (simulation.py, 20,013 chars / 4,599
tokens) — a different bucket and a different file from the one the mechanism was
developed against. Target facts spread across the whole token range, one per
~250 tokens from ~115 to ~4,818, enforced by tests: not clustered in the head
(<=35% answerable from the first eighth), a quarter requiring past the midpoint,
spanning opening to closing, and no gap over 25% of the document. At P=128 of
4,599 only 2.8% of the file is retained, so truncation should score near zero and
any arm above that is doing real work.
ROUTING IS DELIBERATELY BYPASSED. 14 of the first 20 questions routed to other
buckets, because simulation.py is the orchestrator and overlaps semantically with
most of the repo. Every fixture now pins its bucket, and cm_kv_sweep honours the
field. This experiment asks whether a bucket's CACHE carries its content, not
whether the router finds the bucket — routing is measured separately (Phase 1:
1/15 -> 14/15) and conflating them would make a cache result depend on retrieval
quality while silently dropping most fixtures.
TWO BUGS FOUND IN MY OWN WORK WHILE BUILDING THIS:
1. The min_P helper used int(len/2)-1 where grounding_score needs CEIL(len/2)
hits. Off by one for odd keyword counts, understating min_P and making
fixtures look more head-available than they are. It had hidden two late
fixtures from the gap check. Recomputing the bc6b90e2 ceiling with the correct
index changes 9->8 at P=128 and 15->14 at P=384: the retraction above STANDS.
2. cm_kv_sweep inherited max_chars=4096 from the distillation path, which capped
a 20,013-char bucket at its first 1,021 tokens — defeating the point of picking
a large bucket and putting most stratified fixtures outside the cache entirely.
Raw extraction has no teacher-context constraint; the budget now defaults to
unbounded.
934 -> 955 unit tests, 1 skipped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…50%. Retracts 0h
Position-stratified sweep on a SECOND bucket (95c8e099 / simulation.py, 20,013
chars / 4,599 tokens), 26 fixtures spread from token ~115 to ~4,818 with the spread
enforced by tests.
P compression score fraction retained
floor — 0/26 —
32 143.7x 0/26 0.7%
64 71.9x 1/26 1.4%
128 35.9x 1/26 2.8%
256 18.0x 2/26 5.6%
512 9.0x 2/26 11%
1024 4.5x 6/26 22%
2048 2.25x 8/26 45%
4599 1.0x 13/26 100%
distilled cartridge (P=128) 2/26
1. PREFIX TRUNCATION DOES NOT COMPRESS. Score is proportional to fraction retained,
near-linearly. At 36x it sits at the positional ceiling. There is no regime where
a small prefix carries information about text it dropped.
2. THE UNCOMPRESSED CEILING IS 13/26 (50%). With the entire document present as KV,
unmodified, the 3B answers half and fails half. This bounds every compression
scheme; the best compressed arm at any real ratio is 6/26 at 4.5x.
3. DISTILLATION ~= TRUNCATION at matched size: 2/26 vs 1/26, both near zero.
RETRACTS CM-B.0h. "98 minutes of distillation leaves the cartridge worse than its
own initialisation" (13/16 vs 8/16) was an artifact: those fixtures cluster in the
first quarter of agents.py and kv_first keeps the FIRST P positions, so it won by
containing the answers. At matched P on stratified fixtures the cartridge is if
anything marginally ahead. The 0h entry now carries an inline RETRACTED banner —
its table is real, its interpretation was wrong. The precursor "compression curve"
is withdrawn for the same reason.
Both retracted claims came from fixtures written by reading the target bucket. That
is the third distinct way self-authored fixtures measured the wrong thing this
session, after the keyword-echo leak (floor 4/16) and the 8-fixture set's absent
resolving power (2/8 at both P=32 and P=384).
Two tooling bugs, both mine: min_P used int(len/2)-1 where grounding_score needs
ceil(len/2) hits, understating head-availability; and cm_kv_sweep inherited
max_chars=4096, capping a 20,013-char bucket at its first 1,021 tokens. Correcting
the index changes 9->8 at P=128 and 15->14 at P=384 — the retraction stands either
way.
STAGE 1 CLOSED in cm-b-plan.md, because the premise dissolved rather than failed.
Cheap repair is only a research question if rebuild is expensive; rebuild is now one
forward pass. Scaffolding stays in the tree — edit generation and staleness
detection are reusable if a working compressor ever makes per-bucket artifacts
costly again.
No compression mechanism has been demonstrated. That is exactly the gap a LEARNED
compressor addresses (AutoCompressor, arXiv 2305.14788: 40 tokens per summary
vector, one forward pass per document), which also reports the same limitation we
found independently — their plain text retrieval beats their own summary vectors.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CM-B.0i's ceiling was 13/26 with the whole bucket in cache. Every compression number in the log is a fraction of that, so an artifactual ceiling would mean the log is measured against a bent ruler. This checks it against a live text prefill, which causal attention makes mathematically identical to a full-cache prefix. floor 0/26 text in prompt 14/26 live prefill, never serialised whole cache P=5100 13/26 bf16 -> CPU -> float32 -> index_select cartridge P=128 2/26 The +1 is decode noise, not a lossy cache path: the 5 disagreements are BIDIRECTIONAL (text wins y15/y17/y22, cache wins y18/y24). One-directional loss is what a broken round-trip looks like. So 1/26 at 35.9x is 8% of a real ceiling and the CM-B.0i negative stands. 0i also reproduces exactly (13/26, 2/26) on a different code path. Defects found building it: - generate_answer truncated silently at max_length=3500, truncation_side right. The control prompt is 4,626 tokens, so 24% would have been cut from the TAIL, where the stratified set puts a quarter of its answers. That fails toward a FALSE confirmation of the ceiling. Now raises. - The same cap is shared by both teacher paths, so every cartridge here was distilled from a teacher that saw 76% of its bucket. Not changed (would re-scope prior runs) but it bounds the 2/26 figure. - DynamicCache mutates in place, so a shared cache would append fixture N's answer to N+1's prefix. The API takes a factory, not a cache. - head/tail tokenisation split verified token-exact on all 26 rather than assumed; BPE is not split-invariant in general. MPS profile over 26 fixtures: cur pinned at 7,002 MB (range 0), driver 11,247-12,271 MB (no trend). No leak of either kind. The 4.5 GB gap is one 4,599-token prefill's transient workspace, cached and not returned by empty_cache() -- so the real requirement is 12.3 GB, not 7.0. Chunking long prefills should cut that to ~7.5 GB; extract_bucket_kv needs it too. 976 unit tests pass (+21). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
kv_attn is the mechanism this project never tested. Every selector in cm_kv_prune is query-agnostic -- kv_first/kv_last/kv_stride positional, kv_norm by ||K||, which its own comment calls "a cheap stand-in for attention importance". SnapKV/H2O/CompressKV keep what the ACTUAL QUERY attends to and report 97-99% of full-cache accuracy at 3-19% budget. CM-B.0i measured 8% of ceiling at 3% with a dumb selector and concluded no compression mechanism exists; that was never tested against an informed selector. select_indices_attn(...) global: sum attention over layers select_indices_attn(..., per_layer) SnapKV-style: each layer its own top-p cartridge_from_per_layer_selection rectangular builder for the above What it measures, stated so it is not oversold: scoring needs a forward over the FULL cache, so this arm saves nothing at measurement time. It answers "do P well-chosen positions suffice?" A NO would be a much stronger negative than "kv_first is bad", ruling out compression at that ratio by ANY selection method. Two real defects found while building it: - transformers 5.4.0 with SDPA returns attentions=() -- an EMPTY TUPLE, and it does NOT fall back to eager. Verified against gpt2. Unguarded, every score would be zero, topk would return 0..p-1, and kv_attn would silently BE kv_first while being reported as query-aware -- i.e. it would manufacture the null result it exists to test for. Now raises and names the fix (attn_implementation='eager'). - A 1-token continuation forward SIGBUSes GPT-2 on transformers 5.4.0. Reproducible in a minimal repro with no libucks code: cache=199 + piece=1 dies, while 198+2, 196+4 and 150+50 pass in separate processes, so it is shape-specific and not the machine's memory pressure. extract_bucket_kv folds a 1-token tail into the previous segment. chunked cache building (extract_bucket_kv chunk_tokens=, opt-in, default None so all ~10 callers are untouched) is the enabler: eager attention is required for scoring, and eager over 4,599 tokens materialises a (1,heads,4599,4599) matrix per layer. Also addresses the CM-B.0j finding that one long forward permanently caches 4.5 GB empty_cache() won't return. Equivalent under causal attention but NOT bitwise -- matmul shapes differ, so accumulation order does; tests assert ~1e-3, and a chunk size >= input length takes the single-forward path so it stays bit-identical. 1008 unit tests pass (+32). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
scripts/cm_kv_attn.py runs the query-aware arms against the dumb ones at
matched P, plus floor, the full-cache ceiling, and the trained cartridge,
all through one decode loop.
floor no prefix
full_cache the 13/26 ceiling CM-B.0j confirmed is real
cartridge 98 min of distillation at P=128
kv_first@P positional (what init_from_extracted_kv does)
kv_norm@P ||K|| magnitude -- the strongest QUERY-AGNOSTIC ranker, and
the fair comparison: it ranks too, it just cannot see the query
kv_attn@P global: attention summed over layers
kv_attn_L@P per-layer top-p, i.e. what SnapKV actually does
Loads with attn_implementation='eager' -- mandatory, since SDPA returns
attentions=() on transformers 5.4.0 and kv_attn would silently become
kv_first. Extraction is chunked (512) because eager over 4,599 tokens
materialises a (1,heads,4599,4599) matrix per layer, and because CM-B.0j
measured one long forward permanently caching 4.5 GB.
Refactor: attn_scores() + select_from_scores() split out of
select_indices_attn, so the expensive full-cache forward is paid ONCE per
fixture and reused across every budget and both selection modes, rather
than once per variant. A test asserts selection does not re-invoke the
model, and that the split helpers agree with the one-shot API.
1012 unit tests pass (+4).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
First positive mechanism in the cartridge track. At P=128 / 35.9x on the 26 position-stratified fixtures, ceiling 13/26: floor 0/26 0.0% of ceiling kv_norm@128 0/26 0.0% query-agnostic, magnitude kv_first@128 1/26 7.7% query-agnostic, positional cartridge 2/26 15.4% 98 minutes of training kv_attn@128 3/26 23.1% query-aware, global kv_attn_L@128 5/26 38.5% query-aware, per-layer full_cache 13/26 100.0% the ceiling 5x the positional selector and +3 over distillation, with zero training. CM-B.0i concluded no compression mechanism exists, but every selector it tested was query-agnostic -- so that negative was about the selector, not the content. Believed signal, not noise, for two reasons beyond the totals: the per-fixture wins are ASYMMETRIC 5-1 (y01/y04/y19/y20/y23 won, y06 lost), where CM-B.0j's 3-2 bidirectional split was correctly read as decode noise; and per-layer beats global 5 vs 3, independently corroborating SnapKV's reason for selecting per layer. Not overstated: - 38.5% of ceiling where the literature reports 97-99%. Two suspects in our implementation: SnapKV selects per HEAD and pools to keep contiguous clusters; attn_scores sums over heads and picks isolated positions. - Buys attention-COMPUTE compression, not STORAGE: scoring needs the full cache resident, so this is not a precomputable per-bucket cartridge. - One run at n=26 where decode noise is +/-2. +4 clears it, but treat as provisional until repeated. Ceiling 13/26 and cartridge 2/26 now reproduce exactly across CM-B.0i, 0j and 0k on three different code paths. Chunked extraction measured as predicted in 0j: driver 11,653 -> 9,471 MB, cur/driver gap 4,651 -> 2,683 MB, and that understates it since this run also switched to the more memory-hungry eager attention. Memory flat across all 26 fixtures, confirming 0j's no-leak finding independently. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repeat is bit-identical: 0/182 grounded verdicts and 0/182 generated answer texts differ, setup memory line matching byte-for-byte. So the pipeline is deterministic -- there is no seed to vary, and a re-run of a deterministic experiment contributes ZERO independent evidence about effect size. Repetition tests flakiness, not effect size. That forced the test I should have run before writing the first verdict. Exact two-sided McNemar on the discordant pairs: kv_attn_L vs kv_first 5-1 6 discordant p = 0.219 kv_attn vs kv_first 3-1 4 discordant p = 0.625 kv_attn_L vs cartridge 3-0 3 discordant p = 0.250 None significant. Six pairs splitting 5-1 is well inside fair-coin range, so "believed to be signal" was the wrong claim. The 5x ratio and +4 absolute are real; the significance is not there. Status downgraded from "FIRST POSITIVE MECHANISM" to "BEST LEAD, NOT ESTABLISHED". Clearing p<0.05 needs 6-0, 8-1 or 9-1 -- about three more net wins from 5-1, reachable only via more fixtures, more budgets, or a stronger selector. Next steps reordered accordingly: P sweep first (each budget is another paired comparison, and a monotone trend is much harder to get by chance than one point), then per-head selection + SnapKV pooling, then a second bucket. Also logged: this is the second time in one session that a pattern was called stronger than the data supported, both times favouring the hypothesis being hoped for -- earlier "11/11 agreement" finished 21/26. Compute the test before writing the verdict. Ceiling 13/26 and cartridge 2/26 still reproduce exactly, now four times. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
arms = 3 + len(selectors) * len(prefix_lens), and every arm is one generation per fixture. Four selectors over six budgets is 27 arms = 702 generations (~3h at observed rates), which is why the sweep narrows to the pair that actually generates evidence: kv_first@P vs kv_attn_L@P, the paired comparison McNemar runs on. 15 arms = 390 generations. Dropping the other two is justified by measurement, not convenience: kv_norm scored 0/26 at P=128 (weakest arm, below even positional) and global kv_attn lost to per-layer 3 vs 5. Both are characterised at P=128 and redundant for the power question. Default remains all four. Unknown selector names exit rather than KeyError mid-run, and the summary table builds its header and its aware-minus-dumb column from the selected set instead of assuming all four are present. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A 10-minute stall watchdog killed a healthy P sweep. With only a per-fixture log line, the first output appears after ALL 15 arms complete, so the gap between "390 generations" and "[ 1/26]" is one entire fixture -- and a 15-arm fixture whose P=512/1024 arms decode over caches 4-8x larger than P=128 can exceed 600s under swap. The threshold was set from the 7-arm run's 120s/fixture cadence without scaling for arm count. Raising the threshold would paper over it. The actual defect is that a healthy-but-slow run and a wedged one produced identical logs. Now each arm logs its grounded verdict and elapsed seconds, and each fixture logs its scoring-forward time plus memory, so the log grows steadily and any stall detector works correctly. Also yields per-arm timings, which is what the bad estimate was missing in the first place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three P-sweep attempts have now degraded or been killed under memory pressure, and results were written only at the end -- so a kill at fixture 20 discarded 20 fixtures of paired data, which is most of the statistical power the sweep exists to buy. _dump() is now called after every fixture, via temp-file-and-rename so being killed DURING a write cannot leave truncated JSON where a partial-but-valid file should be (that would turn "some data" into "no data", the exact failure this prevents). A `complete` flag tells a reader whether the totals are final or the per-question rows are a trustworthy snapshot with partial scores. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Six budgets against kv_first, n=26, ceiling 13/26:
P ratio kv_first kv_attn_L delta w/l McNemar p %ceiling
32 143.7x 0/26 1/26 +1 1/0 1.000 7.7%
64 71.9x 1/26 3/26 +2 3/1 0.625 23.1%
128 35.9x 1/26 5/26 +4 5/1 0.219 38.5%
256 18.0x 3/26 2/26 -1 1/2 1.000 15.4%
512 9.0x 2/26 5/26 +3 4/1 0.375 38.5%
1024 4.5x 5/26 4/26 -1 1/2 1.000 30.8%
Both arguments pre-registered in CM-B.0k fail. No budget is significant
(best p=0.219). Not monotone -- and that is evidence AGAINST, not merely
absent support: a real mechanism is smooth in P, so +4 at 36x flanked by
-1 at 18x is a random walk. CM-B.0k's headline is the maximum of six
comparisons.
No knee, which was the physical test: query-aware never exceeds 38.5% of
ceiling at ANY budget including 4.5x, and DECLINES at P=1024 where
positional overtakes it 5 vs 4 -- a contiguous 22% block beats 1024
scattered positions, which is the fragmentation cost SnapKV's pooling
addresses.
CM-B.0i's negative now survives its strongest test. The claim is no
longer "positional selection fails" but: cartridge distillation,
positional/magnitude selection, and query-aware per-layer selection (the
SnapKV family, reporting 97-99% of full-cache accuracy at 3-19% budget
elsewhere) ALL fail to compress this content at 3B. Four mechanisms, one
verified ceiling.
The reframe: compression was never the binding constraint. With the entire
bucket in cache and full attention the 3B answers 13/26 -- half. Selection
research optimises delivery of information the reader cannot exploit.
Incidental: on y20 a focused prefix BEAT full context (full_cache=0,
kv_attn_L@128=1), so attention dilution is real and 13/26 is an aggregate
reference, not a per-fixture upper bound.
0k downgraded to SUPERSEDED with its numbers kept and its reading marked
wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
CM-B.0l referenced this file for "stop treating the latent channel as the headline" and it still described Stages 1/2 as live work -- a fresh session would have landed on a stale plan. Adds a status banner at the top: four mechanisms failed (Phase 4-C cache aug, CM-A distillation, positional selection, query-aware selection), and the reframe that matters more -- with the ENTIRE bucket in cache the 3B answers 13/26, so every compression number is a fraction of a ceiling set by the reader, not the channel. Names the single gate that decides whether any of it resumes: does the ceiling move with scale? floor + full_cache at 0.5B/1.5B/3B, all locally cached, same family. Flat near 45-50% across 6x params means scale is not the lever and no further method work is justified. 7-8B is not runnable locally -- 15.2 GB of bf16 weights against 16 GB unified. Also records what is NOT closed and is where the value is: routing 1/15->14/15, chunk rerank 10->16/30, hybrid 19.5+/-1.7/30, raw KV prefill cache 13/26 vs 0/26 floor at one forward pass, the self-evolving substrate, and the measurement apparatus that caught five false positives. Stage 2 marked CLOSED explicitly rather than left contingent on an already-closed Stage 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.