V2

.gitignore

@@ -1,4 +1,4 @@
- .libucks/
+.libucks/
 __pycache__/
 *.py[cod]
 *.egg-info/
@@ -9,7 +9,22 @@ venv/
 .pytest_cache/
 .ruff_cache/
 *.so
-.venv/
-.libucks/
-__pycache__/
-*.egg-info/
+.DS_Store
+
+# Local secrets — never commit
+.env
+.env.local
+
+# Local training/eval run logs (transient; results snapshot to tests/eval/results/)
+cm_*.log
+.claude/scheduled_tasks.lock
+
+# Vendored dep removed in the CM-B cleanup (installed from PyPI or local wheel instead)
+mps_bitsandbytes-0.7.0/
+mps_bitsandbytes-0.7.0.tar.gz
+
+# Stale Claude worktree checkouts (were committed as orphaned gitlinks; safe to rm -rf)
+.claude/worktrees/
+
+# Local run logs (kept out of git; cm_*.log already covered above)
+logs/

.mcp.json

@@ -0,0 +1,11 @@
+{
+  "mcpServers": {
+    "libucks": {
+      "command": "/Users/ecaterina/Developer/libucks/.venv/bin/libucks",
+      "args": ["serve"],
+      "env": {
+        "HF_HUB_OFFLINE": "1"
+      }
+    }
+  }
+}

ARCHITECTURE.md

@@ -112,6 +112,8 @@ signed with RS256. Expiry is enforced at 15 minutes for access tokens and
 
 **Design constraint:** The Watchdog performs zero AI inference. It is a pure data-extraction process. This keeps it fast, always-on, and independently restartable.
 
+**Deployment note:** `libucks serve` does NOT start WatchdogService by default. The primary update path is git-hook–driven: `git post-commit` → Unix socket → `StartupRecovery.run()`. Use `libucks install-hooks` to wire a target repo. WatchdogService is available as an opt-in for repos that do not use git hooks (e.g. non-git directories), but must be started explicitly.
+
 ---
 
 ### 3.3 Central Agent (Router / Dispatcher)
@@ -485,11 +487,11 @@ repo_cache         = "~/.libucks/repos"
 
 ```
 libucks/                         ← this repository (the libucks tool itself)
-├── main.py                      ← CLI entry point (click group)
+├── libucks/_cli.py              ← CLI entry point (click group, via [project.scripts])
 ├── pyproject.toml
 ├── tools_v1.json                ← versioned MCP tool schema manifest
 ├── ARCHITECTURE.md
-├── IMPLEMENTATION_PLAN.md
+├── docs/cartridges-plan.md      ← active roadmap (superseded plans in docs/archive/)
 │
 ├── libucks/
 │   ├── config.py
@@ -648,13 +650,60 @@ divides the loss by 8 before each `backward()` call and only advances the
 optimizer after every 8th bucket (or at end of epoch).  Effective batch size = 8
 without requiring padded batching, keeping peak MPS memory bounded.
 
+#### Query dropout (Phase 12.8 fix)
+
+**Problem discovered:** With tight teacher-generated Q&A pairs, L_sep collapsed to 0.0000
+throughout all training epochs. The model learned to reconstruct answers directly from
+query tokens — the latent prefix carried zero additional information and gradient never
+flowed through it.
+
+**Root cause:** `L_task` can be minimised without ever attending to the latent when
+`[query_prefix → target_text]` is a memorisable mapping. The model correctly ignores
+the latent because doing so is sufficient.
+
+**Fix:** Query dropout (`query_dropout_rate = 0.5`). Each training step independently
+samples whether to include the query prefix:
+
+```python
+use_query = (random.random() >= 0.5)
+# prefix = [bop, latent_K, eop, query (or empty), target]
+```
+
+On dropout steps the model sees only `[bop, latent_K, eop, "Answer:\n", target]`.
+Without query tokens, the only source of signal for reconstructing `target_text` is
+the latent. This forces L_sep > 0 and gradient to flow through the latent channel.
+Non-dropout steps retain query conditioning so the model also learns the combined path
+used at inference time.
+
+**Hard-negative selection:** The wrong-path bucket is now the centroid-nearest neighbour
+(highest cosine similarity) rather than the first non-matching bucket. A hard negative
+with semantically close content makes L_sep harder to satisfy, producing a stronger
+discrimination gradient.
+
+**Updated λ_sep:** Raised from 0.1 → 0.3 so the separation term can compete with a
+near-zero task loss once L_task converges.
+
+**L_sep computed at position [:1] only (Phase 12.9):** Applying `separation_loss` over
+all T=64 target positions dilutes the genuine signal by 64×. Positions 1..T-1 attend to
+teacher-forced target tokens that are *identical* in both correct and wrong paths, making
+their logit distributions nearly identical. Only position 0 (predicting the first target
+token from the prefix alone) is genuinely latent-conditioned. Fix: `logits_tgt[:1]` in
+`LoRAReceiverTrainer._forward_and_losses`.
+
+**Wrong-path always r=0 (Phase 12.9):** The wrong soft-prompt uses `CurriculumMixer.mix`
+with `r=0.0` (pure latents), while the correct path uses the sampled `r ∈ [0,1]`. This
+ensures all K latent positions differ between correct and wrong paths, maximising the
+discriminative signal at position 0 regardless of the correct path's mixing rate.
+
 #### Recommended hyperparameters
 
-| Param | Phase 12.6 (broken) | Phase 12.7 (fixed) | Rationale |
-|---|---|---|---|
-| `lora_r` | 32 | 4 | r=32 overfits 47 samples; r=4 sufficient (Hu et al. 2022 Table 2) |
-| `lora_alpha` | 64 | 8 | Preserves scaling ratio alpha/r = 2 |
-| `lr` | 1e-3 | 2e-4 | Standard LoRA fine-tuning range (1e-4 – 3e-4) |
+| Param | Phase 12.6 (broken) | Phase 12.7 (fixed) | Phase 12.8 (fixed) | Rationale |
+|---|---|---|---|---|
+| `lora_r` | 32 | 4 | 4 | r=32 overfits 47 samples |
+| `lora_alpha` | 64 | 8 | 4.0 | alpha/r = 1 (conservative) |
+| `lr` | 1e-3 | 2e-4 | 2e-4 | Standard LoRA range |
+| `query_dropout` | — | — | 0.5 | Forces latent conditioning |
+| `λ_sep` | 0.1 | 0.1 | 0.3 | Competes with low L_task |
 
 ### Implementation Files
 

CLAUDE.md

@@ -5,13 +5,21 @@ You are the Principal Systems Architect and Lead Python Engineer building the `l
 ## 1. Project Blueprints (MANDATORY ROUTING)
 Before executing complex code changes, you MUST consult the blueprints:
 - `ARCHITECTURE.md`: Contains the system design, data flows, and the critical V2 Latent Space constraints.
-- `IMPLEMENTATION_PLAN.md`: Contains our strict, phase-gated roadmap.
+- `docs/cartridges-plan.md`: The current active roadmap (Cartridge Memory track). Superseded V1/V2/Phase-4 plans are archived under `docs/archive/` (including the closed Phase 4-C negative-result log at `docs/archive/phase-4c/`).
 
 ## 2. The Golden Rules
 - **Strict TDD:** You are never allowed to move to Phase N+1 until the Testing Gate for Phase N is 100% green. Always write the `test_*.py` file first, run it to watch it fail, then write the implementation to make it pass.
 - **Latent Space Constraint:** Librarians only produce `Representation` objects. ONLY the `Translator` is allowed to call `decode()` and output natural language.
-- **No Automatic Mitosis:** V1 uses manual mitosis only. Do not build k-means clustering.
+- **Auto-mitosis and auto-merge are on.** `HealthMonitor` runs every 5 min (started by `libucks serve` at `mcp_bridge.py:257`). It calls `MitosisService.split()` (k-means k=2 in `mitosis.py`) on buckets that exceed `mitosis_threshold` tokens or fall below the coherence threshold, and `MergingService.run_merge_pass()` for cosine-similar pairs. This was originally scoped as "manual only" in V1; that constraint was relaxed when HealthMonitor landed (Phase 6-E/F).
+- **Auto-bucket-creation is on.** `NovelBucketService` (started by `libucks serve` alongside `HealthMonitor`) drains `CentralAgent.create_bucket_queue` to spawn new buckets when commits land substantial + cosine-novel content. Gates: `min_bucket_seed_tokens` (default 1500) AND `is_novel(embedding)`. Small or similar diffs are routed into the nearest existing bucket — coherence-driven mitosis splits later if pressure builds. Buckets are subject-specific knowledge units; single-file/single-chunk spawning is forbidden.
 - **API First:** V1 uses standard API calls (OpenAI/Anthropic), not a local Ollama daemon.
+- **LoRA training MUST use query dropout:** `query_dropout_rate=0.5` in `_train_lora_receiver`. Without it, L_sep collapses to 0.0000 and the model ignores the latent entirely. See ARCHITECTURE.md §10 Phase 12.8.
+- **Update pipeline is git-hook driven:** `libucks serve` does NOT start WatchdogService. Updates arrive via git post-commit hooks → Unix socket → `StartupRecovery`. For each changed file: if an existing bucket owns it → `UpdateEvent` directly. If no bucket owns it → `StartupRecovery._handle_unmatched_file` does the size+novelty check above and either enqueues a `CreateBucketEvent` or routes to the nearest existing bucket. Use `libucks install-hooks` to wire a target repo.
+- **Check sep in training logs:** If `sep=0.0000` persists past epoch 3 in any training run, STOP. The latent is being ignored. Do not continue training — investigate the query dropout code path first.
 
 ## 3. Session Start Protocol
-When a new session begins, scan the current files in `tests/unit/` to determine exactly which Phase and Step we are currently on in the `IMPLEMENTATION_PLAN.md` before taking action.
+When a new session begins:
+1. Read `docs/cartridges-log.md` (newest entry) to find the current stage/gate status, and `docs/cartridges-plan.md` for the active roadmap.
+2. Scan `tests/unit/` to confirm which tests exist and which are green.
+3. Check if `lora_receiver.pt` exists in the target repo's `.libucks/` before assuming LoRA is trained.
+4. Never assume training was successful — always verify `sep > 0.0000` in logs.