feat(phase-54): Integration Truth — wire orchestrator and close silent no-ops

.planning/RETROSPECTIVE.md

@@ -2,6 +2,45 @@
 
 *A living document updated after each milestone. Lessons feed forward into future planning.*
 
+## Milestone: v3.0 — Competitive Parity & Benchmarks
+
+**Shipped:** 2026-05-14 (Phases 51, 51.5, 52, 53.5 merged; Phase 53 PR pending)
+**Phases:** 51-53 + 51.5 + 53.5
+**Honest grade:** the code is strong; the *claims* were not.
+
+### What Was Built
+- `memory-orchestrator` crate (expand → fan-out → rank fusion → rerank → context)
+- Simple `memory` CLI (search/context/recall/add/timeline/summary)
+- Benchmark harness + LOCOMO adapter skeleton
+- Cross-project federated query (`all_projects`)
+
+### What Failed (the reason v3.1 exists)
+Verification was **by existence, not execution**:
+- `memory-orchestrator` had zero dependents; CLI called `RouteQuery` directly
+- LLM reranker was a mock `.reverse()`; CrossEncoder warned and fell back
+- BM25 outbox `IndexEvent`/`UpdateToc` reported success while indexing nothing
+- `CrateLayer::Hybrid` delegated to BM25
+- Two divergent RRF implementations
+- LOCOMO adapter never ran (invented schema, 404 download URL, substring scoring)
+- `recall_at_5` identical to accuracy; `compression_ratio` summed file-path lengths
+- TOC navigation p50 ≈ 64.6s on a 240-event corpus, percentiles from 3 samples
+- Phase 53 "17/17 verified, no gaps" — every item was an existence check
+- No root README/LICENSE; Cargo repository URL pointed at the wrong org
+
+### Process Changes Bound to v3.1
+1. **Execution-evidence:** VERIFICATION.md for run-dependent requirements must cite a committed artifact from actually running them.
+2. **Reachability:** a new crate is not done until `cargo tree -i <crate>` shows a binary dependent, or it is declared dormant.
+3. **human_verification items are blockers** when they gate the milestone goal.
+4. This retrospective exists (it did not, as of v3.0 close).
+
+### Key Lessons
+1. "The struct exists and a unit test constructs it" is not a shipped feature.
+2. Silent `Ok(())` while skipping work will lie to every downstream metric.
+3. A flag that logs "not implemented" and continues is a lying flag.
+4. Benchmarks that cannot be re-run are marketing, not evidence.
+
+---
+
 ## Milestone: v2.5 — Semantic Dedup & Retrieval Quality
 
 **Shipped:** 2026-03-10

.planning/ROADMAP.md

@@ -12,6 +12,7 @@
 - ✅ **v2.6 Cognitive Retrieval** — Phases 39-44 (shipped 2026-03-16)
 - ✅ **v2.7 Multi-Runtime Portability** — Phases 45-50 (shipped 2026-03-22)
 - **v3.0 Competitive Parity & Benchmarks** — Phases 51-53 + Phase 51.5 (in progress; Phase 51.5 merged 2026-04-28)
+- **v3.1 Make It True** — Phases 54-58 (in progress; Phase 54 Integration Truth executing)
 
 ## Phases
 
@@ -251,4 +252,28 @@ Out-of-band insertion (mirrors Phase 51.5 pattern). Originally planned as Phase
 
 ---
 
-*Updated: 2026-05-14 — Phase 53 (Benchmark Suite) opening PR to close v3.0*
+## v3.1 Make It True (Phases 54-58)
+
+See: `docs/plans/v3.1-make-it-true-plan.md`
+
+Close the claim/reality gap, then open the shop window. No new capabilities.
+
+### Phase 54: Integration Truth (6/6 plans) — IN EXECUTION 2026-08-30
+
+- [ ] 54-01: Wire orchestrator into RouteQuery + real LLM reranker
+- [ ] 54-02: Fix BM25 outbox no-op (index events)
+- [ ] 54-03: Make Hybrid layer hybrid
+- [ ] 54-04: One rank-fusion implementation
+- [ ] 54-05: Honest daemon flags + attach indexes
+- [ ] 54-06: Lock-poisoning recover_lock policy
+
+### Phase 55: Performance Truth (0/2)
+
+### Phase 56: Honest Benchmarks (0/3)
+
+### Phase 57: Shop Window & Positioning (0/3)
+
+### Phase 58: Launch (side quest)
+
+*Updated: 2026-08-30 — Phase 54 Integration Truth in execution*
+

.planning/STATE.md

@@ -1,16 +1,16 @@
 ---
 gsd_state_version: 1.0
-milestone_name: Competitive Parity & Benchmarks
+milestone_name: Make It True
 status: in_progress
 stopped_at: null
-last_updated: "2026-05-14T00:00:00.000Z"
-last_activity: 2026-05-14 — Phase 53 (Benchmark Suite) rebased onto main; opening PR closes v3.0
+last_updated: "2026-08-30T08:00:00.000Z"
+last_activity: 2026-08-30 — Phase 54 Integration Truth implemented on feature/phase-54-integration-truth
 progress:
   total_phases: 5
-  completed_phases: 4
-  total_plans: 8
-  completed_plans: 8
-  percent: 100
+  completed_phases: 0
+  total_plans: 14
+  completed_plans: 6
+  percent: 43
 ---
 
 # Project State
@@ -20,22 +20,24 @@ progress:
 See: .planning/PROJECT.md (updated 2026-03-22)
 
 **Core value:** Agent can answer "what were we talking about last week?" without scanning everything
-**Current focus:** v3.0 Phase 53 — Benchmark Suite (PR review; closes v3.0)
+**Current focus:** v3.1 Phase 54 — Integration Truth (wire orchestrator, fix silent no-ops)
 
 ## Current Position
 
-Phase: 53 of 53 (Benchmark Suite) — opening PR
-Plan: 3 of 3 complete (53-01 foundation, 53-02 runner/scorer/CLI, 53-03 LOCOMO adapter)
-Status: Phase 51 + 51.5 + 52 + 53.5 merged; Phase 53 (Benchmark Suite) PR opens; v3.0 fully shipped on merge
-Last activity: 2026-05-14 — Rebased gsd/phase-53-benchmark-suite onto post-Phase-53.5 main; opening PR
+Phase: 54 of 58 (Integration Truth)
+Plan: 01-06 implemented on `feature/phase-54-integration-truth` (PR pending)
+Status: Phase 54 code complete; awaiting PR review
+Last activity: 2026-08-30 — RouteQuery spliced through MemoryOrchestrator; BM25 outbox indexes events; Hybrid fuses BM25+vector; recover_lock policy; honest `--background`
 
-Progress: [██████████] 100% (4 of 4 phases; Phase 53 PR pending)
+Progress: [████░░░░░░] ~43% (6/14 plans; Phase 54 of 54-58)
 
 ## Out-of-band Work
 
 ### Open PRs
 
-(none — Phase 53 PR opening shortly)
+| PR | What | Notes |
+|---|---|---|
+| #31 | v3.1 design spec (docs only) | Keep separate from this implementation PR |
 
 ### Recently Merged
 
@@ -46,79 +48,11 @@ Progress: [██████████] 100% (4 of 4 phases; Phase 53 PR pend
 | #28 | Phase 51: Retrieval Orchestrator | 2026-04-28 |
 | #27 | Phase 51.5: API summarizer wiring | 2026-04-27 |
 
-### Local-only Branches (still stacked)
-
-- `gsd/phase-{54..58}` — 5-phase stack of GSD work covering v3.1 (Phases 54-56: export/backup/import) and v3.2 (Phases 57-58: runtime registration). Each branch backed up to origin 2026-05-12 (no PRs). Pending strategic decision: per-milestone PRs vs. per-phase. **Note:** the planning files on these branches describe v3.0/v3.1 as "shipped" — that reflects local execution intent, not origin/main reality.
-
-## Performance Metrics
-
-**Velocity:**
-- Total plans completed: 146 (across 9 milestones)
-- Average duration: ~15 min
-- Total execution time: ~36 hours
-
-**Milestone History:**
-See .planning/MILESTONES.md
-
 ## Decisions
 
-- v3.0 scope: Retrieval orchestrator, simple CLI API, benchmark suite (3 phases) + Phase 51.5 (out-of-band summarizer wiring)
-- Orchestrator wraps existing RetrievalExecutor (no changes to memory-retrieval crate)
-- CLI uses new `memory` binary (memory-daemon and hook handlers unchanged)
-- LOCOMO dataset never committed (gitignored)
-- Existing implementation plans in docs/superpowers/plans/ will be converted to GSD plans
-- Phase 51.5 inserted as a decimal phase (out-of-band insertion pattern from `/gsd:insert-phase`) since the summarizer wiring shipped before Phase 51 itself
-- [Phase 51]: RerankMode defaults to Heuristic (no LLM cost by default)
-- [Phase 51]: RankedResult uses f64 for fusion precision, SearchResult uses f32
-- [Phase 51]: RRF deduplicates by doc_id, keeping first-seen SearchResult
-- [Phase 51]: HeuristicReranker trims to top 10 (MAX_RESULTS const)
-- [Phase 51]: Token estimation: chars * 0.75 + 50 overhead
-- [Phase 51]: MemoryOrchestrator accepts Box<dyn Reranker> via with_reranker() for test injection
-- [Phase 52]: All CLI commands route through gRPC (no direct RocksDB access) — daemon stays single source of truth
-- [Phase 52]: JsonEnvelope output pattern: ok/error/context_ok constructors, TTY detection via IsTerminal
-- [Phase 52]: New `memory-cli` crate (binary name: `memory`) added to workspace — separate from `memory-daemon`
-- Phase 53.5 inserted as a decimal phase (out-of-band, mirrors 51.5) for cross-project federation work originally authored against the Phase 51 slot — preserves v3.0 Competitive Parity scope (Phases 51-53)
-- [Phase 53.5]: TOC-based primary fallback in `federated_query` when BM25/vector indexes aren't built — ensures cross-project mode always works
-- [Phase 53.5]: Project attribution stored in `metadata["project"]` — same convention as `metadata["agent"]` from v2.1
-- [Phase 53.5]: `federated_query` is a pure function — matches existing `enrich_with_salience` pattern
-- [Phase 53.5]: `open_read_only` uses `DB::open_cf_for_read_only` from rocksdb 0.22 with `create_if_missing(false)`
-- [Phase 53]: New `memory-bench` crate with TOML fixture loader, runner/scorer/report/baseline modules, and LOCOMO adapter
-- [Phase 53]: Benchmark dataset (LOCOMO) gitignored — adapter loads from local path; never committed
-- [Phase 53]: CI benchmark smoke test added to verify the harness runs (not the full LOCOMO score)
-
-## Blockers
-
-- None
-
-## Accumulated Context
-
-- Spec reference: docs/superpowers/specs/2026-03-21-v3-competitive-parity-design.md
-- Phase A plan: docs/superpowers/plans/2026-03-21-v3-phase-a-retrieval-orchestrator.md
-- Phase B plan: docs/superpowers/plans/2026-03-21-v3-phase-b-simple-cli-api.md
-- Phase C plan: docs/superpowers/plans/2026-03-21-v3-phase-c-benchmark-suite.md
-
-## Milestone History
-
-See: .planning/MILESTONES.md for complete history
-
-- v1.0.0 MVP: Shipped 2026-01-30 (8 phases, 20 plans)
-- v2.0.0 Scheduler+Teleport: Shipped 2026-02-07 (9 phases, 42 plans)
-- v2.1 Multi-Agent Ecosystem: Shipped 2026-02-10 (6 phases, 22 plans)
-- v2.2 Production Hardening: Shipped 2026-02-11 (4 phases, 10 plans)
-- v2.3 Install & Setup Experience: Shipped 2026-02-12 (2 phases, 2 plans)
-- v2.4 Headless CLI Testing: Shipped 2026-03-05 (5 phases, 15 plans)
-- v2.5 Semantic Dedup & Retrieval Quality: Shipped 2026-03-10 (4 phases, 11 plans)
-- v2.6 Cognitive Retrieval: Shipped 2026-03-16 (6 phases, 13 plans)
-- v2.7 Multi-Runtime Portability: Shipped 2026-03-22 (6 phases, 11 plans)
-
-## Cumulative Stats
-
-- ~60,000 LOC Rust across 17 crates (memory-orchestrator, memory-cli, memory-bench all new in v3.0)
-- 53 phases (Phase 1-53 + 51.5 + 53.5), 157 plans across 9 milestones
-- 50+ E2E tests + 144 bats CLI tests + orchestrator + memory-cli + memory-bench tests + 9 federated unit tests + 4 cross-project e2e tests + CI benchmark smoke test
-
-## Session Continuity
-
-**Last Session:** 2026-05-12
-**Stopped At:** Phase 52 rebased onto main; opening PR
-**Resume File:** None
+- v3.1 scope: Make It True — no new capabilities; close claim/reality gap (Phases 54-58)
+- Orchestrator is wired on the daemon/service side behind RouteQuery (gRPC callers benefit)
+- Canonical fusion API: `fuse` / `fuse_weighted` in memory-orchestrator (only site matching rrf|reciprocal)
+- Lock policy: recover_lock, never panic
+- `--background` exits non-zero; default start is foreground
+- Execution-evidence + crate-reachability + human_verification-as-blocker rules in `.planning/config.json`

.planning/config.json

@@ -11,5 +11,11 @@
     "_auto_chain_active": false,
     "nyquist_validation": true
   },
-  "granularity": "fine"
-}
+  "granularity": "fine",
+  "verification": {
+    "execution_evidence": true,
+    "crate_reachability": true,
+    "human_verification_is_blocker": true,
+    "notes": "v3.1 process rules: VERIFICATION.md items for run-dependent requirements must cite a committed artifact from actually running them. A new crate is not done until cargo tree -i <crate> shows a binary dependent, or it is explicitly declared dormant. human_verification items that gate the milestone goal are blockers."
+  }
+}

.planning/phases/54-integration-truth/54-01-PLAN.md

@@ -0,0 +1,34 @@
+---
+phase: 54-integration-truth
+plan: 01
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - proto/memory.proto
+  - crates/memory-orchestrator/src/orchestrator.rs
+  - crates/memory-orchestrator/src/rerank.rs
+  - crates/memory-service/src/retrieval.rs
+  - crates/memory-service/Cargo.toml
+  - crates/memory-client/src/client.rs
+  - crates/memory-cli/src/commands/search.rs
+  - crates/memory-toc/src/summarizer/api.rs
+autonomous: true
+requirements: [INT-01, INT-02, INT-03]
+must_haves:
+  truths:
+    - "cargo tree -i memory-orchestrator lists a dependent binary crate"
+    - "RouteQuery explainability.fusion_stage is rank_fusion"
+    - "rerank_mode=llm with a mock completer reorders results vs heuristic"
+  artifacts:
+    - path: "crates/memory-service/src/retrieval.rs"
+      provides: "RouteQuery runs MemoryOrchestrator::query_ranked"
+      contains: "query_ranked"
+    - path: "proto/memory.proto"
+      provides: "rerank_mode and fusion_stage proto fields"
+      contains: "fusion_stage"
+---
+
+<objective>
+Wire memory-orchestrator behind RouteQuery. Real LLM reranker via Completer + ApiSummarizer. CrossEncoder hard-errors. CLI --rerank forwarded. memory recall = search --rerank=llm --top=10.
+</objective>

.planning/phases/54-integration-truth/54-02-PLAN.md

@@ -0,0 +1,23 @@
+---
+phase: 54-integration-truth
+plan: 02
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - crates/memory-indexing/src/bm25_updater.rs
+  - crates/memory-search/src/document.rs
+  - crates/memory-search/src/indexer.rs
+  - crates/memory-search/src/schema.rs
+  - docs/ARCHITECTURE.md
+autonomous: true
+requirements: [INT-04]
+must_haves:
+  truths:
+    - "IndexEvent/UpdateToc index the event into BM25 (DocType::Event)"
+    - "Missing events warn and increment BM25_SKIPPED_NOOP, never silent Ok-while-noop"
+---
+
+<objective>
+Fix the BM25 outbox no-op. Event bodies are indexed so ingest → drain → teleport is true.
+</objective>

.planning/phases/54-integration-truth/54-03-PLAN.md

@@ -0,0 +1,18 @@
+---
+phase: 54-integration-truth
+plan: 03
+type: execute
+wave: 1
+depends_on: [54-04]
+files_modified:
+  - crates/memory-service/src/retrieval.rs
+autonomous: true
+requirements: [INT-05]
+must_haves:
+  truths:
+    - "CrateLayer::Hybrid merges BM25 + vector via fuse_weighted; fused order differs from either input when ranks diverge"
+---
+
+<objective>
+Make the Hybrid layer hybrid: BM25 + vector fused with equal weights, retagged Hybrid.
+</objective>

.planning/phases/54-integration-truth/54-04-PLAN.md

@@ -0,0 +1,20 @@
+---
+phase: 54-integration-truth
+plan: 04
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - crates/memory-orchestrator/src/fusion.rs
+  - crates/memory-service/src/hybrid.rs
+autonomous: true
+requirements: [INT-06]
+must_haves:
+  truths:
+    - "rg -l reciprocal|rrf crates/ shows one implementation site (fusion.rs)"
+    - "Weighted fusion differs from either input when ranks diverge; empty input fail-open"
+---
+
+<objective>
+One rank-fusion implementation: weighted live form lives in memory-orchestrator::fusion. HybridSearch RPC calls fuse_weighted.
+</objective>

.planning/phases/54-integration-truth/54-05-PLAN.md

@@ -0,0 +1,23 @@
+---
+phase: 54-integration-truth
+plan: 05
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - crates/memory-daemon/src/cli.rs
+  - crates/memory-daemon/src/commands.rs
+  - crates/memory-daemon/src/main.rs
+  - crates/memory-service/src/server.rs
+  - crates/memory-service/src/ingest.rs
+autonomous: true
+requirements: [INT-07, INT-08]
+must_haves:
+  truths:
+    - "--background exits non-zero; default start is honest foreground"
+    - "run_server_with_scheduler attaches BM25/vector/topics via QueryIndexBundle"
+---
+
+<objective>
+Honest daemon flags. Attach live indexes so RouteQuery is not agentic-only.
+</objective>

.planning/phases/54-integration-truth/54-06-PLAN.md

@@ -0,0 +1,23 @@
+---
+phase: 54-integration-truth
+plan: 06
+type: execute
+wave: 1
+depends_on: []
+files_modified:
+  - crates/memory-types/src/sync.rs
+  - crates/memory-service/src/vector.rs
+  - crates/memory-vector/src/hnsw.rs
+  - crates/memory-scheduler/src/registry.rs
+  - crates/memory-storage/src/usage.rs
+autonomous: true
+requirements: [INT-09]
+must_haves:
+  truths:
+    - "Production std lock sites recover via recover_lock; LOCK_POISON_RECOVERIES counts events"
+    - "No production .lock()/.read()/.write().unwrap() in vector.rs, hnsw.rs, registry.rs, usage.rs"
+---
+
+<objective>
+Lock-poisoning policy: recover the guard and count, never panic the daemon.
+</objective>