diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index f14356a..e3e31c1 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -124,17 +124,25 @@ jobs: shell: bash run: | VERSION=${{ steps.version.outputs.version }} - ARCHIVE_DIR="memory-daemon-${VERSION}-${{ matrix.name }}" + ARCHIVE_DIR="agent-memory-${VERSION}-${{ matrix.name }}" mkdir -p "dist/${ARCHIVE_DIR}" - # Copy binaries + # Copy binaries. All four are required: the README quickstart uses + # `memory` for search and `memory-installer` for hook setup, so an + # archive missing either is a broken release, not a partial one. if [[ "${{ runner.os }}" == "Windows" ]]; then - cp target/${{ matrix.target }}/release/memory-daemon.exe "dist/${ARCHIVE_DIR}/" - cp target/${{ matrix.target }}/release/memory-ingest.exe "dist/${ARCHIVE_DIR}/" || true + EXT=".exe" else - cp target/${{ matrix.target }}/release/memory-daemon "dist/${ARCHIVE_DIR}/" - cp target/${{ matrix.target }}/release/memory-ingest "dist/${ARCHIVE_DIR}/" || true + EXT="" fi + for bin in memory-daemon memory-ingest memory memory-installer; do + src="target/${{ matrix.target }}/release/${bin}${EXT}" + if [[ ! -f "$src" ]]; then + echo "::error::release archive would be missing ${bin}${EXT} for ${{ matrix.name }}" + exit 1 + fi + cp "$src" "dist/${ARCHIVE_DIR}/" + done # Copy documentation cp LICENSE "dist/${ARCHIVE_DIR}/" || true @@ -145,7 +153,7 @@ jobs: shell: bash run: | VERSION=${{ steps.version.outputs.version }} - ARCHIVE_DIR="memory-daemon-${VERSION}-${{ matrix.name }}" + ARCHIVE_DIR="agent-memory-${VERSION}-${{ matrix.name }}" cd dist tar -czvf "${ARCHIVE_DIR}.tar.gz" "${ARCHIVE_DIR}" rm -rf "${ARCHIVE_DIR}" @@ -155,7 +163,7 @@ jobs: shell: pwsh run: | $VERSION = "${{ steps.version.outputs.version }}" - $ARCHIVE_DIR = "memory-daemon-${VERSION}-${{ matrix.name }}" + $ARCHIVE_DIR = "agent-memory-${VERSION}-${{ matrix.name }}" cd dist Compress-Archive -Path $ARCHIVE_DIR -DestinationPath "${ARCHIVE_DIR}.zip" Remove-Item -Recurse -Force $ARCHIVE_DIR diff --git a/.planning/MILESTONES.md b/.planning/MILESTONES.md index 25a5edb..5b845b1 100644 --- a/.planning/MILESTONES.md +++ b/.planning/MILESTONES.md @@ -1,5 +1,67 @@ # Project Milestones: Agent Memory +## v3.1 Make It True (Shipped: 2026-08-31) + +**Delivered:** no new capabilities. Four phases closing the gap between what the +project claimed and what it did, after a v3.0 verification document self-graded +17/17 on a benchmark that had never been run and a crate no shipped binary could +reach. + +**Phases completed:** 54, 54.5, 55, 56, 57 (5 phases, 14 plans + a cleanup phase) + +**Key accomplishments:** + +- `memory-orchestrator` made reachable: `RouteQuery` calls it, `memory search` + is a client of that RPC (`cargo tree -i` now shows a shipped dependent) +- Hybrid retrieval actually fuses BM25 and vector; BM25 outbox drain no longer a + no-op; event documents store text so previews and LLM reranking work +- Explainability reports what actually ran — a failed LLM rerank says + `rerank=heuristic`, `layers_attempted` lists only invoked layers, client stop + conditions reach the orchestrator instead of being echoed and ignored +- Performance measurement split setup from query: the retired "64.6s TOC + navigation" figure was ingest-time rollup; warm query p50 is 0.13 ms. + Percentiles withheld below 10 (p90) / 30 (p99) samples +- Honest benchmark harness: the substring metric is named `context_hit_rate`, + not "LOCOMO"; committed results are labelled mock-backend / mock-judge, and no + comparative accuracy claim ships anywhere +- Root README, LICENSE, positioning writeup, CHANGELOG — the repo had none +- Supported-surface tiering: Tier 1 (Claude Code, Codex) gates PRs, Tier 2 + (Gemini, Copilot) runs weekly. OpenCode stub deleted rather than shipped +- Release archives now contain all four binaries; previously the CLI the + quickstart depends on was not shipped at all + +**Process changes (bound to this milestone):** + +- Execution-evidence rule: run-dependent requirements must cite a committed + artifact produced by running them +- Reachability rule: a new crate is not complete until `cargo tree -i` shows a + dependent binary, or it is explicitly declared dormant +- `human_verification` items gating the milestone goal are blockers by default +- The v3.0 retrospective was written: *the code is strong; the claims were not* + +**Defects found by executing documentation rather than reading it:** + +- First-run daemon created no index directories, so it accepted events and + answered every query with an empty result set — successfully +- `admin rebuild-toc` printed a TODO and exited 0; `--dry-run` advertised a + rebuild that did nothing +- `admin rebuild-bm25` is a prune, not a rebuild — relabelled rather than renamed + +**Known Gaps:** + +- No backfill for events indexed before v3.1 (`text_preview` stays empty) +- No real-backend / real-judge LOCOMO run, so no comparative claim +- Vector retrieval quality is not benchmarked +- GitHub repo description, topics, Discussions, and a recorded demo are + maintainer actions, not code + +**Stats:** + +- 64,626 LOC Rust across 20 crates +- Timeline: 2026-08-30 → 2026-08-31 + +--- + ## v2.7 Multi-Runtime Portability (Shipped: 2026-03-22) **Delivered:** Rust-based multi-runtime installer that converts canonical Claude plugin source into runtime-specific installations for 6 targets, replacing 5 manually-maintained adapter directories with a single conversion pipeline. diff --git a/.planning/ROADMAP.md b/.planning/ROADMAP.md index 76d0824..074c3a6 100644 --- a/.planning/ROADMAP.md +++ b/.planning/ROADMAP.md @@ -12,7 +12,7 @@ - ✅ **v2.6 Cognitive Retrieval** — Phases 39-44 (shipped 2026-03-16) - ✅ **v2.7 Multi-Runtime Portability** — Phases 45-50 (shipped 2026-03-22) - **v3.0 Competitive Parity & Benchmarks** — Phases 51-53 + Phase 51.5 (in progress; Phase 51.5 merged 2026-04-28) -- **v3.1 Make It True** — Phases 54-58 (in progress; Phases 54, 54.5, 55, 56 merged 2026-08-30, Phase 57 executing) +- ✅ **v3.1 Make It True** — Phases 54-58 (shipped 2026-08-31) ## Phases @@ -285,13 +285,22 @@ Close the claim/reality gap, then open the shop window. No new capabilities. - [x] 56-02: LOCOMO adapter v2 (real schema, mock vs llm-judge) - [x] 56-03: Smoke artifacts + HOLD comparison marketing -### Phase 57: Shop Window & Positioning (3/3 plans) — IN EXECUTION 2026-08-30 +### Phase 57: Shop Window & Positioning (3/3 plans) — COMPLETE 2026-08-31 (PR #36) - [x] 57-01: Repo hygiene (root README, LICENSE, repository URL) - [x] 57-02: Positioning writeup vs Mem0 / Zep / MemMachine / Letta - [x] 57-03: Scope trim — Tier 1/Tier 2 surface; OpenCode stub deleted -### Phase 58: Launch (side quest) +### Phase 58: Launch (side quest) — IN EXECUTION 2026-08-31 -*Updated: 2026-08-30 — Phase 56 merged (#34); Phase 57 Shop Window in execution* +- [x] Version bumped to 3.1.0 (was still 2.7.0 across v3.0 and v3.1) +- [x] CHANGELOG.md and v3.1 upgrade notes +- [x] Release archives ship all four binaries; assets renamed `agent-memory-*` +- [x] Blog post and Show HN / reddit copy drafted in `docs/launch/` +- [ ] Tag `v3.1.0` (maintainer — publishes public binaries) +- [ ] Repo description, topics, Discussions (maintainer — repo settings) +- [ ] Recorded demo (maintainer) +- [ ] Post the blog and the launch threads (maintainer) + +*Updated: 2026-08-31 — Phase 57 merged (#36); v3.1 shipped; Phase 58 launch prep* diff --git a/.planning/STATE.md b/.planning/STATE.md index d9248a9..138da78 100644 --- a/.planning/STATE.md +++ b/.planning/STATE.md @@ -1,16 +1,16 @@ --- gsd_state_version: 1.0 milestone_name: Make It True -status: in_progress +status: shipping stopped_at: null -last_updated: "2026-08-30T22:30:00.000Z" -last_activity: 2026-08-30 — Phase 56 merged (#34); Phase 57 Shop Window & Positioning in execution +last_updated: "2026-08-31T01:30:00.000Z" +last_activity: 2026-08-31 — Phase 57 merged (#36); v3.1 shipped; Phase 58 launch prep (version 3.1.0, CHANGELOG, launch drafts) progress: total_phases: 6 - completed_phases: 4 + completed_phases: 5 total_plans: 14 - completed_plans: 11 - percent: 79 + completed_plans: 14 + percent: 100 --- # Project State @@ -20,17 +20,19 @@ progress: See: .planning/PROJECT.md (updated 2026-03-22) **Core value:** Agent can answer "what were we talking about last week?" without scanning everything -**Current focus:** v3.1 Phase 57 — Shop Window & Positioning (root README, LICENSE, positioning writeup, supported-surface tiering) +**Current focus:** v3.1 Phase 58 — Launch (side quest): version 3.1.0, CHANGELOG, release archive fixes, launch drafts. The tag and the public posts are maintainer actions. ## Current Position -Phase: 57 of 58 (Shop Window & Positioning) -Plan: 01-03 implemented on `claude/phase-54-toolchain-drift-3k4fer` -Status: Phases 54, 54.5, 55, 56 merged; Phase 57 in review -Last activity: 2026-08-30 — #34 merged; Phase 57 README/LICENSE/positioning/scope-trim +Phase: 58 of 58 (Launch — side quest, not a GSD phase) +Status: all v3.1 GSD phases merged (54, 54.5, 55, 56, 57). Launch prep in review. +Last activity: 2026-08-31 — #36 merged; version bumped 2.7.0 → 3.1.0 -Progress: [████████░░] 11/14 plans merged; Phase 57's 3 plans are implemented and in review -(Phase 58 is a side quest, not a GSD phase) +Progress: [██████████] 14/14 plans merged. v3.1 GSD work complete. + +Remaining launch steps are maintainer actions: tag `v3.1.0` (publishes public +binaries), set the repo description/topics/Discussions, record the demo, and +post the blog and launch threads. ## Out-of-band Work @@ -38,12 +40,13 @@ Progress: [████████░░] 11/14 plans merged; Phase 57's 3 plan | PR | What | Status | |---|---|---| -| _(none open)_ | | | +| _(Phase 58 launch prep)_ | version 3.1.0, CHANGELOG, release fix, launch drafts | Open | ### Recently Merged | PR | What | Merged | |---|---|---| +| #36 | Phase 57 Shop Window & Positioning | 2026-08-31 | | #34 | Phase 56 Honest Benchmarks | 2026-08-30 | | #35 | Phase 54.5 truth leaks + rustc 1.97 pin | 2026-08-30 | | #33 | Phase 55 Performance Truth | 2026-08-30 | @@ -64,3 +67,6 @@ Progress: [████████░░] 11/14 plans merged; Phase 57's 3 plan - Phase 57 tiering: Tier 1 = Claude Code + Codex CLI (PR gate); Tier 2 = Gemini + Copilot (weekly schedule) - Phase 57: OpenCode removed rather than archived — a converter whose methods return empty is a false success, not a gap - Phase 57: no comparative benchmark claim ships while the only committed results are mock-backend / mock-judge +- Phase 58: version is 3.1.0 — it had been stuck at 2.7.0 through the whole v3.0 and v3.1 line, and there are no tags in the repo +- Phase 58: release archives are `agent-memory--` and carry all four binaries; the CLI the quickstart needs was previously not shipped +- Phase 58: `admin rebuild-bm25` is a prune, not a rebuild — relabelled rather than renamed, and there is no event backfill path diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..be2dcdc --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,100 @@ +# Changelog + +Notable changes per release. Dates are the release date, not the merge date of +the last commit. + +The guiding rule for this file, after the v3.0 retrospective: **a change is +listed only if it is true of the shipped code.** Claims that turned out to be +aspirational are recorded as retractions, not quietly dropped. + +## v3.1.0 — Make It True (2026-08-31) + +No new capabilities. This release closes the gap between what the project +claimed and what it did. + +### Added + +- Root `README.md` and `LICENSE` (MIT) — the repository had neither +- `docs/positioning/agent-memory-vs-competition.md` — head-to-head against + Mem0, Zep, MemMachine and Letta, with a claims ledger and the platform-risk + argument stated up front +- `docs/verification/57-quickstart-transcript.md` — the quickstart executed on + a clean machine, including the three defects the first run exposed +- `docs/benchmarks.md` — what the performance harness measures and what it does + not +- Supported-surface tiering: Tier 1 (Claude Code, Codex CLI) gates every PR; + Tier 2 (Gemini, Copilot) runs on a weekly schedule +- Release archives now ship all four binaries (`memory-daemon`, + `memory-ingest`, `memory`, `memory-installer`); previously the CLI the + quickstart depends on was not in the release at all + +### Fixed + +- **First-run daemon indexed nothing.** A fresh store had no `db/search` or + `db/vector`, so the outbox indexing job never registered and every query + returned an empty result set with no error. `start_daemon` now creates them +- **`memory-orchestrator` was unreachable** from any shipped binary. `RouteQuery` + now calls it, and `memory search` is a client of that RPC +- **Hybrid search was not hybrid** — it now fetches BM25 and vector results and + fuses them +- **BM25 outbox drain was a no-op**; events are indexed, and misses warn and + increment a counter instead of passing silently +- **BM25 stored no text for events**, so previews were empty and the LLM + reranker was judging blank bodies. New event documents are `TEXT | STORED` +- **A successful LLM rerank was undone** by a salience re-sort afterwards +- **Explainability misreported what ran**: a failed LLM rerank said + `rerank=llm`, `layers_attempted` listed layers that returned nothing, and + client `stop_conditions` / `mode_override` were echoed back while being + ignored. All now report and behave truthfully +- Per-event grip full-scan on the BM25 drain (an O(n²) drain) removed +- Retrieval fan-out is concurrent unless explicitly sequential +- Query, prune and dedup share one HNSW handle; embedding dimension comes from + the embedder rather than a hardcoded 384 +- Lock poisoning is recovered and counted rather than panicking +- CI pinned to Rust 1.97 via `rust-toolchain.toml`, after a floating-stable + Clippy lint reddened `main` + +### Changed — now fails loudly instead of silently + +- `memory-daemon start --background` exits non-zero with guidance; background + daemonization is not implemented +- `memory-daemon admin rebuild-toc` exits non-zero; offline TOC rebuild is not + implemented and no longer prints a TODO and exits 0 +- `CrossEncoderReranker` returns an explicit `NotImplemented` error rather than + degrading quietly +- An unknown `--rerank` value is rejected rather than falling back to heuristic +- `admin rebuild-bm25` is relabelled: it prunes documents below `--min-level` + and re-indexes nothing, which is what it always did + +### Removed (breaking) + +- **OpenCode is no longer a supported runtime.** Every method of its converter + returned empty, so `memory-installer --agent opencode` exited 0 and wrote no + files. The converter, the `Runtime::OpenCode` variant, its tool mappings, its + bats suite and the archived plugin directory are gone. `--agent opencode` now + exits 2. The runtime-agnostic `memory-ingest --agent opencode` path is + unaffected. See [UPGRADING](docs/UPGRADING.md) + +### Benchmarks + +- The "64.6 second TOC navigation" figure is **retracted**. It timed + ingest-time summarization rollup and labelled it navigation. Warm query + `single.toc` p50 is 0.13 ms over 30 samples +- Percentiles are withheld below 10 samples (p90) and 30 samples (p99) instead + of being interpolated from 3 +- Committed results in `benchmarks/results/` are **mock-backend and mock-judge** + and are labelled as such. No comparative accuracy claim ships anywhere in this + repository until a real-backend, real-judge run is committed beside it + +### Known gaps + +- No backfill for events indexed before v3.1 — their `text_preview` stays empty +- Vector search requires the embedding model download on first daemon start; + with no network the daemon warns and runs BM25-only +- Ingest to searchable is a ~1 minute scheduled outbox drain, not synchronous + +## Earlier releases + +Milestones v1.0 through v3.0 predate this file. Their scope is recorded in +`.planning/MILESTONES.md` and `.planning/ROADMAP.md`, and upgrade notes for +v2.0.0 through v2.2.0 are in [docs/UPGRADING.md](docs/UPGRADING.md). diff --git a/Cargo.toml b/Cargo.toml index ae21fa1..1d97571 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -24,7 +24,7 @@ members = [ ] [workspace.package] -version = "2.7.0" +version = "3.1.0" edition = "2021" license = "MIT" repository = "https://github.com/SpillwaveSolutions/agent-memory" diff --git a/README.md b/README.md index b90efdd..9581775 100644 --- a/README.md +++ b/README.md @@ -86,6 +86,10 @@ cargo build --release -p memory-daemon -p memory-ingest -p memory-cli export PATH="$PWD/target/release:$PATH" ``` +Prefer not to compile? Each release ships one archive per platform containing +all four binaries — see +[docs/setup/quickstart.md](docs/setup/quickstart.md#option-b-prebuilt-binaries). + ### 2. Start the daemon The daemon runs in the foreground. There is no built-in background mode — use @@ -159,7 +163,7 @@ is experimental. | Passive hook capture → `memory-ingest` | **Solid** | Covered by the bats CLI suites on Linux + macOS | | TOC build and drill-down navigation | **Solid** | Year → Month → Week → Day → Segment → Grip | | Grips / provenance | **Solid** | Excerpts link back to the events they came from | -| BM25 keyword search (Tantivy) | **Solid** | Exact tokens, no stemming (`jwt` does not match `JWTs`). Indexes built before v3.1 do not store text — rebuild to get event previews | +| BM25 keyword search (Tantivy) | **Solid** | Exact tokens, no stemming (`jwt` does not match `JWTs`). Events indexed before v3.1 have empty `text_preview` and there is no backfill command — see [UPGRADING](docs/UPGRADING.md) | | Vector search (HNSW + Candle) | **Solid** | First daemon start downloads the embedding model; with no network the daemon warns and runs BM25-only | | Topic graph | **Works** | Clustering quality is not benchmarked | | Hybrid fusion + `RouteQuery` orchestration | **Works** | Wired end-to-end in Phase 54; explainability reports what actually ran | @@ -215,6 +219,7 @@ unaffected by tiering. | [docs/verification/57-quickstart-transcript.md](docs/verification/57-quickstart-transcript.md) | The transcript of this quickstart being run on a clean machine, defects and all | | [docs/positioning/agent-memory-vs-competition.md](docs/positioning/agent-memory-vs-competition.md) | Head-to-head vs Mem0 / Zep / MemMachine / Letta | | [docs/UPGRADING.md](docs/UPGRADING.md) | Version-to-version migration notes | +| [CHANGELOG.md](CHANGELOG.md) | What changed per release, including retractions | ## Contributing diff --git a/crates/memory-adapters/Cargo.toml b/crates/memory-adapters/Cargo.toml index 1ea4c5f..7fac15f 100644 --- a/crates/memory-adapters/Cargo.toml +++ b/crates/memory-adapters/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "memory-adapters" -version = "2.7.0" +version.workspace = true edition = "2021" description = "Agent adapter SDK for multi-agent memory integration" license = "MIT" diff --git a/crates/memory-daemon/src/cli.rs b/crates/memory-daemon/src/cli.rs index 6d6e33b..41ce1c7 100644 --- a/crates/memory-daemon/src/cli.rs +++ b/crates/memory-daemon/src/cli.rs @@ -306,7 +306,8 @@ pub enum AdminCommands { dry_run: bool, }, - /// Rebuild BM25 index with level filtering + /// Prune BM25 documents below a TOC level. Despite the name this does not + /// re-index anything -- it only removes documents below `--min-level`. RebuildBm25 { /// Minimum TOC level to keep: segment, day, week, month, year #[arg(long, default_value = "day")] diff --git a/crates/memory-daemon/src/commands.rs b/crates/memory-daemon/src/commands.rs index 6c2fb0f..d111b90 100644 --- a/crates/memory-daemon/src/commands.rs +++ b/crates/memory-daemon/src/commands.rs @@ -1532,7 +1532,8 @@ fn handle_prune_vectors( /// Handle the rebuild-bm25 command. /// -/// Rebuilds the BM25 index keeping only documents at or above the specified level. +/// Despite the name, this prunes rather than rebuilds: it removes documents +/// below `min_level` and re-indexes nothing. fn handle_rebuild_bm25(db_path: &str, min_level: &str, search_path: Option) -> Result<()> { use memory_search::{SearchIndex, SearchIndexConfig, SearchIndexer}; @@ -1554,8 +1555,9 @@ fn handle_rebuild_bm25(db_path: &str, min_level: &str, search_path: Option-.tar.gz` and contain all four binaries + (`memory-daemon`, `memory-ingest`, `memory`, `memory-installer`). Previously + the CLI the quickstart depends on was not shipped at all. +- **Runtimes are tiered.** Claude Code and Codex CLI (Tier 1) gate every PR; + Gemini and Copilot (Tier 2) moved to a weekly scheduled CI run. Tiering is + not removal — the Tier 2 converters and their tests are unchanged. +- **Explainability now reports what actually ran.** A failed LLM rerank reports + `rerank=heuristic` rather than `rerank=llm`, and `layers_attempted` lists + only layers that were invoked. +- **Toolchain is pinned** to Rust 1.97 via `rust-toolchain.toml`. + +### Not changed + +Storage format, gRPC API, config file schema, and hook payloads are unchanged. +No data migration is required. + +--- + ## v2.1.0 to v2.2.0 (Multi-Agent Ecosystem) **Release Focus:** Cross-agent discovery, multi-adapter support, and ecosystem documentation @@ -337,6 +409,7 @@ memory-daemon admin rebuild-index --type all | Version | Release Date | Key Changes | |---------|--------------|-------------| +| v3.1.0 | 2026-08-31 | Phase 54-57: Make It True — orchestrator wired, honest benchmarks, README/LICENSE, OpenCode removed | | v2.2.0 | TBD | Phase 18-23: Multi-agent ecosystem, cross-agent discovery, CLOD format | | v2.1.0 | TBD | Phase 16-17: Ranking enhancements, index lifecycle | | v2.0.0 | 2026-02-01 | Topic graph, vector search, hybrid search | @@ -344,4 +417,4 @@ memory-daemon admin rebuild-index --type all --- -*Last Updated: 2026-02-06* +*Last Updated: 2026-08-31* diff --git a/docs/launch/launch-copy.md b/docs/launch/launch-copy.md new file mode 100644 index 0000000..931b209 --- /dev/null +++ b/docs/launch/launch-copy.md @@ -0,0 +1,163 @@ +# Launch copy — v3.1.0 + +Drafts for Phase 58. Nothing here has been posted. Read the notes at the bottom +before posting anything. + +--- + +## Show HN + +**Title** (80 char limit; the guideline is "Show HN: " plus a plain description, +no adjectives, no exclamation marks): + +``` +Show HN: Agent-Memory – local-first memory for AI coding agents, zero-token capture +``` + +**Text:** + +``` +Every memory layer I looked at has the same capture contract: the agent decides +what is worth remembering and calls an API. That costs tokens on every turn, and +it gets skipped exactly when the context window is under pressure — which is when +memory matters most. + +Agent-Memory captures passively instead. CLI hooks (SessionStart, UserPromptSubmit, +PostToolUse, Stop) pipe events into an append-only RocksDB log. The agent is not in +the loop and pays nothing. Retrieval is a time-hierarchical table of contents the +agent drills down — Year → Month → Week → Day → Segment → Grip — rather than a +similarity search over everything, so answering "what were we talking about last +week?" doesn't mean loading last week into context. + +It's Rust. Everything is local: the event log, a Tantivy BM25 index, an HNSW vector +index, and Candle embeddings all run on your machine. Nothing leaves it unless you +turn on LLM summarization or reranking and supply a key. + +The README has a status table saying which parts are solid, which are experimental, +and which are not implemented, because the last four phases of this project were +spent finding out that my own verification process had been grading claims instead +of behavior. Some specifics, since they're more interesting than the pitch: + +- A crate I'd certified as "integrated" was unreachable from any shipped binary +- A benchmark showing 64.6s TOC navigation was a harness defect — it timed + ingest-time summarization and labelled it navigation. Real warm p50 is 0.13ms +- The quickstart in my own README returned zero results for content just ingested, + on a fresh install, with exit code 0 and no error + +There is no benchmark comparison in the repo. The only committed results are +mock-backend runs that test the harness, not the retrieval quality, and I'm not +publishing a LoCoMo number until I've run it properly. + +https://github.com/SpillwaveSolutions/agent-memory +``` + +--- + +## r/rust + +**Title:** + +``` +Agent-Memory 3.1: local-first conversational memory for AI coding agents (Rust, RocksDB + Tantivy + HNSW) +``` + +**Body:** + +``` +Local-first memory layer for AI coding CLIs — the agent's conversation history +becomes a queryable store instead of evaporating at the end of a session. + +Stack, since that's what this sub actually cares about: + +- **RocksDB** for the append-only event log (immutable; summaries derived out of band) +- **Tantivy** for BM25 keyword retrieval +- **usearch** HNSW for vector retrieval, with **Candle** running the embedding model + locally — no Python, no inference service +- **tonic**/gRPC daemon, **clap** CLI, **tokio-cron-scheduler** for the background + rollup and indexing jobs +- Toolchain pinned via `rust-toolchain.toml` after a floating-stable Clippy lint + reddened main overnight + +The design bit I'd be interested in feedback on: retrieval is *navigational* rather +than purely similarity-based. A time hierarchy (Year → Month → Week → Day → Segment) +is built by scheduled rollup jobs, and queries drill down it, reading a summary at +each level. Three layers (BM25, vector, topic graph) feed a fusion step. The intent +is that answering "what did we decide about auth last month" reads a handful of +summary nodes rather than embedding-searching the entire corpus. + +This release added no features. It was four phases of closing the gap between what +the project claimed and what it did — including a benchmark number that turned out +to be timing the wrong thing by five orders of magnitude, and a first-run bug where +the daemon accepted events and answered every query with an empty result set, +successfully, because the index directories didn't exist yet. + +MIT. Builds on stable 1.97. Feedback on the retrieval architecture very welcome. + +https://github.com/SpillwaveSolutions/agent-memory +``` + +--- + +## r/LocalLLaMA + +**Title:** + +``` +Agent-Memory: fully local memory for coding agents — RocksDB + BM25 + local embeddings, nothing phones home +``` + +**Body:** + +``` +Built this because every agent memory option I found was a hosted API, and I do +client work where the conversation history is the sensitive artifact. + +Everything runs on your box: + +- Append-only event log (RocksDB) +- BM25 keyword index (Tantivy) +- Vector index (HNSW) with embeddings generated locally by Candle — + all-MiniLM-L6-v2 downloaded once, then no network +- gRPC daemon + CLI + +No account, no service, no telemetry. LLM summarization and LLM reranking are +optional and off unless you supply a key; without one it falls back to a heuristic +ranker and *says so* in the response payload rather than pretending. + +Capture is passive — CLI hooks pipe events in, so the model never spends tokens +deciding what to remember and can't skip it when the context fills up. Works +alongside Claude Code and Codex CLI today (Gemini and Copilot are best-effort). + +Honest status, since this sub can smell marketing: BM25 works well and is exact-token +(no stemming). Vector search works but the retrieval quality is not benchmarked — the +only committed benchmark artifacts are mock-backend runs that test the harness, and I +am not publishing a LoCoMo number until I've run a real one. The README has a table +listing what's solid, what's experimental, and what's not implemented. + +MIT, Rust, Linux/macOS. + +https://github.com/SpillwaveSolutions/agent-memory +``` + +--- + +## Before posting + +1. **Tag `v3.1.0` first** so the release links resolve and Show HN visitors can + download binaries rather than compile. +2. **Set the GitHub repo description and topics** (`ai-agents`, `memory`, `rust`, + `claude-code`, `local-first`) and enable Discussions. A Show HN landing on a + repo with no description reads as abandoned. +3. **Post the blog post first**, then link it from the HN thread as a comment + rather than submitting it as the HN URL — the repo is the better submission. +4. **Sequence, don't shotgun.** HN and one subreddit on day one; the second + subreddit later. Simultaneous posts read as a launch campaign and get less + patience from commenters. +5. **Be around for the first three hours.** The platform-risk question ("won't + Anthropic just build this in?") will come up; the answer is in + `docs/positioning/agent-memory-vs-competition.md` and should be given in your + own words, not pasted. +6. **Do not add a benchmark claim under pressure.** If someone asks how it scores + on LoCoMo, the answer is "I haven't run it properly, so I'm not going to quote a + number" — which is a stronger answer here than a weak score, and the whole + reason the last milestone existed. diff --git a/docs/launch/v3.1-blog-post.md b/docs/launch/v3.1-blog-post.md new file mode 100644 index 0000000..a5cadc6 --- /dev/null +++ b/docs/launch/v3.1-blog-post.md @@ -0,0 +1,195 @@ +# My AI pipeline graded an unrun benchmark 17/17. Here's what fixing that took. + +*Draft for Hightower's AI Harness Engineering. Status: ready for a human edit +pass. Every number and quote in here is checkable in the repo — links inline.* + +--- + +There is a failure mode in agentic development that no amount of code review +catches, because the code is fine. + +I build [agent-memory](https://github.com/SpillwaveSolutions/agent-memory), a +local-first conversational memory system for AI coding agents. It runs a +GSD-style planning loop: milestones decompose into phases, phases into plans, +each plan produces a VERIFICATION document that checks requirements off. Sixty +thousand lines of Rust, a hundred-odd phases of history, the whole thing driven +by agents working through planning files. + +At the end of milestone v3.0, the verification document said **17 of 17 +requirements met, no gaps.** + +The benchmark it certified had never been run. The crate it certified as +"integrated" was not reachable from any binary that ships. `cargo tree -i +memory-orchestrator` returned nothing — no shipped binary depended on it. It +was 3,000 lines of retrieval orchestration that existed, compiled, passed its +own unit tests, and could not be called by anything a user runs. + +Nothing lied. That is the part worth sitting with. + +## Verification by existence + +Every check in that document was true as written. "Benchmark suite implemented" +— there was a benchmark suite. "Orchestrator integrated" — there was an +integration module with tests. "Retrieval reranking implemented" — there was a +reranker. + +The checks tested for the *existence of structures*, not the *occurrence of +events*. A struct exists. A flag is defined. A test file is present. Each is +trivially confirmable by an agent reading the codebase, and none of them tells +you whether the thing ever ran. + +This is what I now call **verification by existence**, and it is the natural +attractor for any agentic pipeline that grades its own work. The agent writing +the verification has the code in context. Confirming a struct exists costs one +grep. Confirming a benchmark produced a number requires running it, waiting, +and reading an artifact that may not be there. Under any pressure at all — +context, time, a long plan — the loop converges on the cheap check. Not through +deception. Through gradient. + +The tell was there the whole time: **17/17, no gaps.** Real engineering work +does not come out clean. A verification document with no unchecked boxes is +evidence about the checker, not the code. + +## The four rules + +The fix was not better prompting. It was changing what counts as a check. + +**1. Execution-evidence rule.** Any requirement whose truth depends on +something running — a benchmark, an end-to-end behavior, a performance number, +a quickstart — must cite a committed artifact produced by actually running it. +A results file. A CI run. A transcript. The existence of a struct, a flag, or a +test file verifies the existence of a struct, a flag, or a test file. + +**2. Reachability rule.** A phase that adds a crate is not complete until +`cargo tree -i ` shows a dependent binary — or the phase explicitly +declares the crate dormant with a tracking requirement. "Integrated" now has a +mechanical definition that an agent cannot argue its way around. + +**3. `human_verification` items are blockers by default** when they gate the +milestone's stated goal. In v3.0 the *entire goal* was the benchmark, and the +benchmark item was waved through as a nice-to-have. + +**4. Write the retrospective.** The v3.0 retrospective did not exist until this +milestone. Its first line now reads: *the code is strong; the claims were not.* + +## What four phases of enforcement actually turned up + +I ran a milestone — "Make It True" — that added no features. Only closing the +gap between claim and reality. Here is what came out, because the specifics are +more useful than the principle. + +**The orchestrator got wired.** `RouteQuery` now calls it, and the CLI is a +client of that RPC. `cargo tree -i memory-orchestrator` returns a path to a +shipped binary. Rule 2, applied once, closed a gap that had been open for a +whole milestone. + +**Hybrid search was not hybrid.** It fetched one layer. It now fetches BM25 and +vector and fuses them. + +**The BM25 outbox drain was a no-op.** Events went into the queue and nothing +came out. Now they are indexed, and a miss warns and increments a counter +instead of passing silently. + +**The 65-second number was a harness defect.** A recorded benchmark showed TOC +navigation at 64,576 ms p50 — catastrophic for a system whose entire pitch is +"answer questions about last week without scanning everything." The timer +bracketed *ingest-time summarization rollup* of a 240-event segment and labelled +the result "navigation." The actual warm query p50 is **0.13 ms**. The p90 and +p99 in that old file were interpolated from three samples; the harness now +refuses to emit p90 below 10 samples and p99 below 30. + +Note the direction. Honest measurement did not make the numbers worse. It made +them *five orders of magnitude better* and retracted a claim that had been +quietly poisoning my own roadmap. + +**Then a second-order problem.** With the orchestrator wired, the explainability +payload — the thing that tells you which retrieval layers ran and how results +were ranked — was itself lying. A failed LLM rerank reported `rerank=llm`. +`layers_attempted` listed layers that had returned nothing. Client-supplied +stop conditions were echoed back in the response while being ignored by the +code. The feature whose entire job was *telling the truth about what ran* was +not telling the truth about what ran. + +That is the thing about this failure mode. It has a second floor. + +## The one that got me + +Last phase. Shop window: write a README, a license, a positioning document. +Make the repo legible to a stranger. Low risk, mostly prose. + +I wrote the README first, then ran its quickstart verbatim on a clean container +— no toolchain, no store, nothing cached. Build. Start the daemon. Add two +facts. Search for one of them. + +```console +$ memory add --content "We chose RS256 over HS256 for the auth service JWTs" --agent claude +{"status":"ok","results":{"created":true,"event_id":"01M1ABYF8RR4F8JCGGE2Z0V864"}} + +$ memory search "which JWT signing algorithm did we pick" --top 5 +{"status":"ok","results":[],"meta":{"retrieval_ms":0,"confidence":0.0}} +``` + +Empty. `confidence 0.0`. No error, no warning, exit code zero. + +A fresh store has no index directories. The outbox indexing job only registers +if its directory already exists. So a first-run daemon accepted events forever +and answered every single query with nothing — successfully. + +Every unit test passed. Every integration test passed. CI was green on both +operating systems. The product did not work if you had never used it before. + +It gets better. The remedy the daemon itself printed in its startup log — +"run rebuild-indexes to initialize the search index" — failed on the RocksDB +lock while the daemon was running, and reported "No documents found" when it +wasn't, because it indexes summary nodes rather than raw events. And a sibling +command, `admin rebuild-toc`, printed "TOC rebuild not yet fully implemented" +and **exited 0**, while its `--dry-run` mode advertised "to actually rebuild, +run without --dry-run." + +Three commands. Three different ways of reporting success while doing nothing. + +None of this is exotic. It is the most ordinary bug in software: the +happy path nobody walks because everybody's machine is already warm. What made +it invisible for months was that my verification loop had no step that started +from nothing. + +## The generalization + +If you are running an agentic pipeline that produces its own verification, the +question is not whether your agents are honest. Mine were. The question is +whether your checks can be satisfied without the thing happening. + +Three that transfer: + +- **Grade on artifacts, not assertions.** "The benchmark is implemented" is + free. "Here is `results.json` with a timestamp and a git SHA" is not. Make + the expensive one the only one that counts. +- **Reachability is mechanical — so mechanize it.** "Is this integrated?" is a + judgment call an agent will resolve generously. `cargo tree -i` is not. + Every ecosystem has an equivalent. Use it. +- **Start from nothing, regularly.** Not a test fixture. An actual clean + machine, following your actual documentation, in order, adding nothing. It is + the single highest-yield hour I spent across four phases. + +And one that is less about tooling: **a verification document with no unchecked +boxes is a red flag, not a milestone.** The current one has four items in a +section headed "Not done, stated rather than waved through." That is what a +real one looks like. + +## Where the project is + +v3.1 ships today. No new features — the whole milestone was closing this gap. +The [README](https://github.com/SpillwaveSolutions/agent-memory) has a status +table that says which parts are solid, which are experimental, and which are +not implemented. The +[positioning writeup](https://github.com/SpillwaveSolutions/agent-memory/blob/main/docs/positioning/agent-memory-vs-competition.md) +compares it against Mem0, Zep, MemMachine and Letta, including a section on +where they are ahead. + +It makes no benchmark comparison. The only committed results are mock-backend +and mock-judge runs, which measure whether the harness works, not whether the +memory is good. MemMachine publishes 0.9169 on LoCoMo. I publish nothing on +LoCoMo, because I have not run it properly, and the point of the last four +phases was to stop doing exactly that. + +The quickstart works now. I know, because I ran it. diff --git a/docs/setup/quickstart.md b/docs/setup/quickstart.md index 6a64f24..ee4fb21 100644 --- a/docs/setup/quickstart.md +++ b/docs/setup/quickstart.md @@ -57,15 +57,26 @@ cargo build --release - [ ] Download the latest release for your platform - [ ] Unpack to a local bin directory +One archive per platform contains all four binaries: `memory-daemon`, +`memory-ingest`, `memory` (the CLI), and `memory-installer`. Assets are named +`agent-memory--.tar.gz`, and the tarball unpacks into a +directory of that name. + ```bash mkdir -p ~/.local/bin -PLATFORM=$(uname -s | tr '[:upper:]' '[:lower:]') -ARCH=$(uname -m) -curl -L "https://github.com/SpillwaveSolutions/agent-memory/releases/latest/download/memory-daemon-${PLATFORM}-${ARCH}.tar.gz" | tar xz -C ~/.local/bin -curl -L "https://github.com/SpillwaveSolutions/agent-memory/releases/latest/download/memory-ingest-${PLATFORM}-${ARCH}.tar.gz" | tar xz -C ~/.local/bin -chmod +x ~/.local/bin/memory-daemon ~/.local/bin/memory-ingest +VERSION=3.1.0 +case "$(uname -s)" in Darwin) OS=macos ;; Linux) OS=linux ;; esac +case "$(uname -m)" in x86_64|amd64) ARCH=x86_64 ;; arm64|aarch64) ARCH=aarch64 ;; esac +ASSET="agent-memory-${VERSION}-${OS}-${ARCH}" + +curl -fL "https://github.com/SpillwaveSolutions/agent-memory/releases/download/v${VERSION}/${ASSET}.tar.gz" \ + | tar xz -C /tmp +install -m 0755 /tmp/${ASSET}/memory-daemon /tmp/${ASSET}/memory-ingest \ + /tmp/${ASSET}/memory /tmp/${ASSET}/memory-installer ~/.local/bin/ ``` +Verify the download against the release's `SHA256SUMS.txt` if you want to. + ### 3) Add binaries to PATH - [ ] Ensure `~/.local/bin` is on your PATH @@ -77,6 +88,7 @@ export PATH="$HOME/.local/bin:$PATH" > Verify now (optional): > - `memory-daemon --version` > - `memory-ingest --version` +> - `memory --version` ### 4) Create a minimal config (single-agent defaults)