Correction: close the gaps a memory-systems survey found, and gate the eval - #13
Merged
Conversation
…t line
Doc chunks were being retrieved and then ignored. Measured on a real
store via recall_event: chunks are *shown* 1542 times against memories'
1589 — retrieval is not the problem — but at the same rank position
memories get opened 33.5% of the time and chunks 12.5%. Chunks also
reach the top 3 half the time, so ranking is not the problem either.
The problem is the line itself. `agent_line` truncated the body from its
first character, with no idea what the query was, at
output.snippet_chars (120). That is survivable for a memory, whose title
is a self-contained claim. It is not for a doc chunk, whose title is a
breadcrumb ("Doc › Section › Sub") — 92% of them are — so the body
snippet is the only thing saying why the hit came back. Asked "prompt
cache breakpoints", the top hit's preview read "You cannot compress text
and have the API bill fewer tokens…", the section's opening sentence,
containing none of the query. The chunk did answer the question, 400
chars later.
`agent_line_for_query` takes the query and picks the snippet_chars-wide
window covering the most *distinct* query stems. Wired at the two call
sites that have a query in hand — CLI recall and MCP recall. The other
four (remember-echo, anchor trigger, single-node echoes) genuinely have
none and are byte-identical; `agent_line` delegates with None, the shape
`search_hybrid_scored`/`search_hybrid_with_legs_scored` already uses.
Tokenisation reuses memory::tokens and fts::is_stopword so the snippet
cannot drift from what the FTS leg actually matched.
Four things were measured rather than assumed:
- Density, not first match. Centring on the earliest match let an
incidental early mention hide the passage that answers; 39% of hits
had a strictly better window. Worth +21% distinct terms shown (1.19 ->
1.44) over first-match on real data.
- Occurrences are indexed once and scored by binary search. The first
cut re-scanned the window text per candidate per stem, which on a
repetitive body degrades badly: 26 of 3000 real bodies cost >1ms each,
worst 1.64ms, and a synthetic log-like body 3.78ms — enough to blow a
single-digit-ms warm recall on one hit. Now: 0 of 3000 over 1ms, worst
~430µs, synthetic 273µs, normal 8-hit recall 92µs.
- MATCH_SCAN_CHARS caps the hunt at 20k chars. Unbounded was 7.4ms on
the largest chunk in the store (215k chars). Past the cap we
head-truncate, i.e. exactly the prior behaviour, for 83 of ~19k chunks.
- Lowercasing is ASCII-only, so the char view stays 1:1 with the
original. `char::to_lowercase` is not length-preserving (U+0130 emits
two chars), and an index computed in the lowercased copy then used to
slice the original panicked: "range start index 368 out of range for
slice of length 220". Regression test included.
Word-anchoring the matches was implemented and reverted. It removes the
3% of matches landing inside an unrelated word ("use" in "because";
STEM_PREFIX truncating "forget" to "forge", which then hits
"bookforge"), but also drops legitimate matches inside compound
identifiers: docs unchanged, memory previews 2pp worse. Density scoring
already absorbs the strays. The negative result is recorded on
`occurrences` so it is not re-derived.
Verified against the installed binary over 24 queries, 192 hits per kind
— previews showing at least one query term: doc chunks 53% -> 91%,
memories 38% -> 85%; average distinct terms 0.80 -> 1.46 and 0.47 ->
1.29. Ranked ids and their order are identical in every case; this
changes what a hit says, never which hits come back. fmt, clippy
-D warnings and 362 tests pass. The retrieval eval is unchanged as
expected — it scores ranked id lists and is structurally blind to
snippet text, so it guards the "ranking untouched" half of this and
nothing else.
Not addressed, deliberately: the breadcrumb titles themselves, which
spend ~75 chars on location rather than content. That belongs in
index::chunker where they are built, and would shift ranking, since
title carries the heaviest BM25 weight. The deeper fix for both is to
make Hit carry the real matched span from the search layer, which would
let FTS5's snippet() do this with the actual porter stemmer instead of a
5-char prefix approximation — a three-file structural change, tracked
separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two findings from grading Mimir against an external survey of memory
systems, whose sharpest metric was "can a deleted fact be silently
re-extracted." Mimir could. Reproduced before fixing:
remember → m:M7FQ29 ; forget m:M7FQ29 ; recall → no results
remember (same text) → m:9B80AP, no warning, and it's back
`find_duplicate` filtered `deleted_at IS NULL` on the exact-hash path and
passed `include_superseded: false` on the near-dup path, so the tombstone
was invisible to the one mechanism that could have caught the re-add. An
importer re-run, an extractor re-reading an unchanged source, or an agent
that saw the fact twice each quietly undid the delete.
`remember` now refuses with the date, and `--force` overrides. Details
that turned out to matter more than the main change:
- Precedence. An exact *live* copy stays an ordinary duplicate even when
an older tombstone of the same text exists, or `--force` would be a
one-way door: revive once, and every later capture is refused forever.
- Rewording. The hash pass alone missed "…at 03:00 utc." because `tokens`
keeps trailing punctuation whole for `file.rs`, so a token-overlap pass
runs over deliberate tombstones too. That is the common shape of
re-extraction, not an edge case.
- Decay archival is excluded. `consolidate` soft-deletes idle low-strength
nodes with `meta.archived = 1`; nobody decided that. Treating it as a
deletion would refuse re-learning ordinary facts and train everyone to
pass `--force` by reflex, at which point the guard on real deletions
stops meaning anything.
- MCP has no `force` parameter, so an agent cannot resurrect what a human
deleted. Import counts these separately from duplicates and says so.
Second: the retrieval eval was a report, not a gate. Corpus committed and
deterministic, but the only assertions were well-formedness with an
`MRR > 0.2` floor, and every baseline was `#[ignore]`d — a ranking change
could land with nothing to notice. Added a committed per-set baseline that
must reproduce exactly in *both* directions (an improvement nobody wrote
down is also unreviewed) and an ablation per scoring knob. Verified by
zeroing `type_prior_alpha`: previously silent, now two failures naming the
metric and the delta.
That ablation immediately found something: `scoring.code_damp` moves
nothing on this corpus, in any category, including the `code-vs-memory`
scenario that exists to guard it — its one question ranks the memory first
with or without the damp. Pinned by a test that fails when it stops being
inert, rather than asserting a win that isn't there.
fmt, clippy -D warnings, 369 tests pass; the forget→re-remember path also
verified end to end against the real binary.
…n't a leak
Two of the three axes where Mimir read thinner than the best of the 151
open-source memory systems in Yves' survey. Grounding is flag-only per
decision — reported, never scored.
## Grounding (`mimir grounding`, `doctor`, inline on `get`)
Every other signal on a memory is a policy: strength rises because it got
used, a type prior favours gotchas, a human marks it useful. None can be
proven wrong by the code that assigns them — they are opinions about a
claim, computed from how people treated the claim.
A link to an indexed artifact is different. The indexer and `graph build`
soft-delete what stops existing, so "this note is about
`retry_with_backoff`" is checkable, and stops being true when the symbol
goes. Verified the whole cycle against the real binary: link by name →
1 grounded / 0 stale; delete the source, reindex, rebuild → 0 grounded /
1 stale, and the memory itself now carries
! grounding stale: symbol retry_with_backoff is no longer indexed
Stale is not "wrong" — a note about a renamed function is usually still
good. It means nobody has revisited it since the ground moved, and the
wording says so, because a status people learn to ignore is worse than no
status. A live artifact beats a dead one, so ordinary re-chunking (which
soft-deletes and re-inserts) can't flap it. Links to other memories ground
nothing: two assertions agreeing is not evidence, and counting them would
make grounding trivially self-satisfiable.
## Refusal ledger (`mimir refusals`)
The secret guard refused things and kept no record at all, so nothing
could tell you something had been refused, or that the same value was
being retried. It now records a blake3 fingerprint, the detector's label,
the surface, and first/last seen with a count — never the value. Repeats
increment one row: one secret offered forty times is an agent in a loop,
and forty rows would bury that. Guarded by a test that sweeps every column
of every table for the plaintext, because a record of a leak that contains
the leak is worse than no record — it's a second copy in a table nobody
audits.
## Found while building it
`remember --link <symbol>` never worked. Both surfaces advertise "a code
symbol or node"; the CLI only called `resolve_ref`, which resolves ids, so
linking to a symbol by name failed outright. MCP already had the fallback.
Left alone this would have shipped grounding with its main entry point
broken — the same shape as anchors sitting at 0 of 612 adoption because
the only way to set one was at capture time.
New migration (refusal table); upgrade path covered by a test that keeps
an existing store's rows and writes the new table, written against
`MIGRATIONS.len() - 1` so it guards whatever lands next rather than
rotting.
fmt, clippy -D warnings, 381 tests. Grounding, ledger and the link fix
each verified end to end against the real binary, not just in unit tests.
…d it is Last of the three axes from Yves' survey. Mimir tracked how often a memory gets used (`strength`, decaying), and whether it has been retired (`superseded_by`, `expires_at`, grounding) — but had no way to say "I was guessing." One scalar carried both, which is how a guess becomes canon: recall it enough and it outranks things that were checked, with nothing left to show it was ever uncertain. `--confidence certain|likely|unsure`, and the same on the MCP tool. Author-declared at capture, deliberately, on the argument `expires_at` and `resolves_when` already rest on: the person writing it is the only one who knows how sure they were, they know it at write time, and nobody ever backfills. A level inferred later by a model is precisely the unfalsifiable policy label this exists to replace — computed from the text, therefore incapable of contradicting the text. Two calls worth stating: - Absent is a real state, not a synonym for `likely`. Defaulting an undeclared memory to the middle puts words in the author's mouth, the same "reject, don't mangle" rule `parse_expires_in` and `sanitize_fires_when` already follow. An unparseable level errors before anything is written, for the same reason a bad `--expires-in` does. - Only `unsure` reaches the compact recall line. That line is what an agent acts from, so the case worth a token is the one where acting without checking would be wrong; printing "certain" on every hit is reassurance nobody asked for at a cost per recall. `get` shows any declared level in full, attributed, and says it isn't a ranking signal. Stored in `meta`, so it replicates and needs no migration. Nothing in `search` or `learn` reads it — verified: the hermetic drift-eval baseline reproduces byte-identically, which is the gate landed two commits ago doing its job on the first change that could have moved it. Verified end to end: a memory declared `unsure` and then opened 8 times shows `↑8` and is still `unsure` — usage moved, the claim did not, which is the whole point. fmt, clippy -D warnings, 384 tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four commits. One fixes something that was outright broken; the rest close
the axes where Mimir read thinner than the best of the 151 open-source
memory systems in Yves' survey, whose sharpest metric was "can a deleted
fact be silently re-extracted." Mimir's could.
A forgotten fact could walk back in (
428508e)Reproduced before fixing:
Duplicate detection only ever looked at live nodes, so a tombstone left
nothing that could catch the re-add. An importer re-run, an extractor
re-reading an unchanged source, or an agent that saw the fact twice each
quietly undid the delete.
remembernow refuses with the date it was forgotten on. Three detailsthat mattered more than the main change:
an older tombstone exists — otherwise
--forceis a one-way door."…03:00 utc.", becausetokenskeeps trailing punctuation whole sofile.rssurvives. Atoken-overlap pass over tombstones catches it; a reword is the common
shape of re-extraction, not an edge case.
consolidatesoft-deletes idlelow-strength nodes with
meta.archived = 1. Nobody decided those, andtreating them as deletions would train everyone to
--forceby reflex,which is how the guard on real deletions stops meaning anything.
MCP has no
forceparameter, so an agent cannot resurrect what a humandeleted.
The eval became a gate (
428508e)The corpus was committed and deterministic, but every baseline was
#[ignore]d and the only live assertions were well-formedness checks withan
MRR > 0.2floor — a ranking change could land with nothing to notice.Now on plain
cargo test: a per-set baseline that must reproduceexactly in both directions (an improvement nobody wrote down is also
unreviewed), plus a per-knob ablation. Verified by zeroing
type_prior_alpha: previously green, now two failures naming the metricand delta.
It immediately found that
scoring.code_dampmoves nothing on thecorpus — same numbers in every category, including the
code-vs-memoryscenario that exists to guard it. Pinned by a test that fails when it stops
being inert, rather than asserting a win that isn't there.
Grounding you can falsify (
287f349)Every other signal on a memory is a policy — strength rises because it got
used, a type prior favours gotchas — and none can be proven wrong by the
code that assigns it. A link to an indexed artifact can be. Verified end to
end: link by symbol name →
1 grounded; delete the source, reindex,rebuild →
1 stale, and the memory carriesFlag-only by decision: reported, never scored. A live artifact beats a dead
one so re-chunking can't flap it; links to other memories ground nothing,
since two assertions agreeing isn't evidence.
A refusal ledger that isn't itself a leak (
287f349)The secret guard refused things and kept no record, so nothing could tell
you a value was being retried. It now records a blake3 fingerprint, the
detector's label, the surface, and first/last seen with a count — never the
value. Repeats increment one row: one secret offered forty times is a loop,
and forty rows would bury that. Guarded by a test that sweeps every column
of every table for the plaintext.
Confidence, separate from retrieval strength (
74d5ec1)strengthconflated "how often this got used" with "how much to believeit", which is how a guess becomes canon.
--confidence certain|likely| unsure, author-declared at capture on the same argumentexpires_atrestson. Absent is a real state, not a synonym for
likely. Onlyunsurereaches the compact recall line — that line is what an agent acts from.
Verified: a memory declared
unsureand opened 8 times shows↑8and isstill
unsure. Usage moved, the claim didn't.Also
remember --link <symbol>never worked on the CLI — both surfacesadvertise "a code symbol or node" but the CLI only resolved ids. MCP had
the fallback already. Left alone this would have shipped grounding with its
main entry point broken.
Review notes
refusaltable). Upgrade path covered by a testthat keeps an existing store's rows and writes the new table, written
against
MIGRATIONS.len() - 1so it guards whatever lands next.scoring, and the hermetic baseline reproduces byte-identically — that is
the check, not an assertion.
clippy -D warnings, 384 tests, plus end-to-end runs againstthe real binary for the forget guard, the ledger, grounding and
confidence. Windows is unverified — this PR is what runs it, and the
last two Windows breakages here were a stack overflow and schema-shaped.