Skip to content

Correction: close the gaps a memory-systems survey found, and gate the eval - #13

Merged
MakerViking merged 4 commits into
mainfrom
feat/query-aware-snippets
Aug 9, 2026
Merged

Correction: close the gaps a memory-systems survey found, and gate the eval#13
MakerViking merged 4 commits into
mainfrom
feat/query-aware-snippets

Conversation

@MakerViking

Copy link
Copy Markdown
Owner

Four commits. One fixes something that was outright broken; the rest close
the axes where Mimir read thinner than the best of the 151 open-source
memory systems in Yves' survey, whose sharpest metric was "can a deleted
fact be silently re-extracted."
Mimir's could.

A forgotten fact could walk back in (428508e)

Reproduced before fixing:

remember → m:M7FQ29 ; forget m:M7FQ29 ; recall → no results
remember (same text) → m:9B80AP, no warning, and it's back

Duplicate detection only ever looked at live nodes, so a tombstone left
nothing that could catch the re-add. An importer re-run, an extractor
re-reading an unchanged source, or an agent that saw the fact twice each
quietly undid the delete.

remember now refuses with the date it was forgotten on. Three details
that mattered more than the main change:

  • Precedence. An exact live copy stays an ordinary duplicate even when
    an older tombstone exists — otherwise --force is a one-way door.
  • Rewording. The hash pass alone missed "…03:00 utc.", because
    tokens keeps trailing punctuation whole so file.rs survives. A
    token-overlap pass over tombstones catches it; a reword is the common
    shape of re-extraction, not an edge case.
  • Decay archival is excluded. consolidate soft-deletes idle
    low-strength nodes with meta.archived = 1. Nobody decided those, and
    treating them as deletions would train everyone to --force by reflex,
    which is how the guard on real deletions stops meaning anything.

MCP has no force parameter, so an agent cannot resurrect what a human
deleted.

The eval became a gate (428508e)

The corpus was committed and deterministic, but every baseline was
#[ignore]d and the only live assertions were well-formedness checks with
an MRR > 0.2 floor — a ranking change could land with nothing to notice.
Now on plain cargo test: a per-set baseline that must reproduce
exactly in both directions (an improvement nobody wrote down is also
unreviewed), plus a per-knob ablation. Verified by zeroing
type_prior_alpha: previously green, now two failures naming the metric
and delta.

It immediately found that scoring.code_damp moves nothing on the
corpus
— same numbers in every category, including the code-vs-memory
scenario that exists to guard it. Pinned by a test that fails when it stops
being inert, rather than asserting a win that isn't there.

Grounding you can falsify (287f349)

Every other signal on a memory is a policy — strength rises because it got
used, a type prior favours gotchas — and none can be proven wrong by the
code that assigns it. A link to an indexed artifact can be. Verified end to
end: link by symbol name → 1 grounded; delete the source, reindex,
rebuild → 1 stale, and the memory carries

! grounding stale: symbol retry_with_backoff is no longer indexed

Flag-only by decision: reported, never scored. A live artifact beats a dead
one so re-chunking can't flap it; links to other memories ground nothing,
since two assertions agreeing isn't evidence.

A refusal ledger that isn't itself a leak (287f349)

The secret guard refused things and kept no record, so nothing could tell
you a value was being retried. It now records a blake3 fingerprint, the
detector's label, the surface, and first/last seen with a count — never the
value. Repeats increment one row: one secret offered forty times is a loop,
and forty rows would bury that. Guarded by a test that sweeps every column
of every table for the plaintext.

Confidence, separate from retrieval strength (74d5ec1)

strength conflated "how often this got used" with "how much to believe
it", which is how a guess becomes canon. --confidence certain|likely| unsure, author-declared at capture on the same argument expires_at rests
on. Absent is a real state, not a synonym for likely. Only unsure
reaches the compact recall line — that line is what an agent acts from.

Verified: a memory declared unsure and opened 8 times shows ↑8 and is
still unsure. Usage moved, the claim didn't.

Also

remember --link <symbol> never worked on the CLI — both surfaces
advertise "a code symbol or node" but the CLI only resolved ids. MCP had
the fallback already. Left alone this would have shipped grounding with its
main entry point broken.

Review notes

  • New migration (the refusal table). Upgrade path covered by a test
    that keeps an existing store's rows and writes the new table, written
    against MIGRATIONS.len() - 1 so it guards whatever lands next.
  • No ranking change. Grounding and confidence are both inert in
    scoring, and the hermetic baseline reproduces byte-identically — that is
    the check, not an assertion.
  • Local: fmt, clippy -D warnings, 384 tests, plus end-to-end runs against
    the real binary for the forget guard, the ledger, grounding and
    confidence. Windows is unverified — this PR is what runs it, and the
    last two Windows breakages here were a stack overflow and schema-shaped.

MakerViking and others added 4 commits August 7, 2026 06:08
…t line

Doc chunks were being retrieved and then ignored. Measured on a real
store via recall_event: chunks are *shown* 1542 times against memories'
1589 — retrieval is not the problem — but at the same rank position
memories get opened 33.5% of the time and chunks 12.5%. Chunks also
reach the top 3 half the time, so ranking is not the problem either.

The problem is the line itself. `agent_line` truncated the body from its
first character, with no idea what the query was, at
output.snippet_chars (120). That is survivable for a memory, whose title
is a self-contained claim. It is not for a doc chunk, whose title is a
breadcrumb ("Doc › Section › Sub") — 92% of them are — so the body
snippet is the only thing saying why the hit came back. Asked "prompt
cache breakpoints", the top hit's preview read "You cannot compress text
and have the API bill fewer tokens…", the section's opening sentence,
containing none of the query. The chunk did answer the question, 400
chars later.

`agent_line_for_query` takes the query and picks the snippet_chars-wide
window covering the most *distinct* query stems. Wired at the two call
sites that have a query in hand — CLI recall and MCP recall. The other
four (remember-echo, anchor trigger, single-node echoes) genuinely have
none and are byte-identical; `agent_line` delegates with None, the shape
`search_hybrid_scored`/`search_hybrid_with_legs_scored` already uses.
Tokenisation reuses memory::tokens and fts::is_stopword so the snippet
cannot drift from what the FTS leg actually matched.

Four things were measured rather than assumed:

- Density, not first match. Centring on the earliest match let an
  incidental early mention hide the passage that answers; 39% of hits
  had a strictly better window. Worth +21% distinct terms shown (1.19 ->
  1.44) over first-match on real data.

- Occurrences are indexed once and scored by binary search. The first
  cut re-scanned the window text per candidate per stem, which on a
  repetitive body degrades badly: 26 of 3000 real bodies cost >1ms each,
  worst 1.64ms, and a synthetic log-like body 3.78ms — enough to blow a
  single-digit-ms warm recall on one hit. Now: 0 of 3000 over 1ms, worst
  ~430µs, synthetic 273µs, normal 8-hit recall 92µs.

- MATCH_SCAN_CHARS caps the hunt at 20k chars. Unbounded was 7.4ms on
  the largest chunk in the store (215k chars). Past the cap we
  head-truncate, i.e. exactly the prior behaviour, for 83 of ~19k chunks.

- Lowercasing is ASCII-only, so the char view stays 1:1 with the
  original. `char::to_lowercase` is not length-preserving (U+0130 emits
  two chars), and an index computed in the lowercased copy then used to
  slice the original panicked: "range start index 368 out of range for
  slice of length 220". Regression test included.

Word-anchoring the matches was implemented and reverted. It removes the
3% of matches landing inside an unrelated word ("use" in "because";
STEM_PREFIX truncating "forget" to "forge", which then hits
"bookforge"), but also drops legitimate matches inside compound
identifiers: docs unchanged, memory previews 2pp worse. Density scoring
already absorbs the strays. The negative result is recorded on
`occurrences` so it is not re-derived.

Verified against the installed binary over 24 queries, 192 hits per kind
— previews showing at least one query term: doc chunks 53% -> 91%,
memories 38% -> 85%; average distinct terms 0.80 -> 1.46 and 0.47 ->
1.29. Ranked ids and their order are identical in every case; this
changes what a hit says, never which hits come back. fmt, clippy
-D warnings and 362 tests pass. The retrieval eval is unchanged as
expected — it scores ranked id lists and is structurally blind to
snippet text, so it guards the "ranking untouched" half of this and
nothing else.

Not addressed, deliberately: the breadcrumb titles themselves, which
spend ~75 chars on location rather than content. That belongs in
index::chunker where they are built, and would shift ranking, since
title carries the heaviest BM25 weight. The deeper fix for both is to
make Hit carry the real matched span from the search layer, which would
let FTS5's snippet() do this with the actual porter stemmer instead of a
5-char prefix approximation — a three-file structural change, tracked
separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two findings from grading Mimir against an external survey of memory
systems, whose sharpest metric was "can a deleted fact be silently
re-extracted." Mimir could. Reproduced before fixing:

    remember → m:M7FQ29 ; forget m:M7FQ29 ; recall → no results
    remember (same text) → m:9B80AP, no warning, and it's back

`find_duplicate` filtered `deleted_at IS NULL` on the exact-hash path and
passed `include_superseded: false` on the near-dup path, so the tombstone
was invisible to the one mechanism that could have caught the re-add. An
importer re-run, an extractor re-reading an unchanged source, or an agent
that saw the fact twice each quietly undid the delete.

`remember` now refuses with the date, and `--force` overrides. Details
that turned out to matter more than the main change:

- Precedence. An exact *live* copy stays an ordinary duplicate even when
  an older tombstone of the same text exists, or `--force` would be a
  one-way door: revive once, and every later capture is refused forever.
- Rewording. The hash pass alone missed "…at 03:00 utc." because `tokens`
  keeps trailing punctuation whole for `file.rs`, so a token-overlap pass
  runs over deliberate tombstones too. That is the common shape of
  re-extraction, not an edge case.
- Decay archival is excluded. `consolidate` soft-deletes idle low-strength
  nodes with `meta.archived = 1`; nobody decided that. Treating it as a
  deletion would refuse re-learning ordinary facts and train everyone to
  pass `--force` by reflex, at which point the guard on real deletions
  stops meaning anything.
- MCP has no `force` parameter, so an agent cannot resurrect what a human
  deleted. Import counts these separately from duplicates and says so.

Second: the retrieval eval was a report, not a gate. Corpus committed and
deterministic, but the only assertions were well-formedness with an
`MRR > 0.2` floor, and every baseline was `#[ignore]`d — a ranking change
could land with nothing to notice. Added a committed per-set baseline that
must reproduce exactly in *both* directions (an improvement nobody wrote
down is also unreviewed) and an ablation per scoring knob. Verified by
zeroing `type_prior_alpha`: previously silent, now two failures naming the
metric and the delta.

That ablation immediately found something: `scoring.code_damp` moves
nothing on this corpus, in any category, including the `code-vs-memory`
scenario that exists to guard it — its one question ranks the memory first
with or without the damp. Pinned by a test that fails when it stops being
inert, rather than asserting a win that isn't there.

fmt, clippy -D warnings, 369 tests pass; the forget→re-remember path also
verified end to end against the real binary.
…n't a leak

Two of the three axes where Mimir read thinner than the best of the 151
open-source memory systems in Yves' survey. Grounding is flag-only per
decision — reported, never scored.

## Grounding (`mimir grounding`, `doctor`, inline on `get`)

Every other signal on a memory is a policy: strength rises because it got
used, a type prior favours gotchas, a human marks it useful. None can be
proven wrong by the code that assigns them — they are opinions about a
claim, computed from how people treated the claim.

A link to an indexed artifact is different. The indexer and `graph build`
soft-delete what stops existing, so "this note is about
`retry_with_backoff`" is checkable, and stops being true when the symbol
goes. Verified the whole cycle against the real binary: link by name →
1 grounded / 0 stale; delete the source, reindex, rebuild → 0 grounded /
1 stale, and the memory itself now carries

    ! grounding stale: symbol retry_with_backoff is no longer indexed

Stale is not "wrong" — a note about a renamed function is usually still
good. It means nobody has revisited it since the ground moved, and the
wording says so, because a status people learn to ignore is worse than no
status. A live artifact beats a dead one, so ordinary re-chunking (which
soft-deletes and re-inserts) can't flap it. Links to other memories ground
nothing: two assertions agreeing is not evidence, and counting them would
make grounding trivially self-satisfiable.

## Refusal ledger (`mimir refusals`)

The secret guard refused things and kept no record at all, so nothing
could tell you something had been refused, or that the same value was
being retried. It now records a blake3 fingerprint, the detector's label,
the surface, and first/last seen with a count — never the value. Repeats
increment one row: one secret offered forty times is an agent in a loop,
and forty rows would bury that. Guarded by a test that sweeps every column
of every table for the plaintext, because a record of a leak that contains
the leak is worse than no record — it's a second copy in a table nobody
audits.

## Found while building it

`remember --link <symbol>` never worked. Both surfaces advertise "a code
symbol or node"; the CLI only called `resolve_ref`, which resolves ids, so
linking to a symbol by name failed outright. MCP already had the fallback.
Left alone this would have shipped grounding with its main entry point
broken — the same shape as anchors sitting at 0 of 612 adoption because
the only way to set one was at capture time.

New migration (refusal table); upgrade path covered by a test that keeps
an existing store's rows and writes the new table, written against
`MIGRATIONS.len() - 1` so it guards whatever lands next rather than
rotting.

fmt, clippy -D warnings, 381 tests. Grounding, ledger and the link fix
each verified end to end against the real binary, not just in unit tests.
…d it is

Last of the three axes from Yves' survey. Mimir tracked how often a memory
gets used (`strength`, decaying), and whether it has been retired
(`superseded_by`, `expires_at`, grounding) — but had no way to say "I was
guessing." One scalar carried both, which is how a guess becomes canon:
recall it enough and it outranks things that were checked, with nothing
left to show it was ever uncertain.

`--confidence certain|likely|unsure`, and the same on the MCP tool.
Author-declared at capture, deliberately, on the argument `expires_at` and
`resolves_when` already rest on: the person writing it is the only one who
knows how sure they were, they know it at write time, and nobody ever
backfills. A level inferred later by a model is precisely the
unfalsifiable policy label this exists to replace — computed from the
text, therefore incapable of contradicting the text.

Two calls worth stating:

- Absent is a real state, not a synonym for `likely`. Defaulting an
  undeclared memory to the middle puts words in the author's mouth, the
  same "reject, don't mangle" rule `parse_expires_in` and
  `sanitize_fires_when` already follow. An unparseable level errors before
  anything is written, for the same reason a bad `--expires-in` does.
- Only `unsure` reaches the compact recall line. That line is what an
  agent acts from, so the case worth a token is the one where acting
  without checking would be wrong; printing "certain" on every hit is
  reassurance nobody asked for at a cost per recall. `get` shows any
  declared level in full, attributed, and says it isn't a ranking signal.

Stored in `meta`, so it replicates and needs no migration. Nothing in
`search` or `learn` reads it — verified: the hermetic drift-eval baseline
reproduces byte-identically, which is the gate landed two commits ago
doing its job on the first change that could have moved it.

Verified end to end: a memory declared `unsure` and then opened 8 times
shows `↑8` and is still `unsure` — usage moved, the claim did not, which
is the whole point.

fmt, clippy -D warnings, 384 tests.
@MakerViking
MakerViking merged commit 3ab4fb5 into main Aug 9, 2026
3 checks passed
@MakerViking
MakerViking deleted the feat/query-aware-snippets branch August 9, 2026 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant