Memory Graph

A long-term memory system for LLMs, built from scratch. Talk to it; it extracts atomic memories, links them in a knowledge graph, resolves conflicts, and recalls the relevant ones later — combining semantic (embedding) search with graph traversal.

Two operations, deliberately boring:

• store(text) — turn a message into dated, attributed, atomic facts and wire them into the graph.
• recall(query) — retrieve the relevant memories via three merged strategies, then answer over them.

On the LoCoMo long-term-conversation benchmark (the one mem0 / Zep publish against), it scores 75% on a 152-question conversation using the same GPT-4o-mini judge the papers use — with temporal reasoning at 86% and multi-hop at 62%. See Benchmarks (and the honest caveats).

Left: the live force-directed memory graph (blue = Memory, orange = Entity, red = Emotion, green = Topic, purple = Event). Right: chat, with a per-answer RECALLED panel showing the scored memories that grounded each reply. Bottom: the contradiction inbox surfacing conflicting facts to resolve.

---

Architecture

Layer	Stack	Role
Backend (backend/)	FastAPI · Neo4j · Claude · OpenAI embeddings	All memory logic — store(), recall(), conflict resolution, relationships
Frontend (frontend/)	React · Vite · react-force-graph-2d	Obsidian-style graph visualizer + chat + debug panels. Read-only onto the backend

The graph is the memory. Embeddings are how you find your way into it.

Graph model

• Nodes: Memory (atomic fact + embedding + bi-temporal valid_from/valid_to + confidence + status), Entity, Emotion, Topic, Event.
• Edges: MENTIONS, EVOKES{intensity}, ABOUT, OCCURRED_AT, RELATED_TO{type} (entity↔entity), SIMILAR_TO{score}, and the conflict trio SUPERSEDES / CONTRADICTS / SUPPORTS.

How store() works

1. Extract (Claude) — split the message into atomic facts. Resolves first-person to the speaker's name ("I do X" → "John does X"), resolves relative dates to absolute ISO dates ("last Saturday" → 2023-05-20), and pulls entities, emotions, topics, events, confidence.
2. Embed each fact (OpenAI text-embedding-3-small).
3. Find similar existing memories (vector search).
4. Conflict-classify (Claude) — new fact vs each similar one → DUPLICATE / UPDATE / CONTRADICT / COMPLEMENT / UNRELATED.
5. Apply — dedup + confidence bump / supersede + time-close / flag contradiction / link. Nothing is hard-deleted; superseded memories stay for history.
6. Wire the graph — MENTIONS, EVOKES, ABOUT, OCCURRED_AT, RELATED_TO, SIMILAR_TO.

How recall() works — three merged paths

1. Vector search — the fuzzy entry point. Best for "the one fact closest to the query."
2. Entity-anchored — if the query names an entity, follow its MENTIONS edges to pull every memory about it, regardless of embedding distance. This is what makes aggregation ("what activities does she do?") and completeness work.
3. Graph-expand — from the top vector hits, pull memories sharing entities/topics/ events. The multi-hop bridge ("my sister" → Sarah → "Sarah works at Google").

Merge → dedupe → re-rank (0.6·similarity + 0.25·confidence + 0.15·recency). Aggregation questions return a larger candidate set so the answer model sees the whole set.

How /answer works

Recall → build context (each memory prefixed with its date) → Claude answers using only those memories. It's allowed to infer across them ("home country is Sweden" + "moved 4 years ago" → "Sweden") and to reason over dates. Abstains only when nothing is relevant. No writes — QA never mutates the graph.

---

Quickstart

1. Neo4j

docker compose up -d          # Neo4j at bolt://localhost:7687, browser UI http://localhost:7474

2. Backend

cd backend
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env          # add ANTHROPIC_API_KEY and OPENAI_API_KEY
uvicorn app.main:app --reload --port 8000

3. Frontend

cd frontend
npm install
npm run dev                   # http://localhost:5174  (proxies /api -> :8000)

---

Configuration (backend/.env)

ANTHROPIC_API_KEY=sk-ant-...          # extraction, conflict, answers
CLAUDE_MODEL=claude-sonnet-5          # or claude-opus-4-8 / claude-haiku-4-5

# Embeddings — provider "openai" (recommended) or "local" (sentence-transformers)
EMBED_PROVIDER=openai
EMBED_MODEL=text-embedding-3-small    # 1536-dim
EMBED_DIM=1536
OPENAI_API_KEY=sk-...

NEO4J_URI=bolt://localhost:7687
NEO4J_USER=neo4j
NEO4J_PASSWORD=password123

RECALL_TOP_K=30                       # memories returned per query
RECALL_HOPS=2                         # graph-expansion toggle

Switching the embedder changes the vector dimension — the backend drops + recreates the Neo4j vector index on boot, and POST /admin/reembed re-embeds existing memories in place (no re-extraction).

---

Benchmarks

LoCoMo

Replays dated multi-session conversations into the engine, answers each question via the read-only /answer endpoint, and grades with an LLM judge. Matches the mem0/Zep methodology (GPT-4o-mini judge, adversarial category excluded).

cd backend && source venv/bin/activate
python eval/locomo/download_locomo.py                         # once — fetches locomo10.json

# mem0-comparable run (10 conversations, all questions, GPT-4o-mini judge)
python eval/locomo/run_locomo.py \
  --sample 10 --all-q --exclude-adversarial --judge openai --fast-ingest

# cheap iteration against an already-ingested graph (no re-extraction)
python eval/locomo/run_locomo.py --skip-ingest --all-q --judge openai

Key flags:

Flag	Effect
--sample N	number of conversations
--all-q	grade every question (not just the first 20)
--exclude-adversarial	drop category-5 questions (mem0/Zep do)
--judge openai|anthropic	judge provider (default openai / GPT-4o-mini)
--fast-ingest	skip conflict-classify during bulk load (~40% fewer calls, no quality loss for LoCoMo)
--ingest-model	cheaper model for extraction only (e.g. claude-haiku-4-5)
--skip-ingest	reuse the current graph — QA only (free iteration)
--workers N	parallel ingestion workers

Result (152-question conversation, GPT-4o-mini judge):

category	this system	mem0¹	Zep¹
temporal	86%	55.5	49.3
single-hop	76%	67.1	61.7
multi-hop	62%	51.2	41.4
open-domain	69%	72.9	76.6
overall	75%	66.9	66.0

¹ mem0/Zep numbers are from the mem0 paper, averaged over all 10 conversations. This system's number is one 152-question conversation — a strong signal, not a head-to-head win. The full 10-conversation run is the real apples-to-apples comparison and is pending.

Engine unit-evals

eval/run_eval.py drives scripted scenarios (store→recall, supersede, contradiction, duplicate, multi-hop) and asserts graph behavior — covers what LoCoMo doesn't (contradiction handling, emotion linkage).

python eval/run_eval.py                 # all scenarios
python eval/run_eval.py --only contradiction

---

API

Method	Path	Purpose
POST	/chat	reply + recall + store in one turn (used by the UI)
POST	/store	extract & persist memories (timestamp, resolve_conflicts, model optional)
POST	/recall	three-path recall for a query
POST	/answer	read-only recall + answer (no store)
GET	/contradictions	list unresolved flagged contradictions
POST	/contradictions/resolve	pick a winner; loser is superseded
POST	/contradictions/auto-resolve	resolve all (confidence, then recency)
GET	/graph	full graph for the visualizer
GET	/memory/{id}	single memory: props, neighbors, supersede history
POST	/clear	wipe all memories
POST	/admin/reembed	re-embed all memories with the current model (no re-extraction)
GET	/admin/cache-stats	prompt-cache token usage

---

Frontend

• Obsidian-style force graph — nodes sized by degree, colored by type; hover to trace neighbors; recalled memories glow; contradiction edges in red, supersede in amber.
• Chat with a live debug panel (what was recalled / stored per turn).
• Contradiction inbox — resolve flagged conflicts (keep-this / auto-resolve).
• Inspector — click a memory for its metadata, edges, and supersede history.
• Light / dark toggle, persisted.
• Chat commands: /clear-chat (transcript only) · /clear (everything).

---

Known limitations

• Deep multi-hop (3+ links) isn't reliable — recall does one graph-expansion round (depth-2 bridges). Genuinely deep chains would need iterative retrieval / query decomposition. LoCoMo tops out at 2-hop, which the current paths cover.
• Prompt caching is a no-op here — the system prompts are below the model's minimum cacheable prefix. The cost lever is --fast-ingest / a cheaper ingest model / the Batch API.
• Extraction is probabilistic — LLM extraction varies run-to-run, so scores move a few points between re-ingests.

Roadmap

• Full 10-conversation LoCoMo run for a head-to-head number
• Iterative retrieval for deep multi-hop
• LongMemEval (knowledge-update + abstention — grills the conflict logic)
• Reflection / consolidation (cluster memories into higher-level insights)
• WebSocket live graph push · timeline scrubber (bi-temporal) · multi-user

Layout

backend/
  app/
    main.py                 FastAPI app + startup (schema, embedding warmup)
    config.py               env settings
    db/                     Neo4j client + schema (constraints, vector index)
    memory/                 extract · embed · store · recall · conflict · graph · models
    llm/claude.py           Claude client (retry/backoff)
    api/routes.py           endpoints
  eval/
    run_eval.py             engine unit-evals
    locomo/                 LoCoMo harness (download + run)
frontend/
  src/
    components/             GraphView · ChatPanel · Inspector · ContradictionInbox
    lib/api.ts              backend client
docker-compose.yml          Neo4j

License

MIT © Aditya Painuli