Recall

RAG system that remembers. Checks episodic memory before hitting the document store. If a past answer is confident and fresh, it skips retrieval entirely — answering in 266ms instead of 49s.

Core Advantages

Speed. Memory hits bypass the full retrieval+generation pipeline entirely:

Path	Measured latency	What happens
Memory hit	~260ms	Embed → Chroma lookup → return cached answer
Retrieval + generation	~49s (cold), ~6s (warm)	Embed → HyDE expand → doc search → LLM generation via OpenRouter
Reconciled	~5–8s	Both paths run → LLM arbiter picks the better answer

The first query is always slow (cold-starting embedding model + LLM API). The second identical or similar query hits memory — 180x faster. This is the core value: repeated queries are nearly free.

Scope awareness. A cached answer to "list all state capitals" won't get served for "capital of Karnataka." Recall classifies query scope ("single" entity vs. "set" of items) at write time and checks it at query time — scope mismatches bypass the memory shortcut even when cosine similarity is high.

No hallucination on unknowns. When the retrieved documents don't contain the answer, Recall says so instead of making things up. Safety-filter responses from the LLM (e.g. "User Safety: safe") are detected and rejected — never cached in memory.

No infrastructure. Chroma runs in-process as a local persistent store. No Docker, no vector DB service, no external dependencies beyond API keys.

Self-correcting. Source document content is hashed. If the ingested docs change, the memory is flagged stale and re-retrieved. The confidence model (logistic regression trained on eval data) predicts answer quality from similarity, correction history, age, and scope match — not just raw cosine distance.

Observable. Every query returns latency_ms in the JSON response. Every search_memory call logs collection size and query latency. Memory pruning kicks in when the collection exceeds MAX_MEMORY_ENTRIES.

Why

Standard RAG re-fetches everything on every query. That's wasteful when you asked the same thing yesterday. Recall treats retrieval as the expensive fallback, not the default — like how you don't re-read a manual you just read.

Use cases:

• Repeated questions in support bots — answer from memory, not re-retrieval
• Research workflows — ask follow-ups without re-chunking the same PDF
• Internal knowledge bases — employees ask similar questions daily
• Any RAG setup where the same queries surface more than once

How It Works

query -> embed -> search_memory -> [confident?] -> check_staleness -> [fresh?] -> answer_from_memory
                                   \-> [not confident] -> hyde_expand -> retrieve_docs -> [memory also present?] -> reconcile
                                                                                \-> answer_from_retrieval
                         (every path) -> write_memory (skip if junk answer) -> append to session -> END

Every answer gets written to episodic memory with a scope classification ("single" or "set"). Next time the same (or similar) question comes in, it hits memory first — no retrieval needed. When both memory and documents have something to say, the reconcile node picks the better answer and marks the other as corrected.

Junk answer filtering: LLM safety responses ("User Safety: safe"), fallback strings, and error messages are detected and never cached in memory. If a cached memory somehow contains a junk answer, it's evicted at search time and the query falls through to fresh retrieval.

Scope-mismatch handling: If the query scope doesn't match the cached memory's scope (e.g., broad list cached, narrow question asked), the memory confidence is forced below the moderate threshold, so it falls through to retrieval + reconcile. Reconcile treats retrieval as authoritative on scope mismatches — no LLM arbitration needed.

Confidence is scored per-memory using similarity, correction history, age, and scope match via a learned logistic regression model. Staleness is detected by hashing source document content — if the ingested docs change, the memory is flagged stale and re-retrieved.

Stack

Component	Role
LangGraph	Async orchestration
Chroma	Two collections: episodic_memory, documents (local, persistent)
Groq	HyDE expansion + reconcile decisions
OpenRouter	Generation pass (streaming)
Langfuse	Tracing (optional, skipped if keys unset)
FastAPI	Async serving + SSE streaming + web UI + PDF/URL upload
Streamlit	Optional chat UI with progressive streaming display
Sessions	JSON-backed conversation persistence under data/sessions/

Setup

uv venv --python 3.11 .venv
uv pip install -r requirements.txt
cp .env.example .env   # fill in API keys

Chroma data lives under CHROMA_PATH (default ./data/chroma). No Docker.

Model config

In .env, set your generation model:

# Recommended — reliable, no safety filter issues:
OPENROUTER_MODEL=openai/gpt-4o-mini

# Free tier — may trigger safety filter on some queries:
OPENROUTER_MODEL=openrouter/free

The HyDE and reconcile nodes use Groq (GROQ_MODEL=llama-3.3-70b-versatile).

Running Web UI (default)

uv run uvicorn app.api:app --reload --reload-exclude '.venv'

Open http://localhost:8000. Dark-mode chat UI with session management in the sidebar. Ingest text, PDFs, or URLs directly from the UI.

Ingest the demo PDF

uv run python scripts/ingest_pdf_demo.py

Downloads arXiv:2410.12837 (RAG survey, 52K chars, 117 chunks).

Try it — the demo flow

After ingesting the demo PDF, ask these in order:

#	Query	Expected	What to watch
1	"What is Retrieval-Augmented Generation?"	Retrieval path, ~49s	First query — cold start, full pipeline
2	"What is Retrieval-Augmented Generation?"	Memory hit, ~260ms	Exact repeat — same embedding, instant cache
3	"What is RAG?"	Memory hit, ~375ms	Near-duplicate — high similarity to cached memory
4	"What is the capital of France?"	"I don't have information..."	Unrelated — graceful decline, no hallucination
5	"List all state capitals" then "Capital of Karnataka?"	Broad cached, narrow retrieves	Scope mismatch — broad answer can't serve narrow query

Key demo moment: Compare the latency_ms field in the JSON response between query 1 (~49,000ms) and query 2 (~260ms). That's the 180x speedup from memory.

Streamlit (optional)

uv run streamlit run app/ui.py

Open http://localhost:8501. Chat UI streams answers token-by-token via SSE. Shows latency, confidence, similarity, and source badge per message.

Threshold sweep

uv run python eval/thresholds_sweep.py --eval-set eval/eval_set.jsonl

Sweeps confidence threshold against labeled Q/A pairs with an LLM judge.

Memory consolidation

uv run python scripts/consolidate_memory.py

Finds near-duplicate memories (cosine > 0.92) and merges them. When the collection exceeds MAX_MEMORY_ENTRIES (default 5000), low-confidence old entries are pruned automatically.

Latency

Every search_memory call logs collection_size and latency_ms via Python logging. Every API response includes latency_ms in the JSON payload.

Measured on a local machine:

Collection size	Memory search latency
100 memories	~5ms
1,000 memories	~15ms
10,000 memories	~40ms

The retrieval path bottleneck is LLM API calls (HyDE + generation), not Chroma search. Memory hits avoid both entirely.

API

Method	Path	Description
GET	/	Web UI
POST	/query	Ask a question (blocking, returns full response with latency_ms)
POST	/query/stream	Ask a question (SSE streaming, token-by-token)
POST	/docs	Ingest plain text
POST	/docs/pdf	Ingest a PDF
POST	/docs/url	Ingest a URL
GET	/docs	List all ingested source documents
DELETE	/docs/{doc_id}	Delete a source and all its chunks
GET	/health	Health check + Chroma counts
GET	/sessions	List sessions
POST	/sessions	Create session
GET	/sessions/{id}	Session detail + messages
PATCH	/sessions/{id}	Rename session
DELETE	/sessions/{id}	Delete session

Response format

{
  "answer": "Retrieval-Augmented Generation (RAG) is a hybrid architecture...",
  "source": "memory",
  "memory_confidence": 0.999,
  "memory_similarity": 0.999,
  "is_stale": false,
  "reconciled": false,
  "latency_ms": 266.0,
  "session_id": "..."
}

Streaming format

POST /query/stream returns SSE with two event types:

data: {"type": "token", "content": "Retrieval"}
data: {"type": "token", "content": "-Augmented"}
data: {"type": "token", "content": " Generation"}
...
data: {"type": "done", "source": "memory", "memory_confidence": 0.999, "memory_similarity": 0.999, "is_stale": false, "reconciled": false, "latency_ms": 266.0}

Tests

uv run pytest -q