citeseek

A RAG service that treats retrieval quality as a measurable engineering property, not a vibe. Every answer carries citations that are validated against the retrieved context, and every change to the pipeline is judged by an evaluation harness that runs in CI — if Recall@5 or citation coverage drops below the gate, the build fails.

When RAG systems fail, retrieval is the culprit far more often than generation. citeseek is built around that fact: retrieval has its own metrics (Recall@K, MRR), separate from answer quality, and the hybrid retriever — vector search and full-text search fused by Reciprocal Rank Fusion — is the part the eval harness scrutinizes hardest.

How it works

flowchart LR
    Q[question] --> E[embed]
    E --> V[pgvector HNSW<br/>cosine top-k]
    Q --> T[Postgres FTS<br/>ts_rank top-k]
    V --> F[RRF fusion]
    T --> F
    F --> R[lexical rerank]
    R --> P[prompt with<br/>numbered passages]
    P --> L[LLM provider<br/>anthropic / openai / ollama / mock]
    L --> A[answer]
    A --> C[citation validation<br/>coverage + invalid markers]

Loading

• Storage: one Postgres table holds both a generated tsvector (GIN index) and a pgvector embedding (HNSW index) — hybrid retrieval is two indexed queries against the same rows, no second datastore.
• Fusion: RRF combines rankings, not scores, so cosine similarity and ts_rank never need to be normalized against each other.
• Providers: embeddings and LLMs sit behind small interfaces. The mock embedder is a hashed bag-of-words projection — deterministic and lexically meaningful, so the entire test suite, CI pipeline and eval baseline run offline with zero API keys. Swap in Anthropic/OpenAI/Ollama with two env vars.
• Citations: the service checks every [n] marker against the passages actually provided, reports sentence-level citation coverage, and flags hallucinated markers.
• Observability: Prometheus metrics for request latency, retrieval latency and token spend per model; structured logs.

Evaluation

A labeled dataset of 26 questions over a bundled 14-document corpus (a synthetic ops handbook — see corpus/README.md) drives the harness:

uv run python -m citeseek.evaluate --dataset eval/dataset.json --gate

Current baseline with fully offline providers (embed=mock, llm=mock):

Metric	Value	Gate
Recall@5 (retrieval)	1.000	≥ 0.85
MRR (retrieval)	1.000	≥ 0.70
Keyword hit rate (answers)	0.923	≥ 0.80
Citation coverage	1.000	≥ 0.90
Invalid citation rate	0.000	≤ 0.05

CI runs this gate on every push. The corpus is deliberately built with one unambiguous source document per fact, so retrieval metrics are exact rather than fuzzy — perfect retrieval on it is the floor a change must preserve, not a brag. The extractive mock answers 92% of questions; the misses are questions whose phrasing shares no tokens with the source sentence — exactly the gap a real LLM provider closes.

Getting started

git clone https://github.com/aminyx/citeseek
cd citeseek
docker compose up --build

# index the bundled corpus
curl -X POST localhost:8000/ingest -H 'content-type: application/json' -d '{"subdir":"meridian"}'

# ask, with citations
curl -X POST localhost:8000/query -H 'content-type: application/json' \
     -d '{"question":"how long are refresh tokens valid?"}'

# stream the answer (SSE)
curl -N -X POST localhost:8000/query/stream -H 'content-type: application/json' \
     -d '{"question":"what happens after the 6th failed job attempt?"}'

Local development (needs uv and Docker for Postgres):

docker run -d --name citeseek-pg -p 5433:5432 \
  -e POSTGRES_USER=citeseek -e POSTGRES_PASSWORD=citeseek -e POSTGRES_DB=citeseek \
  pgvector/pgvector:pg17
uv sync --all-groups
uv run uvicorn citeseek.main:app --reload

Using real models

CITESEEK_LLM_PROVIDER=anthropic CITESEEK_ANTHROPIC_API_KEY=sk-... docker compose up
# or fully local:
CITESEEK_EMBED_PROVIDER=ollama CITESEEK_EMBED_MODEL=bge-m3 \
CITESEEK_LLM_PROVIDER=ollama CITESEEK_LLM_MODEL=qwen3.5 docker compose up

All variables are documented in .env.example. Note: changing the embedding provider changes vector dimensions — re-ingest after switching.

API

Endpoint	Method	Description
/query	POST	Answer with citations, token usage and stage latencies
/query/stream	POST	SSE stream of the answer
/ingest	POST	Index a corpus subdirectory (restricted to corpus/)
/stats	GET	Document and chunk counts
/healthz	GET	Liveness incl. database
/metrics	GET	Prometheus metrics

Testing

uv run pytest                          # unit tests, no infrastructure needed
CITESEEK_TEST_DATABASE_URL=postgresql://citeseek:citeseek@127.0.0.1:5433/citeseek \
  uv run pytest                        # + integration tests against real pgvector

Limitations & trade-offs

• The bundled corpus is small and synthetic by design — it makes eval labels exact, but absolute metric values do not transfer to messy real-world corpora; the harness's value is catching regressions.
• The lexical reranker is a placeholder for a cross-encoder: it fixes cheap misses and defines the interface, but it cannot reorder by semantic relevance the way a real reranker does.
• Chunking measures words, not model tokens; long-token languages will produce oversized chunks.
• No auth on the API — deploy behind a gateway (see threshold).
• /query/stream skips citation validation — validated citations currently require the non-streaming path.

Future improvements

• Cross-encoder reranker provider (BGE-reranker via Ollama)
• Citation validation for the streaming path (post-stream trailer event)
• Corpus versioning and eval history tracking across commits
• Batch ingestion API with progress reporting

License

MIT