A RAG system over FAA regulatory documents for the Cessna 172, built to answer one question: how much does each advanced retrieval technique actually buy you, and where does it break?
Live demo · Every layer is measured against a naive baseline on a hand built test set. Full methodology, per stage results, and failure analysis: ABLATION.md.
Retrieval, full recall@5 on 20 in scope questions (every expected document must be retrieved, not just one):
| Stage | Global | single_hop | cfr_only | multi_hop |
|---|---|---|---|---|
| Naive dense | 70% | 87.5% | 83.3% | 33.3% |
| + Hybrid BM25 + dense | 70% | 87.5% | 100% | 16.7% |
| + Query decomposition | 75% | 100% | 100% | 16.7% |
| + Cross encoder reranking | 75% | 100% | 100% | 16.7% |
| + Deterministic identifier index | 90% | 100% | 100% | 66.7% |
| + Agentic follow up | 95% | 100% | 100% | 83.3% |
Recall is scored at document level, so these figures are an upper bound: retrieving any chunk of an expected document counts as a hit even when the passage carrying the answer was missed. See ABLATION.md.
Generation: 5/5 correct refusals on out of scope questions, 0 hallucinations in the final pipeline (2 in the baseline).
Four findings worth the read:
- Hybrid retrieval regressed multi hop performance, and cross encoder reranking was net neutral. Both are standard recommendations.
- On one question the naive baseline beats the final pipeline outright, and for a structural reason rather than by chance.
- The largest single gain came from a deterministic symbolic lookup, not from a better model or embedding.
- The most dangerous model behaviour found was not a hallucinated fact but a confidently asserted absence of a regulatory exception.
jeremymaille.github.io/rag-aero
Pick any of the 25 test set questions and watch the pipeline run stage by stage: query decomposition, deterministic identifier matches, retrieved excerpts, and the agentic follow up when the retrieved text points at a section nobody asked for. Each run ends with the final answer, the naive baseline answer next to it, the ground truth, and the source excerpts the model was actually given.
Free form input is deliberately disabled. The demo replays the evaluated test set, so every answer shown has a ground truth to check it against and matches the numbers in the table above. It is a static page: no server, no API call, no model loaded. The frozen index hash is printed at the bottom of the page.
Public domain FAA material only:
- 20 Airworthiness Directives for the Cessna 172 (Federal Register API)
- 14 CFR Part 43 and the maintenance sections of Part 91 (eCFR API)
- Type Certificate Data Sheet 3A12
The Pilot's Operating Handbook is deliberately excluded: it is Cessna/Textron proprietary.
question
│
├─ query decomposition (LLM, cached) one sub question per document needed
│
└─ for each sub question:
├─ deterministic identifier lookup exact AD number / CFR section match
├─ hybrid retrieval BM25 + dense, fused on rank (RRF)
├─ cross encoder reranking ms-marco-MiniLM-L-6-v2
└─ agentic follow up (LLM, cached) fetches CFR sections cited inside retrieved text
│
└─ generation Mistral, grounded + cited, refuses when uncovered
Stack: sentence-transformers (bge-base-en-v1.5), Qdrant (local), rank-bm25, Mistral API. No RAG framework: every stage is orchestrated directly so it can be measured in isolation.
src/
ingest/ corpus download and filtering
parse/ PDF and CFR XML extraction
chunk/ chunking strategy
index/ embedding and vector store
retrieve/ one module per ablation stage, each with its eval script
generate/ answer generation (baseline and final)
evaluate/ test set annotation and retrieval scoring
tools/ corpus inspection, and demo asset generation
eval/ test set, LLM caches, and all measured results
demo_assets/ frozen chunks, traces, and index manifest
docs/ the static demo served by GitHub Pages
eval/results_*.json are committed: the numbers in the ablation table are reproducible without re-running any paid API call.
Requires uv and a Mistral API key.
git clone <repo> && cd rag-aero
echo "MISTRAL_API_KEY=your_key" > .envRebuild the corpus (not committed, roughly 600 chunks):
uv run python src/ingest/download_data.py
uv run python src/ingest/filter_corpus.py
uv run python src/parse/parse_pdfs.py
uv run python src/parse/parse_cfr.py
uv run python src/ingest/filter_cfr.py
uv run python src/chunk/chunk_corpus.py
uv run python src/index/build_index.pyThe TCDS 3A12 PDF is fetched manually into data/raw/tcds/ (single file, no stable API endpoint).
Reproduce the ablation:
uv run python src/evaluate/eval_retrieval.py # stage 1
uv run python src/retrieve/eval_hybrid_retrieval.py # stage 2
uv run python src/retrieve/eval_decomposed_retrieval.py # stage 3
uv run python src/retrieve/eval_reranked_retrieval.py # stages 4 and 5
uv run python src/retrieve/eval_agentic_retrieval.py # stage 6Generation:
uv run python src/generate/generate_naive.py
uv run python src/generate/generate_agentic.pyBoth LLM steps are cached per question in eval/, so repeated runs are byte identical. Delete the cache files to regenerate.
The demo is a frozen replay, so it does not follow the eval automatically. After changing anything that moves the numbers, regenerate it:
uv run python src/tools/export_demo_assets.py # freeze chunks and index manifest
uv run python src/tools/precompute_demo_traces.py # freeze the pipeline traces
uv run python src/tools/build_docs_data.py # bundle everything into docs/data.jsonapp.py is a local Gradio version of the same replay, kept for development. The published demo is the static page in docs/.
q18, the one remaining retrieval failure, and the one case where the baseline wins. The question needs 14 CFR 43.7. The naive dense baseline retrieves it on semantic similarity alone and answers correctly; the final pipeline does not. The cross encoder discarded the lone CFR chunk from an AD dominated pool, and neither the identifier index nor the agentic follow up could recover it, since both key on an identifier written in the text and no AD in the corpus cites 43.7. Every stage added after dense retrieval optimises for symbolic matching, and q18 is the only question whose document link is purely semantic. Documented rather than patched.
Generation failures. Two answers refuse despite the correct source being retrieved and present in the context, and two more are correct but incomplete because the passage carrying the detail was not in the selected chunks. Both classes are detailed in ABLATION.md.