A provider-agnostic RAG evaluation framework benchmarked on HotpotQA
Plug in any OpenAI-compatible LLM and run a rigorous, reproducible evaluation of your RAG pipeline —
13 metrics · multi-prompt comparison · failure diagnosis · cost tracking · auto-generated audit report
Status: Independent personal research project
Most RAG evaluation toolkits answer one question: "Is the answer correct?"
This framework answers five — and generates an audit-ready report that tells you why:
| Question | How |
|---|---|
| Is the answer correct? | F1, exact match, ROUGE-L |
| Is it grounded in retrieved documents? | MiniMax embedding similarity |
| Does it cover the key information? | 2-tier completeness cascade |
| Does it faithfully represent the context? | RAGAS LLM-as-judge |
| What does it cost to run and monitor? | Generation vs. evaluation cost separation |
Novel contributions: The completeness cascade and the generation/evaluation cost separation are purpose-built for production RAG monitoring — not found in standard evaluation toolkits.
┌─────────────────────────────────────────────────────────────────────────────┐
│ RAG Evaluation Pipeline │
├──────────┬────────────────┬─────────────────────────┬───────────────────────┤
│ DATA │ RETRIEVAL │ GENERATION │ EVALUATION │
│ │ │ │ │
│ HotpotQA │ Hybrid BM25 + │ 4 prompt variants │ 13 metrics │
│ multi-hop│ dense (60/40) │ temp=0.2 top-k=20 │ QualityGuard gate │
│ benchmark│ top-k=20 │ baseline / concise / │ Failure investigator │
│ │ │ detailed / citation │ Cost tracker │
│ │ │ │ Audit report │
└──────────┴────────────────┴─────────────────────────┴───────────────────────┘
Generated from a 100-question HotpotQA run (text-embedding-3-small, RAGAS LLM-as-judge). All charts produced automatically by the framework.
Accuracy by variant, quality-metric heatmap, faithfulness distribution, and refusal/citation rates — at a glance across all four prompt strategies.
Generation vs. evaluation cost separation (left), the ~20% Tier-2 LLM trigger rate of the completeness cascade (centre), and how the cascade lifts embedding-only scores (right).
Hit-rate distribution against the 70% target, and the correlation between retrieval hit rate and answer F1 per variant.
A full auto-generated audit report from the same run is included in the repo:
- View the Markdown report — renders directly on GitHub
- Download the PDF report — print-ready, ~1.5 MB
The report walks through all 12 sections: executive summary, evaluation scope, metric definitions, per-category results with observation tables, failure diagnosis, cost analysis, and reproduction steps.
Metrics are grouped into three categories that map directly to the report sections:
| # | Metric | Method | Cost |
|---|---|---|---|
| 1 | Exact Match | Token normalization | Free |
| 2 | F1 Score | Bag-of-words overlap + partial match bonus | Free |
| 3 | Contains Match | Gold answer substring in prediction | Free |
| 4 | ROUGE-L | Longest common subsequence | Free |
| # | Metric | Method | Cost |
|---|---|---|---|
| 5 | Answer Relevance | cosine_sim(answer, question) | Embedding |
| 6 | Completeness | 2-tier cascade — see below | Embedding + conditional LLM |
| 7 | Conciseness | Length penalty vs. question length | Free |
| 8 | Refusal Detection | Phrase-pattern regex | Free |
| 9 | Quality Score | Weighted composite (configurable weights) | — |
| # | Metric | Method | Cost |
|---|---|---|---|
| 10 | Context Relevance | cosine_sim(context, question) | Embedding |
| 11 | Groundedness | Embedding: avg_i max_j sim(ans_sent_i, ctx_sent_j) — sentence-level overlap | Embedding |
| 12 | RAGAS Faithfulness | LLM: extract atomic claims → verify each claim is supported by context (YES/NO) | LLM †† |
| 13 | SNR | Ratio of answer words found in retrieved context (word-level overlap) | Free ‡ |
Groundedness vs. Faithfulness — both measure answer-to-context attribution, but with different methods. Groundedness uses embedding similarity (fast, handles paraphrase, but can be fooled by vocabulary overlap without factual agreement). Faithfulness uses an LLM to verify each factual claim individually (more expensive, but catches hallucinations that happen to share vocabulary with the context). Running both catches failure modes that either metric alone would miss. Neither measures retrieval quality — that is Context Relevance (metric 10).
† Embedding cost — one API call to your embedding model per metric that uses it.
‡ SNR (Signal-to-Noise Ratio) — a lightweight lexical heuristic:SNR = (answer words that appear in retrieved context) ÷ (total answer words). A score of 0.70 means 70% of the answer's words are directly traceable to the retrieved documents. Low SNR flags answers padded with meta-commentary ("I cannot find...", "Based on the above...") or containing words that never appear in the context — a cheap first-pass hallucination signal. It complements Groundedness (which uses embedding similarity and handles paraphrasing) with a simpler, interpretable word-overlap check. Because it is pure arithmetic on the tokenised text, it costs nothing to compute — no API calls, no model inference.
†† LLM cost (RAGAS Faithfulness) — makes 2+ LLM calls per answer:
Call 1 (extraction): the LLM reads the answer and breaks it into atomic claims — single, indivisible factual statements (e.g. "The Eiffel Tower is in Paris" rather than "The Eiffel Tower is in Paris and was built in 1889").
Call 2…N (verification): for each extracted claim, a separate LLM call asks whether the retrieved context supports that claim (YES/NO). The final score = verified claims ÷ total claims.
This measures whether every factual statement in the answer is actually grounded in the retrieved documents — catching hallucinations that embedding similarity alone would miss. Refusal answers are excluded (scored asNone, not penalised). The metric runs on every answer and is the main cost driver in the evaluation framework.
Completeness is the hardest metric to compute reliably. A single embedding score produces systematic false positives when answers are short or share vocabulary with context without factual agreement. The cascade fixes this with a trigger-on-demand design:
Every answer
│
▼
Tier 1 — Question-Filtered Context Coverage (embedding, always runs, free-ish)
│ Filter context sentences by cosine similarity to the question
│ Score = avg_i max_j sim(relevant_ctx_i, answer_sent_j)
│
├── Trigger B: |completeness_T1 − groundedness| > 0.3 (cross-metric disagreement)
│ : |completeness_T1 − faithfulness | > 0.3
│
│ completeness_T1 = Tier 1 embedding score computed just above.
│ groundedness and faithfulness are independent embedding / LLM scores
│ that measure answer-to-context alignment from different angles.
│
│ Rationale: all three metrics should roughly agree when the embedding
│ completeness score is reliable. A gap > 0.3 (30 pp) between completeness
│ and either peer metric is a signal that the embedding score is an outlier —
│ likely a false positive caused by vocabulary overlap without factual
│ agreement, or a false negative on paraphrased content.
│
│ The 0.3 threshold is a calibrated hyperparameter, not formally derived.
│ It was set empirically to balance two risks:
│ • too low → triggers fire on normal score variance, wasting LLM calls
│ • too high → genuine embedding failures slip through unchecked
│ It is exposed as `cross_metric_delta` in the code and can be tuned.
│
├── Trigger C: answer < N words AND ≥ M relevant context sentences (short answer)
│
└── No trigger → use Tier 1 score (zero extra cost)
Trigger fires →
▼
Tier 2 — LLM-as-judge (conditional, ~20% of queries)
Structured prompt: score 0.0–1.0 + one-sentence reason
max_tokens=80 · full audit trail per case
In practice: Tier 2 fires on ~20% of answers, keeping per-query LLM overhead at ~$0.000004.
Every question is evaluated across four prompt strategies simultaneously:
| Variant | Design intent | Expected F1 | Best for |
|---|---|---|---|
baseline |
Minimal instruction — natural LLM behaviour | 0.40–0.60 | Benchmarking defaults |
concise |
Maximum precision, minimal tokens | 0.70–0.80 | APIs, cost-sensitive apps |
detailed |
Comprehensive, self-contained answers | 0.60–0.75 | Support, education |
citation |
Attribution-enforced, compliance-ready | 0.70–0.80 | Legal, medical, finance |
After evaluation, the framework generates a structured Markdown report in reports/:
reports/multi_prompt_eval_results_500_report.md
│
├── Executive Summary — LLM-consolidated cross-section findings with causal chains
├── § 2 Evaluation Scope — System architecture, test data, coverage, regulatory alignment
├── § 3 Evaluation Approach — Metrics, sampling, completeness cascade, LLM judge policy
├── § 4 Correctness — F1/EM/ROUGE across variants + observations table
├── § 5 Answer Quality — Completeness, conciseness, refusals + observations table
├── § 6 Attribution — Groundedness, faithfulness, SNR + observations table
├── § 7 Retrieval Quality — Context relevance, hit rates + observations table
├── § 8 Cost Analysis — Generation vs. evaluation cost breakdown
├── § 9 Failure Diagnosis — 16 patterns + k-means clustering of unclassified failures
├── §10 Cross-Section Analysis
├── §11 Conclusions & Recommendations
└── §12 Reproduction Steps
Each LLM-generated section carries an AI disclosure notice and is clearly separated from deterministic (audit-safe) observations. The report is designed to be submitted to model risk management review with SR 11-7 / SR 26-02 and NIST AI RMF cross-references in §2.4.
Low-quality answers are automatically classified by a 3-layer investigator:
Layer 1 — 16 rule-based patterns (runs on all flagged answers):
| Pattern | Metric signature | Diagnosis |
|---|---|---|
| P1 | High F1 + Low Groundedness | Hallucinated details |
| P2 | Low F1 + High Groundedness | Verbose / off-target |
| P4 | Low Context Rel + Low Groundedness | Retrieval failure |
| P5 | High Context Rel + Low Answer Rel | Generation failure |
| P7 | Low F1 + Low Groundedness + Low Faithfulness | Hallucination (non-refusal) |
| P9 | Refusal + Low Context Rel | Retrieval-driven refusal |
| P10 | Refusal + High Context Rel | Over-conservative model |
| P13 | Cascade escalated + still low | Cascade cannot rescue |
| … | … | … |
Layer 2 — k-means clustering (UNK cases only): groups unclassified failures by metric similarity; cluster centroids surface as candidates for new rule-based patterns.
Layer 3 — Human review exports: cluster representatives exported as Markdown for reviewer annotation.
The framework tracks two cost layers independently so you can answer: "How much does my RAG pipeline cost in production, and how much does this monitoring add?"
GENERATION COST — what your RAG system costs in production
LLM answer generation (prompt + context + answer tokens)
EVALUATION COST — what this monitoring framework adds
Embedding : retrieval query, completeness Tier 1, groundedness
LLM judge : RAGAS faithfulness, completeness Tier 2 (~20% of queries)
Sample output (500 questions × 4 prompt variants = 2,000 answers):
GENERATION COSTS (production):
llm_answer_generation $0.2100 ($0.000105/q)
EVALUATION COSTS (framework):
embedding_retrieval $0.0002 ($0.000000/q)
embedding_completeness $0.0008 ($0.000000/q)
llm_judge_faithfulness $0.0420 ($0.000021/q)
llm_judge_completeness $0.0084 ($0.000004/q) ← only ~20% of queries
─────────────────────────────────────────────────────
Evaluation overhead 16.1% of total
Temperature 0.2, not 1.0 — HotpotQA is a factoid benchmark. Deterministic generation produces measurably higher F1 and faithfulness. Temperature 1.0 was the single largest source of score variance in baseline experiments.
How hybrid retrieval works — BM25 + dense embeddings, 60/40 weighted
Every query goes through two independent ranking passes over the full document corpus:
| Pass | Method | What it measures |
|---|---|---|
| Dense (semantic) | Cosine similarity between query embedding and document embeddings | Meaning and intent — catches paraphrase, synonyms |
| Sparse (keyword) | BM25 score — term frequency weighted by inverse document frequency | Exact word matches — names, dates, locations, identifiers |
BM25 (Best Match 25, Okapi variant) is a classical information-retrieval algorithm. It scores each document by how often the query's words appear in it, discounted by how common those words are across the entire corpus. A word like "the" barely moves the score; a rare word like "Okapi" moves it a lot. Unlike embeddings, BM25 is entirely arithmetic — no neural network, no API call, computed in milliseconds.
After both passes, scores are min-max normalised independently (so neither pass dominates due to scale differences), then combined into a single score: 0.6 × semantic + 0.4 × BM25. The top-K documents from this single merged ranking are passed to the LLM as context.
The 60/40 split was tuned empirically on the HotpotQA dev set — semantic search handles the majority of cases (paraphrased questions, indirect references) while BM25 acts as a safety net for exact entity matches that embeddings sometimes miss.
Top-K 20, not 5 — Top-K is the number of documents taken from the final merged ranking to assemble the LLM's context window. HotpotQA is a multi-hop benchmark: most questions require evidence from 2 separate gold documents. With K=5, one of the two gold documents was frequently absent from the context, making the question unanswerable regardless of generation quality. K=20 gives sufficient coverage to reliably include both gold documents, at the cost of a larger (but bounded) context passed to the LLM.
Completeness cascade, not a single score — A single embedding completeness score fails silently for short answers and when context shares vocabulary without factual agreement. The cascade triggers LLM verification only when evidence suggests the embedding score is unreliable.
QualityGuard runs before cost tracking — The real-time gate catches failures before they skew aggregate metrics. Answers that pass F1 by coincidence (parametric LLM knowledge, not retrieval) are still flagged as ungrounded.
- LLM-as-judge bias: RAGAS faithfulness and completeness Tier 2 use the same model that generated the answer. Self-evaluation biases toward higher scores on confident-but-wrong answers. A separate judge model is preferable when budget allows.
- HotpotQA scope: Optimised for extractive/factoid QA. For generative tasks (summarisation, creative writing), F1 and exact match are not meaningful — faithfulness and completeness remain applicable.
- Local embedding quality:
EMBEDDING_CHOICE=local(all-mpnet-base-v2, 768 dims) produces ~10% lower groundedness and completeness scores than API embeddings. Scores are not directly comparable across embedding modes. - BM25 tokenisation: BM25 does not handle morphological variation. For non-English corpora, replace the whitespace tokeniser with a language-appropriate tokeniser.
- Synchronous execution: The evaluation loop is single-threaded. For large corpora (>10k questions), parallelise at the process level, not within the notebook.
Azure OpenAI with IP allowlisting: Colab runs on Google Cloud — its IPs are typically not on corporate allowlists. Use Option B (local) for Azure deployments behind a firewall.
Click the Open in Colab badge at the top of this README. Then:
Step 1 — Get the HotpotQA data
Download hotpot_train_v1.1.json from hotpotqa.github.io (~540 MB) and upload it to your Google Drive:
My Drive/data/hotpot_train_v1.1.json
Step 2 — Add Colab Secrets (🔑 key icon in left sidebar → "Add new secret")
| Secret name | Example value |
|---|---|
OPENAI_API_KEY |
sk-... |
OPENAI_BASE_URL |
https://api.openai.com/v1 |
OPENAI_API_VERSION |
2025-04-01-preview (Azure only — leave blank for OpenAI) |
OPENAI_GENERATION_MODEL |
gpt-4o |
OPENAI_EMBEDDING_MODEL |
text-embedding-3-small |
HOTPOTQA_DATA_PATH |
/content/drive/MyDrive/data/hotpot_train_v1.1.json |
Secrets are stored once in your Colab account and never appear in notebook output.
Step 3 — Run all cells in order
The notebook is checkpointed — safe to interrupt and resume.
git clone https://github.com/minw0607/rag_eval_framework.git
cd rag_eval_framework
pip install -r requirements.txt
python -m nltk.downloader punkt wordnetcp .env.example .env
# Edit .env — uncomment the section for your provider and fill in credentialsThe provider is auto-detected from OPENAI_API_VERSION:
- Set (e.g.
2025-04-01-preview) → Azure OpenAI (AzureOpenAIclient) - Blank → OpenAI direct, Ollama, Groq, or any compatible endpoint (
OpenAIclient)
See docs/provider-setup.md for step-by-step instructions for each provider.
mkdir -p data
# Download hotpot_train_v1.1.json (~540 MB) from https://hotpotqa.github.io/
# Move it to data/hotpot_train_v1.1.json
# Set HOTPOTQA_DATA_PATH=./data/hotpot_train_v1.1.json in your .envWhy isn't the dataset in the repo? At 540 MB it exceeds GitHub's 100 MB per-file limit. Download directly from the official source above.
Recommended — build your own:
The notebook builds the vector store automatically on first run and caches it in checkpoints/. Subsequent runs load from cache instantly. This is the right path if you plan to run your own evaluation.
Quick demo — download a pre-built checkpoint (~14 MB):
If you just want to explore the notebook output without waiting for a build, a pre-built checkpoint (999 documents, matches the sample run in reports/) is hosted on Google Drive:
mkdir -p checkpoints
# Download the checkpoint:
# https://drive.google.com/uc?export=download&id=10Ii-jOFefqfOe6AVBjcTL_HotYp6H4ge
# Save it as: checkpoints/vs_999_docs_small.pklThen in your .env (already the default):
FORCE_REBUILD_VECTOR_STORE=False
The notebook will auto-detect and load it — no build step needed.
Note: The demo checkpoint covers ~999 HotpotQA documents. For a full evaluation run, build your own vector store from the complete dataset.
jupyter notebook notebooks/rag_eval_hotpotqa_demo.ipynbRun cells in order. The notebook is checkpointed — safe to interrupt and resume.
llm-eval-framework/
├── README.md
├── requirements.txt
├── .env.example ← Copy to .env and fill in credentials (never committed)
├── .gitignore
├── LICENSE
│
├── notebooks/
│ ├── rag_eval_hotpotqa_demo.ipynb ← ★ Start here — full pipeline, interactive
│ ├── rag_eval_hotpotqa_full.ipynb ← Production run (500+ questions)
│ └── rag_eval_hotpotqa_concise.ipynb ← Compact version, no markdown narrative
│
├── src/ ← Importable Python modules
│ ├── config.py ← All configuration, reads from .env
│ ├── data_loader.py ← HotpotQA loader, question-centric sampling
│ ├── vector_store.py ← BM25 + dense hybrid retrieval
│ ├── rag_system.py ← Answer generation with 4 prompt variants
│ ├── metrics.py ← 13 evaluation metrics (EvalMetrics class)
│ ├── quality_guard.py ← Real-time embedding-based quality gate
│ ├── evaluation_runner.py ← Orchestrates evaluation loop
│ ├── cost_tracker.py ← Generation vs. evaluation cost separation
│ ├── investigator.py ← Failure diagnosis (16 patterns + k-means)
│ ├── visualizer.py ← Dashboard, retrieval, and cascade charts
│ ├── report_generator.py ← Auto-generates structured audit report
│ └── preflight.py ← Dependency and credential checks
│
├── docs/
│ └── provider-setup.md ← Step-by-step setup for Azure, OpenAI, Ollama, Groq
│
├── assets/ ← Sample visualizations shown in this README
│
├── reports/ ← Audit reports (sample run committed; fresh runs here)
│ ├── multi_prompt_eval_results_100_report.md ← ★ Sample report (rendered above)
│ └── multi_prompt_eval_results_100_report.pdf ← ★ Print-ready PDF
│
├── outputs/ ← Evaluation outputs (gitignored)
│ ├── multi_prompt_eval_results_*.json ← Raw results
│ ├── evaluation_dashboard*.png
│ ├── retrieval_analysis*.png
│ ├── cost_cascade_analysis*.png
│ └── low_quality_analysis*/ ← Failure diagnosis exports
│
├── checkpoints/ ← Vector store checkpoints (gitignored)
│ └── vs_*.pkl ← Auto-loaded on restart; set FORCE_REBUILD=False
│
└── data/ ← Dataset files (gitignored — download separately)
└── hotpot_train_v1.1.json
The framework auto-detects your provider from .env — no code changes required.
OPENAI_API_VERSION |
Provider | SDK client |
|---|---|---|
Set (e.g. 2025-04-01-preview) |
Azure OpenAI | openai.AzureOpenAI |
| Blank | OpenAI / Ollama / Groq / etc. | openai.OpenAI |
Supported providers:
| Provider | OPENAI_BASE_URL |
Notes |
|---|---|---|
| Azure OpenAI | https://<resource>.openai.azure.com |
Set OPENAI_API_VERSION |
| OpenAI (direct) | https://api.openai.com/v1 |
Default |
| Ollama (local) | http://localhost:11434/v1 |
Set EMBEDDING_CHOICE=local |
| Groq | https://api.groq.com/openai/v1 |
— |
| Together AI | https://api.together.xyz/v1 |
— |
| LM Studio | http://localhost:1234/v1 |
— |
See docs/provider-setup.md for step-by-step setup, Azure vs. OpenAI differences, and troubleshooting.
The evaluation design maps directly to model risk management frameworks:
| Framework | Requirement | How This Framework Addresses It |
|---|---|---|
| SR 26-02 §4.2 | Quantitative performance benchmarking | 13 metrics across correctness, quality, attribution |
| SR 26-02 §5.1 | Outcome monitoring and bias testing | QualityGuard gate + failure diagnosis |
| NIST AI RMF MEASURE 2.5 | Ongoing AI output monitoring | Real-time quality gate with per-question audit trail |
| NIST AI RMF GOVERN 1.7 | Transparency and explainability | AI disclosure on every LLM-generated block |
This repository is an independent personal project created outside of my employment using my own time and equipment.
Unless explicitly stated otherwise, the code, notebooks, demonstrations, analyses, and documentation in this repository are developed independently from publicly available research papers, technical documentation, regulations, and other public sources. They do not rely on, incorporate, or disclose any confidential, proprietary, non-public, or client information obtained through my employment or professional engagements.
The views, designs, implementations, and conclusions expressed in this repository are solely my own and do not represent the views of any employer, client, or affiliated organization.
This repository is provided for research and educational purposes only.
If you use this framework in your work, please cite the HotpotQA benchmark:
@inproceedings{yang2018hotpotqa,
title={HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering},
author={Yang, Zhilin and others},
booktitle={EMNLP},
year={2018}
}And the RAGAS faithfulness metric:
@article{es2023ragas,
title={RAGAS: Automated Evaluation of Retrieval Augmented Generation},
author={Es, Shahul and others},
journal={arXiv:2309.15217},
year={2023}
}Made with ❤️ for rigorous, reproducible RAG evaluation
Open an issue · Provider setup guide