🧠 Synapse Lab

Three ways to answer a question with an LLM — measured, not guessed.

LLM-only · Vector RAG · GraphRAG benchmarked head-to-head on the same corpus.

Built for the GraphRAG Inference Hackathon by TigerGraph

Quickstart · Architecture · Dashboard · Results · Repo layout

💡 The idea

LLMs burn tokens re-reading context they don't need. Vector RAG helps, but it retrieves chunks that look similar to the question — it has no idea how entities in your data actually relate to each other. Ask a multi-hop question ("who did X work with before joining Y") and it's often stitching together lucky guesses.

GraphRAG builds an explicit knowledge graph from the corpus and walks relationships directly. Fewer irrelevant tokens in the prompt, better grounding on questions that span multiple entities.

Synapse Lab builds all three approaches on the same news corpus and benchmarks them side by side — token cost, latency, and answer accuracy — so the difference is a number, not a vibe.

	LLM-only	Basic RAG	GraphRAG
Retrieval	❌ none	✅ vector search	✅ graph traversal
Grounded in source data	❌	✅	✅
Handles multi-hop questions	❌	⚠️ hit or miss	✅
Relative token cost	lowest	medium	lower than Basic RAG on multi-hop

---

🏗 Architecture

Stage	What happens
Dataset	CNN/DailyMail news corpus, 2M+ tokens, sentence-aware chunking
Pipeline 1 — LLM-only	Raw prompt straight to Gemini, no retrieval
Pipeline 2 — Basic RAG	Chunk → embed → Chroma vector store → top-k retrieval → prompt
Pipeline 3 — GraphRAG	Entity/relation extraction → TigerGraph knowledge graph → multi-hop GSQL traversal → prompt
Benchmark harness	Same query set, all three pipelines, logs tokens / latency / cost
Accuracy eval	BERTScore (semantic similarity) + LLM-as-judge (PASS / FAIL)
Dashboard	Next.js + Recharts, side-by-side comparison, per-query drill-down

---

📊 Dashboard

Side-by-side token/latency/accuracy comparison across all three pipelines, plus an expandable per-query view showing each pipeline's actual answer.

---

⚡ Quickstart

git clone https://github.com/DishantBhere/Synapse-Lab.git
cd Synapse-Lab
pip install -r requirements.txt
cp .env.example .env          # add your GEMINI_API_KEY

Spin up TigerGraph (local, Docker):

docker compose up -d

Full schema + GSQL install walkthrough → docs/tigergraph_setup.md

Build the dataset, indexes, and graph:

python data/prepare_dataset.py
python -c "from pipelines.basic_rag import index_chunks; index_chunks('data/chunks.jsonl')"
python -c "from pipelines.graph_rag import connect, load_graph_from_articles; connect(); load_graph_from_articles('data/articles.jsonl')"

Run the benchmark:

python benchmark/run_benchmark.py
python eval/accuracy_eval.py     # optional — needs benchmark/reference_answers.json filled in

Launch the dashboard:

cd dashboard && npm install && npm run dev

→ http://localhost:3000

---

📈 Results

(fill in after running the benchmark on your machine)

Metric	LLM-only	Basic RAG	GraphRAG
Avg tokens / query	—	—	—
Avg latency	—	—	—
LLM-as-judge pass rate	—	—	—
BERTScore F1	—	—	—

---

📁 Repo layout

Synapse-Lab/
├── data/
│   ├── prepare_dataset.py       # loads + chunks CNN/DailyMail
│   └── chunk_utils.py           # sentence-aware sliding-window chunker
├── pipelines/
│   ├── llm_only.py              # Pipeline 1
│   ├── basic_rag.py             # Pipeline 2 — Chroma + Gemini embeddings
│   ├── graph_rag.py             # Pipeline 3 — TigerGraph + Gemini
│   └── multi_hop_context.gsql   # graph traversal query
├── benchmark/
│   ├── run_benchmark.py         # runs all 3 pipelines, logs results.csv
│   └── reference_answers.json   # ground truth for accuracy eval
├── eval/
│   └── accuracy_eval.py         # BERTScore + LLM-as-judge
├── dashboard/                   # Next.js comparison UI
├── docs/
│   ├── architecture.svg
│   └── tigergraph_setup.md
├── docker-compose.yml
└── requirements.txt

---

🧰 Stack

Layer	Tech
Data	HuggingFace datasets · custom chunker
LLM	Gemini 1.5 Flash (generation + embeddings)
Vector store	Chroma
Graph	TigerGraph Community Edition · GSQL
Eval	BERTScore · LLM-as-judge
Dashboard	Next.js · Recharts

---

🗺 Roadmap

• Swap Gemini-based entity extraction for spaCy NER (offline graph build)
• Expand test query set — mix of single-hop and multi-hop questions
• Fill reference_answers.json with real ground truth
• Deploy dashboard (Vercel)
• Demo video

---

Built by Dishant Bhere

⭐ star the repo if this comparison was useful to you