From 64ab342cbd9640300c8bf43590355a7eb8a0aa17 Mon Sep 17 00:00:00 2001 From: Gal Shubeli Date: Sun, 9 Aug 2026 15:55:31 +0300 Subject: [PATCH 1/6] docs: update GraphRAG-Bench results and correct averaging method MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replaces the published GraphRAG-Bench figures with a full run across both subsets (August 2026): Novel 66.09, Medical 76.87, Overall 71.48. The previous Novel figure of 63.73 was a question-weighted average of the four task categories, while the leaderboard — and every competitor number shown beside it — uses the unweighted mean. Computed consistently, the old run scored 62.62 on Novel, so the comparison was not like for like. Both formulas are now stated explicitly under the table. Configuration also changes: gpt-4o-mini at temperature 0.7 for construction and generation, text-embedding-3-large at 1024 dimensions, text-to-Cypher enabled, scored with the benchmark's own generation_eval.py unmodified. docs/benchmark.md now covers both subsets, all fifteen leaderboard systems, the full configuration, reproduction steps, and the two settings raised above the SDK defaults. Also corrects claims that attributed 84.8% — an internal 100-question ablation scored 0-10 by an LLM judge, documented in docs/retrieval.md — to the 20-document novel benchmark. The example carrying that claim uses GLiNER2 NER, DescriptionMergeResolution and FixedSizeChunking, none of which produced the benchmark results. CHANGELOG.md is left untouched; it records what was claimed at the time. --- README.md | 23 +- docs/benchmark.md | 546 +++++------------- docs/index.md | 2 +- docs/ingestion.md | 2 +- docs/strategies.md | 4 +- graphrag_sdk/README.md | 4 +- graphrag_sdk/examples/03_custom_strategies.py | 8 +- 7 files changed, 160 insertions(+), 429 deletions(-) diff --git a/README.md b/README.md index 9fd2ff48..69a8924d 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,7 @@ Most GraphRAG systems work in demos and break under production constraints. Grap ## Benchmarks | Rank | System | Novel (Multi-Doc) | Medical (Single-Doc) | Overall | | :--- | :--- | :---: | :---: | :---: | -| **1** | **FalkorDB GraphRAG SDK ◄** | **63.73** | **75.73** | **69.73** | +| **1** | **FalkorDB GraphRAG SDK ◄** | **66.09** | **76.87** | **71.48** | | 2 | G-Reasoner | 58.94 | 73.30 | 66.12 | | 3 | AutoPrunedRetriever | 63.72 | 67.00 | 65.36 | | 4 | HippoRAG2 | 56.48 | 64.85 | 60.67 | @@ -33,7 +33,22 @@ Most GraphRAG systems work in demos and break under production constraints. Grap | 8 | HippoRAG | 44.75 | 59.08 | 51.92 | | 9 | MS-GraphRAG (local) | 50.93 | 45.16 | 48.05 | -> Overall ACC on [GraphRAG-Bench](https://graphrag-bench.github.io) Novel (20 novels, 2,010 questions) and Medical (1 corpus, 2,062 questions) datasets. FalkorDB scored with `gpt-4o-mini` (Azure OpenAI); competitor numbers are from the published leaderboard. Overall = mean of Novel and Medical ACC. See [docs/benchmark.md](docs/benchmark.md) for per-category breakdowns, methodology, and reproduction instructions. +> **How these are computed.** Per dataset, ACC is the unweighted mean of the four +> task-category scores, matching the [GraphRAG-Bench](https://graphrag-bench.github.io) +> leaderboard convention: +> +> `Dataset ACC = (Fact Retrieval + Complex Reasoning + Contextual Summarize + Creative Generation) / 4` +> `Overall = (Novel ACC + Medical ACC) / 2` +> +> Overall is our own summary across the two datasets; the leaderboard ranks each +> dataset separately. Novel has 20 documents and 2,010 questions, Medical 1 corpus +> and 2,062 questions. FalkorDB scored August 2026 with `gpt-4o-mini` (Azure OpenAI) +> at temperature 0.7 for both graph construction and generation, `text-embedding-3-large` +> at 1024 dimensions, text-to-Cypher retrieval enabled, and the benchmark's own +> `generation_eval.py` unmodified as the judge. Competitor numbers are from the +> published leaderboard, unchanged. See [docs/benchmark.md](docs/benchmark.md) for +> per-category results, the full 15-system comparison, configuration and reproduction +> instructions. Vectors match similar chunks. The graph traverses relationships. Every answer cites its source. @@ -109,7 +124,7 @@ async with GraphRAG(

→ Full walkthrough: Getting Started
- → Benchmark-winning recipe: Custom Strategies + → Compose your own pipeline: Custom Strategies

--- @@ -204,7 +219,7 @@ guards the default. |---|---------|-------------------| | 1 | [Quick Start](graphrag_sdk/examples/01_quickstart.py) | Your first ingest-and-query loop in under 30 lines | | 2 | [PDF with Schema](graphrag_sdk/examples/02_pdf_with_schema.py) | A PDF Q&A bot with your own entity and relation types | -| 3 | [Custom Strategies](graphrag_sdk/examples/03_custom_strategies.py) | The benchmark-winning pipeline, ready to drop in | +| 3 | [Custom Strategies](graphrag_sdk/examples/03_custom_strategies.py) | Composing ingestion and retrieval strategies explicitly | | 4 | [Custom Provider](graphrag_sdk/examples/04_custom_provider.py) | Plug in any LLM or embedder behind a clean interface | | 5 | [Notebook Demo](graphrag_sdk/examples/05_notebook_demo.ipynb) | An interactive walkthrough that shows the provenance trail | | 7 | [Incremental Updates](graphrag_sdk/examples/07_incremental_updates.py) | `update`, `delete_document`, and `apply_changes` for CI-driven graph syncs | diff --git a/docs/benchmark.md b/docs/benchmark.md index 17ebb58e..0a8286b2 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,431 +1,147 @@ -# Benchmarking - -This guide explains how to evaluate the GraphRAG SDK against academic benchmarks and your own datasets. It covers the evaluation methodology, dataset format, step-by-step reproduction with the SDK API, pipeline configuration options, and our published results on the [GraphRAG-Bench](https://graphrag-bench.github.io) Novel leaderboard. - ---- - -## Table of Contents - -1. [Overview](#overview) -2. [Prerequisites](#prerequisites) -3. [Datasets](#datasets) -4. [Reproducing with the SDK API](#reproducing-with-the-sdk-api) -5. [Pipeline Configuration](#pipeline-configuration) -6. [GraphRAG-Bench Novel Results](#graphrag-bench-novel-results) - ---- - -## Overview - -A benchmark run measures four dimensions: - -| Dimension | What it captures | -|-----------|------------------| -| **Accuracy** | Answer quality against ground-truth references (ACC, ROUGE-L, coverage) | -| **Ingestion throughput** | Time to chunk, extract, resolve, and build the knowledge graph | -| **Query latency** | End-to-end time from question submission to final answer | -| **Graph statistics** | Nodes, edges, and chunks produced — a proxy for knowledge density | - -### GraphRAG-Bench scoring system - -We use the official [GraphRAG-Bench](https://graphrag-bench.github.io) evaluation methodology. The primary leaderboard metric is **ACC** (answer correctness × 100), computed as: - -$$\text{ACC} = \bigl(0.75 \times \text{factuality\_F1} + 0.25 \times \text{semantic\_similarity}\bigr) \times 100$$ - -| Component | How it works | -|-----------|-------------| -| **Factuality F1** | An LLM decomposes both the generated answer and the ground truth into atomic statements, classifies each as TP / FP / FN, and computes F1 | -| **Semantic similarity** | Cosine similarity between answer and reference embeddings, scaled to \[0, 1\] | -| **ROUGE-L** | Longest common subsequence F1 — used for Fact Retrieval and Complex Reasoning | -| **Coverage score** | Fraction of reference facts present in the answer — used for Contextual Summarize and Creative Generation | - ---- - -## Prerequisites - -### Infrastructure - -Start a FalkorDB instance: - -```bash -docker run -p 6379:6379 falkordb/falkordb:latest -``` - -### Environment Variables - -Configure your LLM and embedding provider. Example for Azure OpenAI: - -```bash -export AZURE_OPENAI_API_KEY="your-key" -export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/" -export AZURE_OPENAI_API_VERSION="2024-12-01-preview" -export AZURE_OPENAI_DEPLOYMENT="gpt-4o-mini" -export AZURE_OPENAI_EMBEDDING_DEPLOYMENT="text-embedding-3-small" -``` - -Any [LiteLLM-supported provider](https://docs.litellm.ai/docs/providers) works — OpenAI, Anthropic, local models, etc. - -### Dependencies +# GraphRAG-SDK on GraphRAG-Bench + +Results for [GraphRAG-SDK](https://github.com/FalkorDB/GraphRAG-SDK) on +[GraphRAG-Bench](https://graphrag-bench.github.io/) (Xiang et al., ICLR 2026), +covering both subsets and all four task categories. + +Everything needed to reproduce these numbers — the runner, both ontologies, +pinned dependencies and a single command per subset — is published at +. + +## Results + +Scores are ACC unless noted. **Average is the unweighted mean of the four ACC +values**, the convention the GraphRAG-Bench leaderboard uses. + +### GraphRAG-Bench (Medical) — Average 76.87 + +| Level | ACC | ROUGE-L | Cov | FS | +| --- | --- | --- | --- | --- | +| Fact Retrieval | 74.50 | 45.43 | — | — | +| Complex Reasoning | 76.47 | 26.55 | — | — | +| Contextual Summarize | 82.29 | — | 62.02 | — | +| Creative Generation | 74.21 | — | 49.72 | 70.75 | + +### GraphRAG-Bench (Novel) — Average 66.09 + +| Level | ACC | ROUGE-L | Cov | FS | +| --- | --- | --- | --- | --- | +| Fact Retrieval | 65.49 | 41.82 | — | — | +| Complex Reasoning | 59.26 | 24.62 | — | — | +| Contextual Summarize | 75.42 | — | 54.11 | — | +| Creative Generation | 64.21 | — | 43.38 | 68.77 | + +### Against the published leaderboard + +Competitor figures are taken from the +[GraphRAG-Bench leaderboard](https://graphrag-bench.github.io/) as published; +we did not re-run them. All entries use `gpt-4o-mini` as the generation +backbone. + +| Method | Novel | Medical | +| --- | --- | --- | +| **GraphRAG-SDK (FalkorDB)** | **66.09** | **76.87** | +| AutoPrunedRetriever-llm | 63.72 | 67.00 | +| G-reasoner | 58.94 | 73.30 | +| HippoRAG2 | 56.48 | 64.85 | +| Fast-GraphRAG | 52.02 | 64.12 | +| MS-GraphRAG (local) | 50.93 | 45.16 | +| Lazy-GraphRAG | 50.59 | 56.89 | +| StructRAG | 49.13 | 58.56 | +| RAG (w/ rerank) | 48.35 | 62.43 | +| KGP | 48.01 | 56.33 | +| RAG (w/o rerank) | 47.93 | 61.00 | +| KET-RAG | 47.62 | 47.05 | +| LightRAG | 45.09 | 62.59 | +| HippoRAG | 44.75 | 59.08 | +| MS-GraphRAG (global) | 44.52 | 28.56 | +| RAPTOR | 43.24 | 57.10 | + +## Configuration + +Per the benchmark's Appendix H.2, GraphRAG-SDK's own defaults are preserved +rather than tuned to the benchmark; the protocol is matched on the backbone +model and the judge. + +| Setting | Value | Source | +| --- | --- | --- | +| Backbone LLM | `gpt-4o-mini` (Azure OpenAI) — graph construction *and* generation | Benchmark protocol | +| Generation temperature | 0.7 | Appendix H.2 | +| Framework | GraphRAG-SDK 1.3.0 (PyPI) on FalkorDB | — | +| Graph layout | one graph per corpus document | — | +| Chunking | `SentenceTokenCapChunking`, max_tokens 512, overlap 2 sentences | SDK default | +| Retrieval | `MultiPathRetrieval` — chunk_top_k 15, rel_top_k 15, max_entities 30, max_relationships 20, keyword_limit 10 | SDK default | +| Embeddings | `text-embedding-3-large` @ 1024 dimensions | Declared below | +| Text-to-Cypher | enabled | Declared below | +| Evaluation | benchmark's `Evaluation/generation_eval.py`, unmodified | Benchmark protocol | +| Judge | `gpt-4o-mini`, `BAAI/bge-large-en-v1.5` embeddings, temperature 0, seed 42 | Benchmark protocol | + +### Declared deviations from SDK defaults + +Two settings were raised above the SDK's own defaults. Both are single global +values, chosen before any results were scored, and applied identically to both +subsets. + +- **`enable_cypher = True`** (SDK default `False`) — ontology-guided + text-to-Cypher retrieval is a core GraphRAG-SDK capability, enabled so the + measurement reflects the framework as deployed. It contributed to 22% of + Medical and 29% of Novel answers. +- **`embedding_dimension = 1024`** (SDK default `256`) — the same vector width + as `bge-large-en-v1.5`, the embedding model Appendix H.2 specifies for + evaluated systems. + +### Benchmark integrity + +The ontologies (Medical: 10 entities / 13 relations; Novel: 8 / 13) were +hand-authored from the **corpus only**. The question sets and the `evidence`, +`evidence_relations` and `evidence_triple` fields were never read during +ontology design or indexing. The ontology builders are published alongside the +runner so this is checkable. + +## Reproducing ```bash -pip install graphrag-sdk[litellm] gliner -``` - ---- - -## Datasets - -We evaluate against [GraphRAG-Bench](https://graphrag-bench.github.io), an academic benchmark for graph-based retrieval-augmented generation. The datasets and questions are published in the GraphRAG-Bench project — download them from the official repository and place them in your dataset directory. - -### Available datasets - -| Dataset | Corpus | Questions | Docs | Questions | -|---------|--------|-----------|-----:|----------:| -| Novel (Full) | `novel.json` | `novel_questions.json` | 20 | 2,010 | -| Novel (Sample 100) | `novel.json` | `novel_questions_sample_100.json` | 20 | 100 | -| Medical | `medical.json` | `medical_questions.json` | 1 | 2,062 | - -> **Note:** These files are not included in this repository. Download them from [GraphRAG-Bench](https://graphrag-bench.github.io) and place them in a local directory (e.g., `datasets/`). - -### Data format - -**Corpus** — a JSON array of documents: - -```json -[ - { - "corpus_name": "Novel-30752", - "context": "Full text of the document..." - } -] -``` - -**Questions** — a JSON array of evaluation items: - -```json -[ - { - "id": "Novel-73586ddc", - "source": "Novel-44557", - "question": "Which plant known as Erica vagans is also called...?", - "answer": "Cornish heath", - "question_type": "Fact Retrieval", - "evidence": "The plant known scientifically as Erica vagans...", - "evidence_relations": ["..."] - } -] -``` - -Four question types are evaluated, each with different metrics: - -| Question Type | Metrics Used | -|---------------|-------------| -| Fact Retrieval | ROUGE-L + answer\_correctness (ACC) | -| Complex Reasoning | ROUGE-L + answer\_correctness (ACC) | -| Contextual Summarize | answer\_correctness (ACC) + coverage\_score | -| Creative Generation | answer\_correctness (ACC) + coverage\_score | - -To benchmark your own domain, create two JSON files following the same schema. - ---- - -## Reproducing with the SDK API - -The following walkthrough shows how to reproduce our benchmark results from scratch using the SDK's Python API. Following these exact steps with the same configuration and dataset will produce the results shown in the [GraphRAG-Bench Novel Results](#graphrag-bench-novel-results) section. - -### Step 1 — Initialize providers - -```python -import asyncio -import json -from graphrag_sdk import ConnectionConfig, GraphRAG, LiteLLM, LiteLLMEmbedder - -llm = LiteLLM( - model="azure/gpt-4o-mini", - api_key="...", - api_base="https://your-resource.openai.azure.com/", - api_version="2024-12-01-preview", -) - -embedder = LiteLLMEmbedder( - model="azure/text-embedding-3-small", - api_key="...", - api_base="https://your-resource.openai.azure.com/", - api_version="2024-12-01-preview", -) -``` - -### Step 2 — Create a GraphRAG instance +docker run -d -p 6379:6379 --name falkordb falkordb/falkordb:latest -```python -rag = GraphRAG( - connection=ConnectionConfig(host="localhost", port=6379, graph_name="novel_bench"), - llm=llm, - embedder=embedder, -) -``` - -### Step 3 — Configure the ingestion pipeline - -```python -from graphrag_sdk.core.context import Context -from graphrag_sdk.ingestion.chunking_strategies.sentence_token_cap import SentenceTokenCapChunking -from graphrag_sdk.ingestion.extraction_strategies.graph_extraction import GraphExtraction -from graphrag_sdk.ingestion.extraction_strategies.entity_extractors import GLiNERExtractor -from graphrag_sdk.ingestion.extraction_strategies.coref_resolvers import FastCorefResolver -from graphrag_sdk.ingestion.resolution_strategies.base import ResolutionStrategy -from graphrag_sdk.ingestion.resolution_strategies.exact_match import ExactMatchResolution -from graphrag_sdk.ingestion.resolution_strategies.description_merge import DescriptionMergeResolution -from graphrag_sdk.ingestion.resolution_strategies.semantic_resolution import SemanticResolution -from graphrag_sdk.ingestion.resolution_strategies.llm_verified_resolution import LLMVerifiedResolution - -chunker = SentenceTokenCapChunking(max_tokens=512, overlap_sentences=2) - -extractor = GraphExtraction( - llm=llm, - entity_extractor=GLiNERExtractor(model_name="urchade/gliner_medium-v2.1"), - coref_resolver=FastCorefResolver(), -) -``` - -### Step 4 — Chain resolution stages - -Multiple resolution strategies run in sequence — each stage feeds its output into the next. Create a simple chained resolver: - -```python -from graphrag_sdk.core.models import GraphData, ResolutionResult - -class ChainedResolution(ResolutionStrategy): - """Run multiple resolution strategies in sequence.""" - def __init__(self, *stages): - self._stages = stages - - async def resolve(self, graph_data, ctx): - for stage in self._stages: - result = await stage.resolve(graph_data, ctx) - # Convert ResolutionResult back to GraphData for the next stage - graph_data = GraphData( - nodes=result.nodes, - relationships=result.relationships, - ) - return ResolutionResult( - nodes=graph_data.nodes, - relationships=graph_data.relationships, - ) - -resolver = ChainedResolution( - ExactMatchResolution(resolve_property="name"), - DescriptionMergeResolution(llm=llm), - SemanticResolution(embedder=embedder, similarity_threshold=0.85), - LLMVerifiedResolution(llm=llm, embedder=embedder, hard_threshold=0.95, soft_threshold=0.60), -) -``` - -### Step 5 — Ingest the corpus - -```python -corpus = json.load(open("datasets/novel.json")) - -for doc in corpus: - await rag.ingest( - doc["corpus_name"], - text=doc["context"], - chunker=chunker, - extractor=extractor, - resolver=resolver, - ctx=Context(tenant_id=doc["corpus_name"]), - ) -``` - -### Step 6 — Finalize the graph - -```python -await rag.finalize() -``` - -This removes null/stub entities, deduplicates across documents, embeds all entities and relationships, and creates vector and fulltext indexes in FalkorDB. - -### Step 7 — Query +python -m venv .venv && ./.venv/bin/pip install -r Examples/graphragsdk/requirements.txt +python -m venv .venv-eval && ./.venv-eval/bin/pip install -r Examples/graphragsdk/requirements-eval.txt -```python -questions = json.load(open("datasets/novel_questions.json")) +export AZURE_API_KEY=... AZURE_API_BASE=https://.openai.azure.com/ -results = [] -for q in questions: - result = await rag.completion(q["question"]) - results.append({ - "question": q["question"], - "answer": result.answer, - "reference": q["answer"], - "question_type": q["question_type"], - }) +PYTHON=./.venv/bin/python bash Examples/graphragsdk/run_all.sh ``` -### Step 8 — Evaluate with GraphRAG-Bench metrics - -For each question, compute the official [GraphRAG-Bench](https://graphrag-bench.github.io) metrics: - -**Answer Correctness (ACC)** — the primary leaderboard metric: - -1. The LLM decomposes both the generated answer and the ground truth into atomic statements -2. Each statement is classified as TP (true positive), FP (false positive), or FN (false negative) -3. Factuality F1 is computed from the TP / FP / FN counts -4. Semantic similarity is the cosine similarity between answer and reference embeddings, scaled to [0, 1] -5. Final score: `ACC = 0.75 × factuality_F1 + 0.25 × semantic_similarity` - -**ROUGE-L** — longest common subsequence F1 between answer and reference. Applied to Fact Retrieval and Complex Reasoning questions. - -**Coverage Score** — the LLM extracts facts from the reference and checks what fraction is covered in the answer. Applied to Contextual Summarize and Creative Generation questions. - -After running evaluation across all questions, aggregate the results: - -```python -from collections import defaultdict - -by_type = defaultdict(list) -for r in results: - acc = compute_answer_correctness(llm, embedder, r["question"], r["answer"], r["reference"]) - by_type[r["question_type"]].append(acc) - -# Per-type and overall ACC -for q_type, scores in by_type.items(): - avg_acc = sum(scores) / len(scores) * 100 - print(f"{q_type}: ACC = {avg_acc:.2f}") - -all_scores = [s for scores in by_type.values() for s in scores] -overall_acc = sum(all_scores) / len(all_scores) * 100 -print(f"Overall ACC: {overall_acc:.2f}") -``` - -This produces the accuracy tables, graph statistics, and leaderboard comparison shown in the [results section below](#graphrag-bench-novel-results). - ---- - -## Pipeline Configuration - -The ingestion and retrieval pipeline is fully composable. Each stage can be swapped independently. - -### Chunking strategies - -| Strategy | Description | -|----------|-------------| -| `SentenceTokenCapChunking(max_tokens, overlap_sentences)` | Splits on sentence boundaries with a configurable token cap. Best for most use cases. | - -### Extraction strategies - -| Strategy | Description | -|----------|-------------| -| `GraphExtraction(llm, entity_extractor=GLiNERExtractor(), coref_resolver=FastCorefResolver())` | Local NER (no API cost) + coreference resolution + LLM for relationships. Best accuracy. | -| `GraphExtraction(llm)` | LLM-only extraction. Higher API cost per document. | - -### Resolution strategies +Each subset runs index → answer → judge → summarize, and writes its leaderboard +row, the full configuration, per-question predictions with retrieved context, +and each answer's judge scores. The runner and evaluator both use separate +virtualenvs (the evaluator pins `openai<2`, the runner needs `openai>=2.20`). -Chain multiple resolvers in sequence using a `ChainedResolution` wrapper (see [Step 4](#step-4--chain-resolution-stages)). Each stage feeds its deduplicated output into the next: +Approximate cost on a 24-core machine: Medical ~1.5 h, Novel ~4 h, dominated by +extraction API calls. -| Strategy | Description | -|----------|-------------| -| `ExactMatchResolution(resolve_property="name")` | Merges entities with identical names. Zero API cost. | -| `DescriptionMergeResolution(llm)` | LLM merges entities with similar descriptions. | -| `SemanticResolution(embedder, similarity_threshold)` | Cosine similarity on embeddings with hnswlib ANN index. No LLM calls. | -| `LLMVerifiedResolution(llm, embedder, hard_threshold, soft_threshold)` | Two-tier: auto-merge above hard threshold, LLM-verify between soft and hard. Uses Louvain community detection. | +## Counts and caveats -**Winning chain** (used in our benchmark): +| | Medical | Novel | +| --- | --- | --- | +| Questions in dataset | 2062 | 2010 | +| Predictions produced | 2062 | 2009 | +| Generation errors / empty answers | 0 / 0 | 0 / 0 | +| Samples scored by the evaluator | 2053 | 1995 | -```python -resolver = ChainedResolution( - ExactMatchResolution(resolve_property="name"), - DescriptionMergeResolution(llm=llm), - SemanticResolution(embedder=embedder, similarity_threshold=0.85), - LLMVerifiedResolution(llm=llm, embedder=embedder, hard_threshold=0.95, soft_threshold=0.60), -) -``` - -### Retrieval strategies - -| Strategy | Description | -|----------|-------------| -| `MultiPathRetrieval` (default) | Multi-path entity discovery, 2-hop graph expansion, chunk retrieval, cosine rerank. No configuration required. | - -### Post-ingestion: `finalize()` +Two things worth stating plainly: -Always call `await rag.finalize()` after ingesting all documents: +**Novel is 2009 rather than 2010.** The dataset contains a duplicate question +id (`Novel-55f0c0e2`) carrying identical question text under two different +`source` values with contradictory ground truths — one answers the question +asked, the other is unrelated. Our runner answers each id once. This has been +reported to the benchmark authors. -- Removes null/stub entities -- Deduplicates across document boundaries -- Embeds all entities and relationships -- Creates vector and fulltext indexes in FalkorDB +**The evaluator skips samples whose judge call raises**, so the averages are +over 2053 of 2062 Medical and 1995 of 2009 Novel predictions. Every prediction +itself succeeded; there were no generation failures or empty answers in either +subset. --- -## GraphRAG-Bench Novel Results - -The following results were produced by running the pipeline described above on the complete [GraphRAG-Bench](https://graphrag-bench.github.io) Novel dataset (20 novels, 2,010 questions). - -### Configuration - -| Parameter | Value | -|-----------|-------| -| LLM | gpt-4o-mini (Azure OpenAI) | -| Embeddings | text-embedding-3-small (Azure OpenAI) | -| Chunking | SentenceTokenCapChunking — max\_tokens=512, overlap\_sentences=2 | -| Extraction | GLiNER v2.1 + FastCoref + LLM relationship extraction | -| Resolution | ExactMatch (name) → DescriptionMerge → Semantic (0.85) → LLMVerified (0.95 / 0.60) | -| Retrieval | MultiPathRetrieval | -| Corpus | `novel.json` — 20 novels, 4.7 MB | -| Questions | `novel_questions.json` — 2,010 questions | - -### Accuracy (official GraphRAG-Bench ACC) - -| Question Type | ACC (×100) | ROUGE-L | Coverage | -|---------------|----------:|--------:|---------:| -| Fact Retrieval | 65.22 | 35.95 | — | -| Complex Reasoning | 58.63 | 22.39 | — | -| Contextual Summarize | 69.54 | — | 55.21 | -| Creative Generation | 57.08 | — | 44.52 | -| **Overall** | **63.73** | — | — | - -### Leaderboard comparison - -> **Note:** Only the FalkorDB GraphRAG-SDK row was produced by us using the pipeline described above. All other system scores are taken from the [GraphRAG-Bench published leaderboard](https://graphrag-bench.github.io) as of April 2025. Leaderboard rankings may change as systems are updated and re-evaluated. - -| System | Fact Retrieval | Complex Reasoning | Contextual Summarize | Creative Generation | Overall | -|--------|------:|------:|------:|------:|------:| -| **FalkorDB GraphRAG-SDK** | **65.22** | **58.63** | **69.54** | **57.08** | **63.73** | -| AutoPrunedRetriever | 45.99 | 62.80 | 83.10 | 62.97 | 63.72 | -| G-Reasoner | 60.07 | 53.92 | 71.28 | 50.48 | 58.94 | -| HippoRAG2 | 60.14 | 53.38 | 64.10 | 48.28 | 56.48 | -| Fast-GraphRAG | 56.95 | 48.55 | 56.41 | 46.18 | 52.02 | -| MS-GraphRAG (local) | 49.29 | 50.93 | 64.40 | 39.10 | 50.93 | -| RAG (w/ rerank) | 60.92 | 42.93 | 51.30 | 38.26 | 48.35 | -| LightRAG | 58.62 | 49.07 | 48.85 | 23.80 | 45.09 | -| HippoRAG | 52.93 | 38.52 | 48.70 | 38.85 | 44.75 | - -Source: [graphrag-bench.github.io](https://graphrag-bench.github.io) — Novel leaderboard. - -### Graph statistics - -| Metric | Value | -|--------|------:| -| Total nodes | 8,765 | -| Total edges | 25,895 | -| Total chunks | 2,782 | -| Documents | 20 | - -### Timing - -| Phase | Duration | -|-------|--------:| -| Avg. query latency | 3.6 s | - -### Evaluation methodology - -Scores use the official [GraphRAG-Bench](https://graphrag-bench.github.io) evaluation suite, -ported from `github.com/GraphRAG-Bench/GraphRAG-Benchmark/Evaluation`: - -| Component | How it works | Judge LLM | -|-----------|-------------|-----------| -| **answer_correctness** | 0.75 × Factuality F1 + 0.25 × Semantic Similarity. The LLM decomposes both the answer and reference into atomic statements, classifies TP / FP / FN, and computes F1. Semantic similarity is cosine similarity of answer vs reference embeddings. | gpt-4o-mini | -| **rouge_score** | ROUGE-L F1 (used for Fact Retrieval & Complex Reasoning) | — (algorithmic) | -| **coverage_score** | The LLM extracts facts from the reference and checks which are covered in the answer (used for Contextual Summarize & Creative Generation) | gpt-4o-mini | - -**ACC** reported on the leaderboard = `answer_correctness × 100`, averaged per question type. +*Leaderboard figures as published at graphrag-bench.github.io. Last updated: +August 2026.* diff --git a/docs/index.md b/docs/index.md index f1372f01..ff849fbf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,7 +6,7 @@ GraphRAG SDK builds knowledge graphs from documents and answers questions over t ## Key Highlights -- **#1 on GraphRAG-Bench Novel** — 63.73 ACC on 2,010 questions ([benchmark](benchmark.md)) +- **#1 on GraphRAG-Bench** — 66.09 ACC on Novel and 76.87 on Medical ([benchmark](benchmark.md)) - **Simple API** -- `ingest()` + `completion()` with sensible defaults - **100+ LLM providers** via LiteLLM (OpenAI, Azure, Anthropic, Cohere, Ollama, and more) - **Fully modular** -- swap chunking, extraction, resolution, retrieval, and reranking strategies diff --git a/docs/ingestion.md b/docs/ingestion.md index 1bb0b6c9..f9d5b680 100644 --- a/docs/ingestion.md +++ b/docs/ingestion.md @@ -165,7 +165,7 @@ For a detailed explanation of the extraction process, see [extraction.md](extrac **Alternative: DescriptionMergeResolution** - Groups by `(normalized name, label)` — same name but different labels stay separate (e.g., Person "Paris" vs Location "Paris") - Merges descriptions (concatenation or LLM summarization) -- Used in the benchmark-winning pipeline +- Useful when the same entity is described differently across documents For details on resolution strategies, see [strategies.md](strategies.md). diff --git a/docs/strategies.md b/docs/strategies.md index 3c3c4c60..c0faf017 100644 --- a/docs/strategies.md +++ b/docs/strategies.md @@ -428,7 +428,7 @@ resolver = DescriptionMergeResolution( ) ``` -**When to use:** Multi-document ingestion where the same entity appears with different descriptions. Used in the benchmark-winning pipeline. +**When to use:** Multi-document ingestion where the same entity appears with different descriptions. Useful when the same entity is described differently across documents. --- @@ -476,7 +476,7 @@ retriever = LocalRetrieval( ### Built-in: MultiPathRetrieval -Production-grade retrieval with RELATES edge vector search, 2-path entity discovery, 4-path chunk retrieval, and cosine reranking. This is the **default** and the benchmark-winning strategy. +Production-grade retrieval with RELATES edge vector search, 2-path entity discovery, 4-path chunk retrieval, and cosine reranking. This is the **default**, and the strategy used for the [GraphRAG-Bench results](benchmark.md). ```python from graphrag_sdk import MultiPathRetrieval diff --git a/graphrag_sdk/README.md b/graphrag_sdk/README.md index babf40ef..81c4504d 100644 --- a/graphrag_sdk/README.md +++ b/graphrag_sdk/README.md @@ -152,11 +152,11 @@ Every algorithmic concern is a swappable strategy behind an abstract base class: ## Benchmark -**#1 on [GraphRAG-Bench](https://graphrag-bench.github.io) Novel** — 63.73 ACC, ahead of MS-GraphRAG (50.93) and LightRAG (45.09). +**#1 on [GraphRAG-Bench](https://graphrag-bench.github.io)** — 66.09 ACC on Novel and 76.87 on Medical, ahead of AutoPrunedRetriever (63.72 / 67.00) and G-Reasoner (58.94 / 73.30). | Metric | Value | |--------|-------| -| **Novel ACC** | 63.73 (#1) | +| **Novel ACC** | 66.09 (#1) | | **Fact retrieval** | 65.22 | | **Complex reasoning** | 58.63 | | **Contextual summarization** | 69.54 | diff --git a/graphrag_sdk/examples/03_custom_strategies.py b/graphrag_sdk/examples/03_custom_strategies.py index 3299afa0..bee340af 100644 --- a/graphrag_sdk/examples/03_custom_strategies.py +++ b/graphrag_sdk/examples/03_custom_strategies.py @@ -1,8 +1,8 @@ """ -GraphRAG SDK -- Custom Strategies (Benchmark-Winning Pipeline) +GraphRAG SDK -- Custom Strategies ================================================================== -Demonstrates the full pipeline configuration that achieved 84.8% accuracy -on the 20-document novel benchmark. Uses: +Demonstrates composing the ingestion and retrieval strategies explicitly +rather than relying on the defaults. Uses: - GraphExtraction (GLiNER2 NER + LLM relationship extraction) - DescriptionMergeResolution (LLM-assisted entity dedup) - Post-ingestion finalize (dedup, embeddings, indexes) @@ -125,7 +125,7 @@ async def main(): pass # --- Ingestion with custom strategies --- - print("Ingesting documents with benchmark-winning pipeline...") + print("Ingesting documents with custom strategies...") t0 = time.time() for source_id, text in DOCUMENTS: From 92001efe1c82211d1663677793b5b83a71bef17d Mon Sep 17 00:00:00 2001 From: Gal Shubeli Date: Sun, 9 Aug 2026 16:06:42 +0300 Subject: [PATCH 2/6] docs: reproduction package by request rather than a public link The runner package is provided on request instead of published, so the benchmark page no longer points at a repository URL. The configuration and the full reproduction procedure stay documented inline, including where the package has to sit inside a GraphRAG-Benchmark checkout for the paths in the commands to resolve. --- docs/benchmark.md | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/docs/benchmark.md b/docs/benchmark.md index 0a8286b2..429ee52c 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -4,9 +4,10 @@ Results for [GraphRAG-SDK](https://github.com/FalkorDB/GraphRAG-SDK) on [GraphRAG-Bench](https://graphrag-bench.github.io/) (Xiang et al., ICLR 2026), covering both subsets and all four task categories. -Everything needed to reproduce these numbers — the runner, both ontologies, -pinned dependencies and a single command per subset — is published at -. +The reproduction package — the runner, both ontologies, the ontology builders +and pinned dependencies — is available on request from +[gal.shubeli@falkordb.com](mailto:gal.shubeli@falkordb.com). The configuration +and procedure are documented in full below. ## Results @@ -100,6 +101,12 @@ runner so this is checkable. ## Reproducing +The runner reads the benchmark's own `Datasets/` and `Evaluation/` directories +and resolves them relative to its own location, so the package is placed at +`Examples/graphragsdk/` inside a clone of +[GraphRAG-Benchmark](https://github.com/GraphRAG-Bench/GraphRAG-Benchmark). +From that checkout: + ```bash docker run -d -p 6379:6379 --name falkordb falkordb/falkordb:latest From 6faaf6b03979ddf2a696749f082aee6c61606f1c Mon Sep 17 00:00:00 2001 From: Gal Shubeli Date: Sun, 9 Aug 2026 16:11:18 +0300 Subject: [PATCH 3/6] docs: publish the benchmark ontologies Adds both hand-authored ontologies used for the GraphRAG-Bench run. They are the artifact behind the corpus-only claim, so publishing them lets that claim be checked without needing the runner: the files contain domain type definitions only, with no question, answer or evidence text. Requests for the runner go through Issues or Discord rather than a personal address, matching the channels used elsewhere in the repo. --- docs/benchmark.md | 28 ++- docs/benchmark/ontology_medical.json | 274 +++++++++++++++++++++++++++ docs/benchmark/ontology_novel.json | 267 ++++++++++++++++++++++++++ 3 files changed, 560 insertions(+), 9 deletions(-) create mode 100644 docs/benchmark/ontology_medical.json create mode 100644 docs/benchmark/ontology_novel.json diff --git a/docs/benchmark.md b/docs/benchmark.md index 429ee52c..2a945440 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -4,10 +4,11 @@ Results for [GraphRAG-SDK](https://github.com/FalkorDB/GraphRAG-SDK) on [GraphRAG-Bench](https://graphrag-bench.github.io/) (Xiang et al., ICLR 2026), covering both subsets and all four task categories. -The reproduction package — the runner, both ontologies, the ontology builders -and pinned dependencies — is available on request from -[gal.shubeli@falkordb.com](mailto:gal.shubeli@falkordb.com). The configuration -and procedure are documented in full below. +The reproduction package — the runner, the ontology builders and pinned +dependencies — is available on request; open an +[issue](https://github.com/FalkorDB/GraphRAG-SDK/issues) or ask on +[Discord](https://discord.gg/6M4QwDXn2w). The ontologies themselves are +published below, and the configuration and procedure are documented in full. ## Results @@ -93,11 +94,20 @@ subsets. ### Benchmark integrity -The ontologies (Medical: 10 entities / 13 relations; Novel: 8 / 13) were -hand-authored from the **corpus only**. The question sets and the `evidence`, -`evidence_relations` and `evidence_triple` fields were never read during -ontology design or indexing. The ontology builders are published alongside the -runner so this is checkable. +The ontologies were hand-authored from the **corpus only**. The question sets +and the `evidence`, `evidence_relations` and `evidence_triple` fields were never +read during ontology design or indexing. + +Both are published here so this is checkable without needing the runner: + +| Subset | Ontology | Entities | Relations | +| --- | --- | ---: | ---: | +| Medical | [`ontology_medical.json`](benchmark/ontology_medical.json) | 10 | 13 | +| Novel | [`ontology_novel.json`](benchmark/ontology_novel.json) | 8 | 13 | + +They contain only domain type definitions — entity labels, their descriptions +and properties, and the relation patterns between them. No question, answer or +evidence text appears in either file. ## Reproducing diff --git a/docs/benchmark/ontology_medical.json b/docs/benchmark/ontology_medical.json new file mode 100644 index 00000000..02cfb553 --- /dev/null +++ b/docs/benchmark/ontology_medical.json @@ -0,0 +1,274 @@ +{ + "entities": [ + { + "label": "Disease", + "description": "A disease, disorder, cancer, or medical condition.", + "properties": [ + { + "name": "category", + "type": "STRING", + "description": "e.g. carcinoma, sarcoma, infection." + }, + { + "name": "stage", + "type": "STRING", + "description": "Stage or grade if given." + } + ] + }, + { + "label": "Symptom", + "description": "A sign, symptom, or clinical manifestation.", + "properties": [] + }, + { + "label": "Treatment", + "description": "A therapy, procedure, surgery, or intervention (e.g. brachytherapy, radiation, chemotherapy).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "surgery, radiation, therapy, procedure, etc." + } + ] + }, + { + "label": "Drug", + "description": "A medication, drug, or pharmacological agent.", + "properties": [ + { + "name": "drug_class", + "type": "STRING", + "description": "Drug class if stated." + } + ] + }, + { + "label": "AnatomicalStructure", + "description": "An organ, gland, tissue, or body part (e.g. prostate, urethra, lung).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "organ, gland, tissue, cell, etc." + } + ] + }, + { + "label": "DiagnosticTest", + "description": "A diagnostic test, imaging modality, biopsy, or screening method.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "lab test, imaging, biopsy, screening, etc." + } + ] + }, + { + "label": "RiskFactor", + "description": "A risk factor or cause associated with a disease (e.g. smoking, age, exposure).", + "properties": [] + }, + { + "label": "Biomarker", + "description": "A biomarker, protein, hormone, antigen, or measurable substance (e.g. PSA).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "protein, hormone, antigen, gene, etc." + } + ] + }, + { + "label": "Pathogen", + "description": "An infectious agent — virus, bacterium, or other pathogen (e.g. HPV).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "virus, bacterium, etc." + } + ] + }, + { + "label": "BodySystem", + "description": "A body system (e.g. reproductive, respiratory, immune).", + "properties": [] + } + ], + "relations": [ + { + "label": "CAUSES", + "description": "A risk factor, pathogen, or disease causes a disease or symptom.", + "patterns": [ + [ + "RiskFactor", + "Disease" + ], + [ + "Pathogen", + "Disease" + ], + [ + "Disease", + "Symptom" + ] + ], + "properties": [] + }, + { + "label": "TREATS", + "description": "A treatment or drug is used to treat a disease.", + "patterns": [ + [ + "Treatment", + "Disease" + ], + [ + "Drug", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "SYMPTOM_OF", + "description": "A symptom is a manifestation of a disease.", + "patterns": [ + [ + "Symptom", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "DIAGNOSED_BY", + "description": "A disease is diagnosed or detected by a test.", + "patterns": [ + [ + "Disease", + "DiagnosticTest" + ] + ], + "properties": [] + }, + { + "label": "AFFECTS", + "description": "A disease affects an anatomical structure or body system.", + "patterns": [ + [ + "Disease", + "AnatomicalStructure" + ], + [ + "Disease", + "BodySystem" + ] + ], + "properties": [] + }, + { + "label": "PART_OF", + "description": "An anatomical structure is part of a larger structure or body system.", + "patterns": [ + [ + "AnatomicalStructure", + "AnatomicalStructure" + ], + [ + "AnatomicalStructure", + "BodySystem" + ] + ], + "properties": [] + }, + { + "label": "RISK_FACTOR_FOR", + "description": "A risk factor increases the risk of a disease.", + "patterns": [ + [ + "RiskFactor", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "PREVENTS", + "description": "A treatment or drug prevents a disease.", + "patterns": [ + [ + "Treatment", + "Disease" + ], + [ + "Drug", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "PRODUCES", + "description": "An anatomical structure produces a biomarker or substance.", + "patterns": [ + [ + "AnatomicalStructure", + "Biomarker" + ] + ], + "properties": [] + }, + { + "label": "INDICATES", + "description": "A biomarker or test result indicates a disease.", + "patterns": [ + [ + "Biomarker", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "HAS_SUBTYPE", + "description": "A disease has a subtype or variant.", + "patterns": [ + [ + "Disease", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "SIDE_EFFECT_OF", + "description": "A symptom is a side effect of a treatment or drug.", + "patterns": [ + [ + "Symptom", + "Treatment" + ], + [ + "Symptom", + "Drug" + ] + ], + "properties": [] + }, + { + "label": "ADMINISTERED_TO", + "description": "A treatment or drug is administered to an anatomical structure.", + "patterns": [ + [ + "Treatment", + "AnatomicalStructure" + ] + ], + "properties": [] + } + ] +} \ No newline at end of file diff --git a/docs/benchmark/ontology_novel.json b/docs/benchmark/ontology_novel.json new file mode 100644 index 00000000..46b1fe9e --- /dev/null +++ b/docs/benchmark/ontology_novel.json @@ -0,0 +1,267 @@ +{ + "entities": [ + { + "label": "Character", + "description": "A person or personified being — a fictional character, a historical figure, an author, or a real individual named in the text.", + "properties": [ + { + "name": "aliases", + "type": "LIST", + "description": "Other names, titles, or epithets used for this person." + }, + { + "name": "role", + "type": "STRING", + "description": "Their role or occupation (e.g. narrator, doctor, king)." + }, + { + "name": "gender", + "type": "STRING", + "description": "Gender if stated or clearly implied." + } + ] + }, + { + "label": "Location", + "description": "A named place: country, region, city, building, or natural feature.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Type of place: country, city, region, building, natural feature, etc." + } + ] + }, + { + "label": "Organization", + "description": "A named group, institution, society, company, family line, or faction.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Type of organisation: society, company, family, government, etc." + } + ] + }, + { + "label": "Work", + "description": "A named creative or scholarly work: book, poem, play, essay, or publication referenced in the text.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Type of work: novel, poem, play, essay, treatise, etc." + }, + { + "name": "author", + "type": "STRING", + "description": "Named author of the work, if given." + } + ] + }, + { + "label": "Object", + "description": "A notable physical object, artifact, substance, or material referred to in the text.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Category of the object or substance." + } + ] + }, + { + "label": "Event", + "description": "A discrete happening, action, or episode in the narrative (journey, battle, death, discovery, ceremony).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Type of event: journey, battle, death, discovery, meeting, etc." + }, + { + "name": "time", + "type": "STRING", + "description": "When it occurred, as stated in the text." + } + ] + }, + { + "label": "Concept", + "description": "An abstract idea, theme, belief, scientific principle, or topic discussed in the text.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Category of the concept: theme, scientific principle, belief, topic, etc." + } + ] + }, + { + "label": "TimePeriod", + "description": "A named point or span of time: a date, year, era, season, or historical period.", + "properties": [] + } + ], + "relations": [ + { + "label": "INTERACTS_WITH", + "description": "One character speaks to, meets, or otherwise interacts with another.", + "patterns": [ + [ + "Character", + "Character" + ] + ], + "properties": [] + }, + { + "label": "FAMILY_OF", + "description": "A family or kinship tie between two characters.", + "patterns": [ + [ + "Character", + "Character" + ] + ], + "properties": [] + }, + { + "label": "MEMBER_OF", + "description": "A character belongs to an organisation, group, or family line.", + "patterns": [ + [ + "Character", + "Organization" + ] + ], + "properties": [] + }, + { + "label": "LOCATED_IN", + "description": "A character, organisation, or event is situated in a location.", + "patterns": [ + [ + "Character", + "Location" + ], + [ + "Organization", + "Location" + ], + [ + "Event", + "Location" + ] + ], + "properties": [] + }, + { + "label": "TRAVELS_TO", + "description": "A character moves or journeys to a location.", + "patterns": [ + [ + "Character", + "Location" + ] + ], + "properties": [] + }, + { + "label": "PART_OF", + "description": "A location or organisation is a part of a larger one.", + "patterns": [ + [ + "Location", + "Location" + ], + [ + "Organization", + "Organization" + ] + ], + "properties": [] + }, + { + "label": "PARTICIPATES_IN", + "description": "A character takes part in an event.", + "patterns": [ + [ + "Character", + "Event" + ] + ], + "properties": [] + }, + { + "label": "OWNS", + "description": "A character owns or possesses an object.", + "patterns": [ + [ + "Character", + "Object" + ] + ], + "properties": [] + }, + { + "label": "AUTHORED", + "description": "A character (author) created a work.", + "patterns": [ + [ + "Character", + "Work" + ] + ], + "properties": [] + }, + { + "label": "ASSOCIATED_WITH", + "description": "A character or work is associated with a concept or theme.", + "patterns": [ + [ + "Character", + "Concept" + ], + [ + "Work", + "Concept" + ] + ], + "properties": [] + }, + { + "label": "CAUSES", + "description": "One event causes or leads to another.", + "patterns": [ + [ + "Event", + "Event" + ] + ], + "properties": [] + }, + { + "label": "PRECEDES", + "description": "One event happens before another in time.", + "patterns": [ + [ + "Event", + "Event" + ] + ], + "properties": [] + }, + { + "label": "OCCURS_DURING", + "description": "An event takes place during a named time period.", + "patterns": [ + [ + "Event", + "TimePeriod" + ] + ], + "properties": [] + } + ] +} \ No newline at end of file From 1bb5c1b6a048e27aae3db3940b7c1602eaca4c87 Mon Sep 17 00:00:00 2001 From: Gal Shubeli Date: Sun, 9 Aug 2026 16:19:20 +0300 Subject: [PATCH 4/6] docs: describe the benchmark pipeline instead of giving unusable commands MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The reproduction commands referenced a package the reader cannot obtain, and the two-virtualenv note is an install detail that only matters once someone has it. Replaced with what the run actually does per stage, what it produces, and what it costs — which is the part that is meaningful without the package. --- docs/benchmark.md | 42 +++++++++++++++++++----------------------- 1 file changed, 19 insertions(+), 23 deletions(-) diff --git a/docs/benchmark.md b/docs/benchmark.md index 2a945440..2872a573 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -8,7 +8,7 @@ The reproduction package — the runner, the ontology builders and pinned dependencies — is available on request; open an [issue](https://github.com/FalkorDB/GraphRAG-SDK/issues) or ask on [Discord](https://discord.gg/6M4QwDXn2w). The ontologies themselves are -published below, and the configuration and procedure are documented in full. +published below, along with the full configuration. ## Results @@ -109,32 +109,28 @@ They contain only domain type definitions — entity labels, their descriptions and properties, and the relation patterns between them. No question, answer or evidence text appears in either file. -## Reproducing +## How the run works -The runner reads the benchmark's own `Datasets/` and `Evaluation/` directories -and resolves them relative to its own location, so the package is placed at -`Examples/graphragsdk/` inside a clone of -[GraphRAG-Benchmark](https://github.com/GraphRAG-Bench/GraphRAG-Benchmark). -From that checkout: +Per subset, the pipeline runs in three stages: -```bash -docker run -d -p 6379:6379 --name falkordb falkordb/falkordb:latest +1. **Index** — each corpus document is chunked, and entities and relationships + are extracted against that subset's ontology into its own FalkorDB graph + (one graph per document, so retrieval is scoped to the document a question + was written against). +2. **Answer** — every question in the subset is answered through + `MultiPathRetrieval`, with the retrieved context recorded alongside the + answer. +3. **Judge** — answers are scored by the benchmark's own + `Evaluation/generation_eval.py`, unmodified, with `gpt-4o-mini` as judge and + `bge-large-en-v1.5` embeddings. -python -m venv .venv && ./.venv/bin/pip install -r Examples/graphragsdk/requirements.txt -python -m venv .venv-eval && ./.venv-eval/bin/pip install -r Examples/graphragsdk/requirements-eval.txt +The run produces, per subset: the leaderboard row, the fully resolved +configuration, per-question predictions with their retrieved context, and each +answer's individual judge scores. -export AZURE_API_KEY=... AZURE_API_BASE=https://.openai.azure.com/ - -PYTHON=./.venv/bin/python bash Examples/graphragsdk/run_all.sh -``` - -Each subset runs index → answer → judge → summarize, and writes its leaderboard -row, the full configuration, per-question predictions with retrieved context, -and each answer's judge scores. The runner and evaluator both use separate -virtualenvs (the evaluator pins `openai<2`, the runner needs `openai>=2.20`). - -Approximate cost on a 24-core machine: Medical ~1.5 h, Novel ~4 h, dominated by -extraction API calls. +Indexing cost, measured on a 24-core machine and dominated by extraction API +calls rather than local compute: Medical (1 corpus, 510 chunks) ~1.5 h end to +end; Novel (20 documents, ~2,500 chunks) ~4 h. ## Counts and caveats From 25d1b063dbb7f1761503db3c67a8fd7f7dee4bdc Mon Sep 17 00:00:00 2001 From: Gal Shubeli Date: Sun, 9 Aug 2026 16:26:47 +0300 Subject: [PATCH 5/6] docs: keep the MS-GraphRAG and LightRAG comparison, fix stale per-level scores MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Restores the original comparison set in the package README and adds AutoPrunedRetriever alongside it rather than in place of it. Also corrects the per-category breakdown, which still held the previous run's figures while the headline had been updated — they averaged to 62.62, not the 66.09 stated above them. Adds the Medical result, which the table omitted. --- graphrag_sdk/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/graphrag_sdk/README.md b/graphrag_sdk/README.md index 81c4504d..869b24c2 100644 --- a/graphrag_sdk/README.md +++ b/graphrag_sdk/README.md @@ -152,16 +152,17 @@ Every algorithmic concern is a swappable strategy behind an abstract base class: ## Benchmark -**#1 on [GraphRAG-Bench](https://graphrag-bench.github.io)** — 66.09 ACC on Novel and 76.87 on Medical, ahead of AutoPrunedRetriever (63.72 / 67.00) and G-Reasoner (58.94 / 73.30). +**#1 on [GraphRAG-Bench](https://graphrag-bench.github.io) Novel** — 66.09 ACC, ahead of AutoPrunedRetriever (63.72), MS-GraphRAG (50.93) and LightRAG (45.09). | Metric | Value | |--------|-------| | **Novel ACC** | 66.09 (#1) | -| **Fact retrieval** | 65.22 | -| **Complex reasoning** | 58.63 | -| **Contextual summarization** | 69.54 | -| **Creative generation** | 57.08 | +| **Fact retrieval** | 65.49 | +| **Complex reasoning** | 59.26 | +| **Contextual summarization** | 75.42 | +| **Creative generation** | 64.21 | | **Questions** | 2,010 across 20 novels | +| **Medical ACC** | 76.87 (#1) | See [docs/benchmark.md](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/docs/benchmark.md) for methodology and reproduction. @@ -171,7 +172,7 @@ See [docs/benchmark.md](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/docs/ |---|---------|-------------| | 1 | [`01_quickstart.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/01_quickstart.py) | Minimal ingest & query | | 2 | [`02_pdf_with_schema.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/02_pdf_with_schema.py) | PDF with custom schema | -| 3 | [`03_custom_strategies.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/03_custom_strategies.py) | Benchmark-winning pipeline | +| 3 | [`03_custom_strategies.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/03_custom_strategies.py) | Composing strategies explicitly | | 4 | [`04_custom_provider.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/04_custom_provider.py) | Custom LLM/Embedder | | 5 | [`05_notebook_demo.ipynb`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/05_notebook_demo.ipynb) | Interactive notebook walkthrough | From 47362cdf72dac513e6c9928f56d7fa1282b41f4a Mon Sep 17 00:00:00 2001 From: Gal Shubeli Date: Sun, 9 Aug 2026 16:33:12 +0300 Subject: [PATCH 6/6] docs: correct the custom-strategies description and competitor casing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The example composes ingestion strategies only — it never passes retrieval_strategy to GraphRAG, so retrieval runs on the default. The docstring and both README entries said otherwise. Standardizes on 'G-reasoner', the spelling the GraphRAG-Bench leaderboard uses, so the same system does not appear under two names across pages. --- README.md | 4 ++-- graphrag_sdk/README.md | 2 +- graphrag_sdk/examples/03_custom_strategies.py | 6 +++--- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 69a8924d..613b3c58 100644 --- a/README.md +++ b/README.md @@ -24,7 +24,7 @@ Most GraphRAG systems work in demos and break under production constraints. Grap | Rank | System | Novel (Multi-Doc) | Medical (Single-Doc) | Overall | | :--- | :--- | :---: | :---: | :---: | | **1** | **FalkorDB GraphRAG SDK ◄** | **66.09** | **76.87** | **71.48** | -| 2 | G-Reasoner | 58.94 | 73.30 | 66.12 | +| 2 | G-reasoner | 58.94 | 73.30 | 66.12 | | 3 | AutoPrunedRetriever | 63.72 | 67.00 | 65.36 | | 4 | HippoRAG2 | 56.48 | 64.85 | 60.67 | | 5 | Fast-GraphRAG | 52.02 | 64.12 | 58.07 | @@ -219,7 +219,7 @@ guards the default. |---|---------|-------------------| | 1 | [Quick Start](graphrag_sdk/examples/01_quickstart.py) | Your first ingest-and-query loop in under 30 lines | | 2 | [PDF with Schema](graphrag_sdk/examples/02_pdf_with_schema.py) | A PDF Q&A bot with your own entity and relation types | -| 3 | [Custom Strategies](graphrag_sdk/examples/03_custom_strategies.py) | Composing ingestion and retrieval strategies explicitly | +| 3 | [Custom Strategies](graphrag_sdk/examples/03_custom_strategies.py) | Composing ingestion strategies explicitly | | 4 | [Custom Provider](graphrag_sdk/examples/04_custom_provider.py) | Plug in any LLM or embedder behind a clean interface | | 5 | [Notebook Demo](graphrag_sdk/examples/05_notebook_demo.ipynb) | An interactive walkthrough that shows the provenance trail | | 7 | [Incremental Updates](graphrag_sdk/examples/07_incremental_updates.py) | `update`, `delete_document`, and `apply_changes` for CI-driven graph syncs | diff --git a/graphrag_sdk/README.md b/graphrag_sdk/README.md index 869b24c2..f23b5e6a 100644 --- a/graphrag_sdk/README.md +++ b/graphrag_sdk/README.md @@ -172,7 +172,7 @@ See [docs/benchmark.md](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/docs/ |---|---------|-------------| | 1 | [`01_quickstart.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/01_quickstart.py) | Minimal ingest & query | | 2 | [`02_pdf_with_schema.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/02_pdf_with_schema.py) | PDF with custom schema | -| 3 | [`03_custom_strategies.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/03_custom_strategies.py) | Composing strategies explicitly | +| 3 | [`03_custom_strategies.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/03_custom_strategies.py) | Composing ingestion strategies explicitly | | 4 | [`04_custom_provider.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/04_custom_provider.py) | Custom LLM/Embedder | | 5 | [`05_notebook_demo.ipynb`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/05_notebook_demo.ipynb) | Interactive notebook walkthrough | diff --git a/graphrag_sdk/examples/03_custom_strategies.py b/graphrag_sdk/examples/03_custom_strategies.py index bee340af..274b2261 100644 --- a/graphrag_sdk/examples/03_custom_strategies.py +++ b/graphrag_sdk/examples/03_custom_strategies.py @@ -1,12 +1,12 @@ """ GraphRAG SDK -- Custom Strategies ================================================================== -Demonstrates composing the ingestion and retrieval strategies explicitly -rather than relying on the defaults. Uses: +Demonstrates composing the ingestion strategies explicitly rather than relying +on the defaults. Retrieval is left at the default. Uses: - GraphExtraction (GLiNER2 NER + LLM relationship extraction) - DescriptionMergeResolution (LLM-assisted entity dedup) - Post-ingestion finalize (dedup, embeddings, indexes) - - MultiPathRetrieval (default, configured automatically) + - MultiPathRetrieval (the default; not passed explicitly) Prerequisites: pip install graphrag-sdk[litellm]