diff --git a/README.md b/README.md index 9fd2ff48..613b3c58 100644 --- a/README.md +++ b/README.md @@ -23,8 +23,8 @@ Most GraphRAG systems work in demos and break under production constraints. Grap ## Benchmarks | Rank | System | Novel (Multi-Doc) | Medical (Single-Doc) | Overall | | :--- | :--- | :---: | :---: | :---: | -| **1** | **FalkorDB GraphRAG SDK ◄** | **63.73** | **75.73** | **69.73** | -| 2 | G-Reasoner | 58.94 | 73.30 | 66.12 | +| **1** | **FalkorDB GraphRAG SDK ◄** | **66.09** | **76.87** | **71.48** | +| 2 | G-reasoner | 58.94 | 73.30 | 66.12 | | 3 | AutoPrunedRetriever | 63.72 | 67.00 | 65.36 | | 4 | HippoRAG2 | 56.48 | 64.85 | 60.67 | | 5 | Fast-GraphRAG | 52.02 | 64.12 | 58.07 | @@ -33,7 +33,22 @@ Most GraphRAG systems work in demos and break under production constraints. Grap | 8 | HippoRAG | 44.75 | 59.08 | 51.92 | | 9 | MS-GraphRAG (local) | 50.93 | 45.16 | 48.05 | -> Overall ACC on [GraphRAG-Bench](https://graphrag-bench.github.io) Novel (20 novels, 2,010 questions) and Medical (1 corpus, 2,062 questions) datasets. FalkorDB scored with `gpt-4o-mini` (Azure OpenAI); competitor numbers are from the published leaderboard. Overall = mean of Novel and Medical ACC. See [docs/benchmark.md](docs/benchmark.md) for per-category breakdowns, methodology, and reproduction instructions. +> **How these are computed.** Per dataset, ACC is the unweighted mean of the four +> task-category scores, matching the [GraphRAG-Bench](https://graphrag-bench.github.io) +> leaderboard convention: +> +> `Dataset ACC = (Fact Retrieval + Complex Reasoning + Contextual Summarize + Creative Generation) / 4` +> `Overall = (Novel ACC + Medical ACC) / 2` +> +> Overall is our own summary across the two datasets; the leaderboard ranks each +> dataset separately. Novel has 20 documents and 2,010 questions, Medical 1 corpus +> and 2,062 questions. FalkorDB scored August 2026 with `gpt-4o-mini` (Azure OpenAI) +> at temperature 0.7 for both graph construction and generation, `text-embedding-3-large` +> at 1024 dimensions, text-to-Cypher retrieval enabled, and the benchmark's own +> `generation_eval.py` unmodified as the judge. Competitor numbers are from the +> published leaderboard, unchanged. See [docs/benchmark.md](docs/benchmark.md) for +> per-category results, the full 15-system comparison, configuration and reproduction +> instructions. Vectors match similar chunks. The graph traverses relationships. Every answer cites its source. @@ -109,7 +124,7 @@ async with GraphRAG(

→ Full walkthrough: Getting Started
- → Benchmark-winning recipe: Custom Strategies + → Compose your own pipeline: Custom Strategies

--- @@ -204,7 +219,7 @@ guards the default. |---|---------|-------------------| | 1 | [Quick Start](graphrag_sdk/examples/01_quickstart.py) | Your first ingest-and-query loop in under 30 lines | | 2 | [PDF with Schema](graphrag_sdk/examples/02_pdf_with_schema.py) | A PDF Q&A bot with your own entity and relation types | -| 3 | [Custom Strategies](graphrag_sdk/examples/03_custom_strategies.py) | The benchmark-winning pipeline, ready to drop in | +| 3 | [Custom Strategies](graphrag_sdk/examples/03_custom_strategies.py) | Composing ingestion strategies explicitly | | 4 | [Custom Provider](graphrag_sdk/examples/04_custom_provider.py) | Plug in any LLM or embedder behind a clean interface | | 5 | [Notebook Demo](graphrag_sdk/examples/05_notebook_demo.ipynb) | An interactive walkthrough that shows the provenance trail | | 7 | [Incremental Updates](graphrag_sdk/examples/07_incremental_updates.py) | `update`, `delete_document`, and `apply_changes` for CI-driven graph syncs | diff --git a/docs/benchmark.md b/docs/benchmark.md index 17ebb58e..2872a573 100644 --- a/docs/benchmark.md +++ b/docs/benchmark.md @@ -1,431 +1,160 @@ -# Benchmarking - -This guide explains how to evaluate the GraphRAG SDK against academic benchmarks and your own datasets. It covers the evaluation methodology, dataset format, step-by-step reproduction with the SDK API, pipeline configuration options, and our published results on the [GraphRAG-Bench](https://graphrag-bench.github.io) Novel leaderboard. - ---- - -## Table of Contents - -1. [Overview](#overview) -2. [Prerequisites](#prerequisites) -3. [Datasets](#datasets) -4. [Reproducing with the SDK API](#reproducing-with-the-sdk-api) -5. [Pipeline Configuration](#pipeline-configuration) -6. [GraphRAG-Bench Novel Results](#graphrag-bench-novel-results) - ---- - -## Overview - -A benchmark run measures four dimensions: - -| Dimension | What it captures | -|-----------|------------------| -| **Accuracy** | Answer quality against ground-truth references (ACC, ROUGE-L, coverage) | -| **Ingestion throughput** | Time to chunk, extract, resolve, and build the knowledge graph | -| **Query latency** | End-to-end time from question submission to final answer | -| **Graph statistics** | Nodes, edges, and chunks produced — a proxy for knowledge density | - -### GraphRAG-Bench scoring system - -We use the official [GraphRAG-Bench](https://graphrag-bench.github.io) evaluation methodology. The primary leaderboard metric is **ACC** (answer correctness × 100), computed as: - -$$\text{ACC} = \bigl(0.75 \times \text{factuality\_F1} + 0.25 \times \text{semantic\_similarity}\bigr) \times 100$$ - -| Component | How it works | -|-----------|-------------| -| **Factuality F1** | An LLM decomposes both the generated answer and the ground truth into atomic statements, classifies each as TP / FP / FN, and computes F1 | -| **Semantic similarity** | Cosine similarity between answer and reference embeddings, scaled to \[0, 1\] | -| **ROUGE-L** | Longest common subsequence F1 — used for Fact Retrieval and Complex Reasoning | -| **Coverage score** | Fraction of reference facts present in the answer — used for Contextual Summarize and Creative Generation | - ---- - -## Prerequisites - -### Infrastructure - -Start a FalkorDB instance: - -```bash -docker run -p 6379:6379 falkordb/falkordb:latest -``` - -### Environment Variables - -Configure your LLM and embedding provider. Example for Azure OpenAI: - -```bash -export AZURE_OPENAI_API_KEY="your-key" -export AZURE_OPENAI_ENDPOINT="https://your-resource.openai.azure.com/" -export AZURE_OPENAI_API_VERSION="2024-12-01-preview" -export AZURE_OPENAI_DEPLOYMENT="gpt-4o-mini" -export AZURE_OPENAI_EMBEDDING_DEPLOYMENT="text-embedding-3-small" -``` - -Any [LiteLLM-supported provider](https://docs.litellm.ai/docs/providers) works — OpenAI, Anthropic, local models, etc. - -### Dependencies - -```bash -pip install graphrag-sdk[litellm] gliner -``` - ---- - -## Datasets - -We evaluate against [GraphRAG-Bench](https://graphrag-bench.github.io), an academic benchmark for graph-based retrieval-augmented generation. The datasets and questions are published in the GraphRAG-Bench project — download them from the official repository and place them in your dataset directory. - -### Available datasets - -| Dataset | Corpus | Questions | Docs | Questions | -|---------|--------|-----------|-----:|----------:| -| Novel (Full) | `novel.json` | `novel_questions.json` | 20 | 2,010 | -| Novel (Sample 100) | `novel.json` | `novel_questions_sample_100.json` | 20 | 100 | -| Medical | `medical.json` | `medical_questions.json` | 1 | 2,062 | - -> **Note:** These files are not included in this repository. Download them from [GraphRAG-Bench](https://graphrag-bench.github.io) and place them in a local directory (e.g., `datasets/`). - -### Data format - -**Corpus** — a JSON array of documents: - -```json -[ - { - "corpus_name": "Novel-30752", - "context": "Full text of the document..." - } -] -``` - -**Questions** — a JSON array of evaluation items: - -```json -[ - { - "id": "Novel-73586ddc", - "source": "Novel-44557", - "question": "Which plant known as Erica vagans is also called...?", - "answer": "Cornish heath", - "question_type": "Fact Retrieval", - "evidence": "The plant known scientifically as Erica vagans...", - "evidence_relations": ["..."] - } -] -``` - -Four question types are evaluated, each with different metrics: - -| Question Type | Metrics Used | -|---------------|-------------| -| Fact Retrieval | ROUGE-L + answer\_correctness (ACC) | -| Complex Reasoning | ROUGE-L + answer\_correctness (ACC) | -| Contextual Summarize | answer\_correctness (ACC) + coverage\_score | -| Creative Generation | answer\_correctness (ACC) + coverage\_score | - -To benchmark your own domain, create two JSON files following the same schema. +# GraphRAG-SDK on GraphRAG-Bench + +Results for [GraphRAG-SDK](https://github.com/FalkorDB/GraphRAG-SDK) on +[GraphRAG-Bench](https://graphrag-bench.github.io/) (Xiang et al., ICLR 2026), +covering both subsets and all four task categories. + +The reproduction package — the runner, the ontology builders and pinned +dependencies — is available on request; open an +[issue](https://github.com/FalkorDB/GraphRAG-SDK/issues) or ask on +[Discord](https://discord.gg/6M4QwDXn2w). The ontologies themselves are +published below, along with the full configuration. + +## Results + +Scores are ACC unless noted. **Average is the unweighted mean of the four ACC +values**, the convention the GraphRAG-Bench leaderboard uses. + +### GraphRAG-Bench (Medical) — Average 76.87 + +| Level | ACC | ROUGE-L | Cov | FS | +| --- | --- | --- | --- | --- | +| Fact Retrieval | 74.50 | 45.43 | — | — | +| Complex Reasoning | 76.47 | 26.55 | — | — | +| Contextual Summarize | 82.29 | — | 62.02 | — | +| Creative Generation | 74.21 | — | 49.72 | 70.75 | + +### GraphRAG-Bench (Novel) — Average 66.09 + +| Level | ACC | ROUGE-L | Cov | FS | +| --- | --- | --- | --- | --- | +| Fact Retrieval | 65.49 | 41.82 | — | — | +| Complex Reasoning | 59.26 | 24.62 | — | — | +| Contextual Summarize | 75.42 | — | 54.11 | — | +| Creative Generation | 64.21 | — | 43.38 | 68.77 | + +### Against the published leaderboard + +Competitor figures are taken from the +[GraphRAG-Bench leaderboard](https://graphrag-bench.github.io/) as published; +we did not re-run them. All entries use `gpt-4o-mini` as the generation +backbone. + +| Method | Novel | Medical | +| --- | --- | --- | +| **GraphRAG-SDK (FalkorDB)** | **66.09** | **76.87** | +| AutoPrunedRetriever-llm | 63.72 | 67.00 | +| G-reasoner | 58.94 | 73.30 | +| HippoRAG2 | 56.48 | 64.85 | +| Fast-GraphRAG | 52.02 | 64.12 | +| MS-GraphRAG (local) | 50.93 | 45.16 | +| Lazy-GraphRAG | 50.59 | 56.89 | +| StructRAG | 49.13 | 58.56 | +| RAG (w/ rerank) | 48.35 | 62.43 | +| KGP | 48.01 | 56.33 | +| RAG (w/o rerank) | 47.93 | 61.00 | +| KET-RAG | 47.62 | 47.05 | +| LightRAG | 45.09 | 62.59 | +| HippoRAG | 44.75 | 59.08 | +| MS-GraphRAG (global) | 44.52 | 28.56 | +| RAPTOR | 43.24 | 57.10 | + +## Configuration + +Per the benchmark's Appendix H.2, GraphRAG-SDK's own defaults are preserved +rather than tuned to the benchmark; the protocol is matched on the backbone +model and the judge. + +| Setting | Value | Source | +| --- | --- | --- | +| Backbone LLM | `gpt-4o-mini` (Azure OpenAI) — graph construction *and* generation | Benchmark protocol | +| Generation temperature | 0.7 | Appendix H.2 | +| Framework | GraphRAG-SDK 1.3.0 (PyPI) on FalkorDB | — | +| Graph layout | one graph per corpus document | — | +| Chunking | `SentenceTokenCapChunking`, max_tokens 512, overlap 2 sentences | SDK default | +| Retrieval | `MultiPathRetrieval` — chunk_top_k 15, rel_top_k 15, max_entities 30, max_relationships 20, keyword_limit 10 | SDK default | +| Embeddings | `text-embedding-3-large` @ 1024 dimensions | Declared below | +| Text-to-Cypher | enabled | Declared below | +| Evaluation | benchmark's `Evaluation/generation_eval.py`, unmodified | Benchmark protocol | +| Judge | `gpt-4o-mini`, `BAAI/bge-large-en-v1.5` embeddings, temperature 0, seed 42 | Benchmark protocol | + +### Declared deviations from SDK defaults + +Two settings were raised above the SDK's own defaults. Both are single global +values, chosen before any results were scored, and applied identically to both +subsets. + +- **`enable_cypher = True`** (SDK default `False`) — ontology-guided + text-to-Cypher retrieval is a core GraphRAG-SDK capability, enabled so the + measurement reflects the framework as deployed. It contributed to 22% of + Medical and 29% of Novel answers. +- **`embedding_dimension = 1024`** (SDK default `256`) — the same vector width + as `bge-large-en-v1.5`, the embedding model Appendix H.2 specifies for + evaluated systems. + +### Benchmark integrity + +The ontologies were hand-authored from the **corpus only**. The question sets +and the `evidence`, `evidence_relations` and `evidence_triple` fields were never +read during ontology design or indexing. + +Both are published here so this is checkable without needing the runner: + +| Subset | Ontology | Entities | Relations | +| --- | --- | ---: | ---: | +| Medical | [`ontology_medical.json`](benchmark/ontology_medical.json) | 10 | 13 | +| Novel | [`ontology_novel.json`](benchmark/ontology_novel.json) | 8 | 13 | + +They contain only domain type definitions — entity labels, their descriptions +and properties, and the relation patterns between them. No question, answer or +evidence text appears in either file. + +## How the run works + +Per subset, the pipeline runs in three stages: + +1. **Index** — each corpus document is chunked, and entities and relationships + are extracted against that subset's ontology into its own FalkorDB graph + (one graph per document, so retrieval is scoped to the document a question + was written against). +2. **Answer** — every question in the subset is answered through + `MultiPathRetrieval`, with the retrieved context recorded alongside the + answer. +3. **Judge** — answers are scored by the benchmark's own + `Evaluation/generation_eval.py`, unmodified, with `gpt-4o-mini` as judge and + `bge-large-en-v1.5` embeddings. + +The run produces, per subset: the leaderboard row, the fully resolved +configuration, per-question predictions with their retrieved context, and each +answer's individual judge scores. + +Indexing cost, measured on a 24-core machine and dominated by extraction API +calls rather than local compute: Medical (1 corpus, 510 chunks) ~1.5 h end to +end; Novel (20 documents, ~2,500 chunks) ~4 h. + +## Counts and caveats + +| | Medical | Novel | +| --- | --- | --- | +| Questions in dataset | 2062 | 2010 | +| Predictions produced | 2062 | 2009 | +| Generation errors / empty answers | 0 / 0 | 0 / 0 | +| Samples scored by the evaluator | 2053 | 1995 | + +Two things worth stating plainly: + +**Novel is 2009 rather than 2010.** The dataset contains a duplicate question +id (`Novel-55f0c0e2`) carrying identical question text under two different +`source` values with contradictory ground truths — one answers the question +asked, the other is unrelated. Our runner answers each id once. This has been +reported to the benchmark authors. + +**The evaluator skips samples whose judge call raises**, so the averages are +over 2053 of 2062 Medical and 1995 of 2009 Novel predictions. Every prediction +itself succeeded; there were no generation failures or empty answers in either +subset. --- -## Reproducing with the SDK API - -The following walkthrough shows how to reproduce our benchmark results from scratch using the SDK's Python API. Following these exact steps with the same configuration and dataset will produce the results shown in the [GraphRAG-Bench Novel Results](#graphrag-bench-novel-results) section. - -### Step 1 — Initialize providers - -```python -import asyncio -import json -from graphrag_sdk import ConnectionConfig, GraphRAG, LiteLLM, LiteLLMEmbedder - -llm = LiteLLM( - model="azure/gpt-4o-mini", - api_key="...", - api_base="https://your-resource.openai.azure.com/", - api_version="2024-12-01-preview", -) - -embedder = LiteLLMEmbedder( - model="azure/text-embedding-3-small", - api_key="...", - api_base="https://your-resource.openai.azure.com/", - api_version="2024-12-01-preview", -) -``` - -### Step 2 — Create a GraphRAG instance - -```python -rag = GraphRAG( - connection=ConnectionConfig(host="localhost", port=6379, graph_name="novel_bench"), - llm=llm, - embedder=embedder, -) -``` - -### Step 3 — Configure the ingestion pipeline - -```python -from graphrag_sdk.core.context import Context -from graphrag_sdk.ingestion.chunking_strategies.sentence_token_cap import SentenceTokenCapChunking -from graphrag_sdk.ingestion.extraction_strategies.graph_extraction import GraphExtraction -from graphrag_sdk.ingestion.extraction_strategies.entity_extractors import GLiNERExtractor -from graphrag_sdk.ingestion.extraction_strategies.coref_resolvers import FastCorefResolver -from graphrag_sdk.ingestion.resolution_strategies.base import ResolutionStrategy -from graphrag_sdk.ingestion.resolution_strategies.exact_match import ExactMatchResolution -from graphrag_sdk.ingestion.resolution_strategies.description_merge import DescriptionMergeResolution -from graphrag_sdk.ingestion.resolution_strategies.semantic_resolution import SemanticResolution -from graphrag_sdk.ingestion.resolution_strategies.llm_verified_resolution import LLMVerifiedResolution - -chunker = SentenceTokenCapChunking(max_tokens=512, overlap_sentences=2) - -extractor = GraphExtraction( - llm=llm, - entity_extractor=GLiNERExtractor(model_name="urchade/gliner_medium-v2.1"), - coref_resolver=FastCorefResolver(), -) -``` - -### Step 4 — Chain resolution stages - -Multiple resolution strategies run in sequence — each stage feeds its output into the next. Create a simple chained resolver: - -```python -from graphrag_sdk.core.models import GraphData, ResolutionResult - -class ChainedResolution(ResolutionStrategy): - """Run multiple resolution strategies in sequence.""" - def __init__(self, *stages): - self._stages = stages - - async def resolve(self, graph_data, ctx): - for stage in self._stages: - result = await stage.resolve(graph_data, ctx) - # Convert ResolutionResult back to GraphData for the next stage - graph_data = GraphData( - nodes=result.nodes, - relationships=result.relationships, - ) - return ResolutionResult( - nodes=graph_data.nodes, - relationships=graph_data.relationships, - ) - -resolver = ChainedResolution( - ExactMatchResolution(resolve_property="name"), - DescriptionMergeResolution(llm=llm), - SemanticResolution(embedder=embedder, similarity_threshold=0.85), - LLMVerifiedResolution(llm=llm, embedder=embedder, hard_threshold=0.95, soft_threshold=0.60), -) -``` - -### Step 5 — Ingest the corpus - -```python -corpus = json.load(open("datasets/novel.json")) - -for doc in corpus: - await rag.ingest( - doc["corpus_name"], - text=doc["context"], - chunker=chunker, - extractor=extractor, - resolver=resolver, - ctx=Context(tenant_id=doc["corpus_name"]), - ) -``` - -### Step 6 — Finalize the graph - -```python -await rag.finalize() -``` - -This removes null/stub entities, deduplicates across documents, embeds all entities and relationships, and creates vector and fulltext indexes in FalkorDB. - -### Step 7 — Query - -```python -questions = json.load(open("datasets/novel_questions.json")) - -results = [] -for q in questions: - result = await rag.completion(q["question"]) - results.append({ - "question": q["question"], - "answer": result.answer, - "reference": q["answer"], - "question_type": q["question_type"], - }) -``` - -### Step 8 — Evaluate with GraphRAG-Bench metrics - -For each question, compute the official [GraphRAG-Bench](https://graphrag-bench.github.io) metrics: - -**Answer Correctness (ACC)** — the primary leaderboard metric: - -1. The LLM decomposes both the generated answer and the ground truth into atomic statements -2. Each statement is classified as TP (true positive), FP (false positive), or FN (false negative) -3. Factuality F1 is computed from the TP / FP / FN counts -4. Semantic similarity is the cosine similarity between answer and reference embeddings, scaled to [0, 1] -5. Final score: `ACC = 0.75 × factuality_F1 + 0.25 × semantic_similarity` - -**ROUGE-L** — longest common subsequence F1 between answer and reference. Applied to Fact Retrieval and Complex Reasoning questions. - -**Coverage Score** — the LLM extracts facts from the reference and checks what fraction is covered in the answer. Applied to Contextual Summarize and Creative Generation questions. - -After running evaluation across all questions, aggregate the results: - -```python -from collections import defaultdict - -by_type = defaultdict(list) -for r in results: - acc = compute_answer_correctness(llm, embedder, r["question"], r["answer"], r["reference"]) - by_type[r["question_type"]].append(acc) - -# Per-type and overall ACC -for q_type, scores in by_type.items(): - avg_acc = sum(scores) / len(scores) * 100 - print(f"{q_type}: ACC = {avg_acc:.2f}") - -all_scores = [s for scores in by_type.values() for s in scores] -overall_acc = sum(all_scores) / len(all_scores) * 100 -print(f"Overall ACC: {overall_acc:.2f}") -``` - -This produces the accuracy tables, graph statistics, and leaderboard comparison shown in the [results section below](#graphrag-bench-novel-results). - ---- - -## Pipeline Configuration - -The ingestion and retrieval pipeline is fully composable. Each stage can be swapped independently. - -### Chunking strategies - -| Strategy | Description | -|----------|-------------| -| `SentenceTokenCapChunking(max_tokens, overlap_sentences)` | Splits on sentence boundaries with a configurable token cap. Best for most use cases. | - -### Extraction strategies - -| Strategy | Description | -|----------|-------------| -| `GraphExtraction(llm, entity_extractor=GLiNERExtractor(), coref_resolver=FastCorefResolver())` | Local NER (no API cost) + coreference resolution + LLM for relationships. Best accuracy. | -| `GraphExtraction(llm)` | LLM-only extraction. Higher API cost per document. | - -### Resolution strategies - -Chain multiple resolvers in sequence using a `ChainedResolution` wrapper (see [Step 4](#step-4--chain-resolution-stages)). Each stage feeds its deduplicated output into the next: - -| Strategy | Description | -|----------|-------------| -| `ExactMatchResolution(resolve_property="name")` | Merges entities with identical names. Zero API cost. | -| `DescriptionMergeResolution(llm)` | LLM merges entities with similar descriptions. | -| `SemanticResolution(embedder, similarity_threshold)` | Cosine similarity on embeddings with hnswlib ANN index. No LLM calls. | -| `LLMVerifiedResolution(llm, embedder, hard_threshold, soft_threshold)` | Two-tier: auto-merge above hard threshold, LLM-verify between soft and hard. Uses Louvain community detection. | - -**Winning chain** (used in our benchmark): - -```python -resolver = ChainedResolution( - ExactMatchResolution(resolve_property="name"), - DescriptionMergeResolution(llm=llm), - SemanticResolution(embedder=embedder, similarity_threshold=0.85), - LLMVerifiedResolution(llm=llm, embedder=embedder, hard_threshold=0.95, soft_threshold=0.60), -) -``` - -### Retrieval strategies - -| Strategy | Description | -|----------|-------------| -| `MultiPathRetrieval` (default) | Multi-path entity discovery, 2-hop graph expansion, chunk retrieval, cosine rerank. No configuration required. | - -### Post-ingestion: `finalize()` - -Always call `await rag.finalize()` after ingesting all documents: - -- Removes null/stub entities -- Deduplicates across document boundaries -- Embeds all entities and relationships -- Creates vector and fulltext indexes in FalkorDB - ---- - -## GraphRAG-Bench Novel Results - -The following results were produced by running the pipeline described above on the complete [GraphRAG-Bench](https://graphrag-bench.github.io) Novel dataset (20 novels, 2,010 questions). - -### Configuration - -| Parameter | Value | -|-----------|-------| -| LLM | gpt-4o-mini (Azure OpenAI) | -| Embeddings | text-embedding-3-small (Azure OpenAI) | -| Chunking | SentenceTokenCapChunking — max\_tokens=512, overlap\_sentences=2 | -| Extraction | GLiNER v2.1 + FastCoref + LLM relationship extraction | -| Resolution | ExactMatch (name) → DescriptionMerge → Semantic (0.85) → LLMVerified (0.95 / 0.60) | -| Retrieval | MultiPathRetrieval | -| Corpus | `novel.json` — 20 novels, 4.7 MB | -| Questions | `novel_questions.json` — 2,010 questions | - -### Accuracy (official GraphRAG-Bench ACC) - -| Question Type | ACC (×100) | ROUGE-L | Coverage | -|---------------|----------:|--------:|---------:| -| Fact Retrieval | 65.22 | 35.95 | — | -| Complex Reasoning | 58.63 | 22.39 | — | -| Contextual Summarize | 69.54 | — | 55.21 | -| Creative Generation | 57.08 | — | 44.52 | -| **Overall** | **63.73** | — | — | - -### Leaderboard comparison - -> **Note:** Only the FalkorDB GraphRAG-SDK row was produced by us using the pipeline described above. All other system scores are taken from the [GraphRAG-Bench published leaderboard](https://graphrag-bench.github.io) as of April 2025. Leaderboard rankings may change as systems are updated and re-evaluated. - -| System | Fact Retrieval | Complex Reasoning | Contextual Summarize | Creative Generation | Overall | -|--------|------:|------:|------:|------:|------:| -| **FalkorDB GraphRAG-SDK** | **65.22** | **58.63** | **69.54** | **57.08** | **63.73** | -| AutoPrunedRetriever | 45.99 | 62.80 | 83.10 | 62.97 | 63.72 | -| G-Reasoner | 60.07 | 53.92 | 71.28 | 50.48 | 58.94 | -| HippoRAG2 | 60.14 | 53.38 | 64.10 | 48.28 | 56.48 | -| Fast-GraphRAG | 56.95 | 48.55 | 56.41 | 46.18 | 52.02 | -| MS-GraphRAG (local) | 49.29 | 50.93 | 64.40 | 39.10 | 50.93 | -| RAG (w/ rerank) | 60.92 | 42.93 | 51.30 | 38.26 | 48.35 | -| LightRAG | 58.62 | 49.07 | 48.85 | 23.80 | 45.09 | -| HippoRAG | 52.93 | 38.52 | 48.70 | 38.85 | 44.75 | - -Source: [graphrag-bench.github.io](https://graphrag-bench.github.io) — Novel leaderboard. - -### Graph statistics - -| Metric | Value | -|--------|------:| -| Total nodes | 8,765 | -| Total edges | 25,895 | -| Total chunks | 2,782 | -| Documents | 20 | - -### Timing - -| Phase | Duration | -|-------|--------:| -| Avg. query latency | 3.6 s | - -### Evaluation methodology - -Scores use the official [GraphRAG-Bench](https://graphrag-bench.github.io) evaluation suite, -ported from `github.com/GraphRAG-Bench/GraphRAG-Benchmark/Evaluation`: - -| Component | How it works | Judge LLM | -|-----------|-------------|-----------| -| **answer_correctness** | 0.75 × Factuality F1 + 0.25 × Semantic Similarity. The LLM decomposes both the answer and reference into atomic statements, classifies TP / FP / FN, and computes F1. Semantic similarity is cosine similarity of answer vs reference embeddings. | gpt-4o-mini | -| **rouge_score** | ROUGE-L F1 (used for Fact Retrieval & Complex Reasoning) | — (algorithmic) | -| **coverage_score** | The LLM extracts facts from the reference and checks which are covered in the answer (used for Contextual Summarize & Creative Generation) | gpt-4o-mini | - -**ACC** reported on the leaderboard = `answer_correctness × 100`, averaged per question type. +*Leaderboard figures as published at graphrag-bench.github.io. Last updated: +August 2026.* diff --git a/docs/benchmark/ontology_medical.json b/docs/benchmark/ontology_medical.json new file mode 100644 index 00000000..02cfb553 --- /dev/null +++ b/docs/benchmark/ontology_medical.json @@ -0,0 +1,274 @@ +{ + "entities": [ + { + "label": "Disease", + "description": "A disease, disorder, cancer, or medical condition.", + "properties": [ + { + "name": "category", + "type": "STRING", + "description": "e.g. carcinoma, sarcoma, infection." + }, + { + "name": "stage", + "type": "STRING", + "description": "Stage or grade if given." + } + ] + }, + { + "label": "Symptom", + "description": "A sign, symptom, or clinical manifestation.", + "properties": [] + }, + { + "label": "Treatment", + "description": "A therapy, procedure, surgery, or intervention (e.g. brachytherapy, radiation, chemotherapy).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "surgery, radiation, therapy, procedure, etc." + } + ] + }, + { + "label": "Drug", + "description": "A medication, drug, or pharmacological agent.", + "properties": [ + { + "name": "drug_class", + "type": "STRING", + "description": "Drug class if stated." + } + ] + }, + { + "label": "AnatomicalStructure", + "description": "An organ, gland, tissue, or body part (e.g. prostate, urethra, lung).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "organ, gland, tissue, cell, etc." + } + ] + }, + { + "label": "DiagnosticTest", + "description": "A diagnostic test, imaging modality, biopsy, or screening method.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "lab test, imaging, biopsy, screening, etc." + } + ] + }, + { + "label": "RiskFactor", + "description": "A risk factor or cause associated with a disease (e.g. smoking, age, exposure).", + "properties": [] + }, + { + "label": "Biomarker", + "description": "A biomarker, protein, hormone, antigen, or measurable substance (e.g. PSA).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "protein, hormone, antigen, gene, etc." + } + ] + }, + { + "label": "Pathogen", + "description": "An infectious agent — virus, bacterium, or other pathogen (e.g. HPV).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "virus, bacterium, etc." + } + ] + }, + { + "label": "BodySystem", + "description": "A body system (e.g. reproductive, respiratory, immune).", + "properties": [] + } + ], + "relations": [ + { + "label": "CAUSES", + "description": "A risk factor, pathogen, or disease causes a disease or symptom.", + "patterns": [ + [ + "RiskFactor", + "Disease" + ], + [ + "Pathogen", + "Disease" + ], + [ + "Disease", + "Symptom" + ] + ], + "properties": [] + }, + { + "label": "TREATS", + "description": "A treatment or drug is used to treat a disease.", + "patterns": [ + [ + "Treatment", + "Disease" + ], + [ + "Drug", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "SYMPTOM_OF", + "description": "A symptom is a manifestation of a disease.", + "patterns": [ + [ + "Symptom", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "DIAGNOSED_BY", + "description": "A disease is diagnosed or detected by a test.", + "patterns": [ + [ + "Disease", + "DiagnosticTest" + ] + ], + "properties": [] + }, + { + "label": "AFFECTS", + "description": "A disease affects an anatomical structure or body system.", + "patterns": [ + [ + "Disease", + "AnatomicalStructure" + ], + [ + "Disease", + "BodySystem" + ] + ], + "properties": [] + }, + { + "label": "PART_OF", + "description": "An anatomical structure is part of a larger structure or body system.", + "patterns": [ + [ + "AnatomicalStructure", + "AnatomicalStructure" + ], + [ + "AnatomicalStructure", + "BodySystem" + ] + ], + "properties": [] + }, + { + "label": "RISK_FACTOR_FOR", + "description": "A risk factor increases the risk of a disease.", + "patterns": [ + [ + "RiskFactor", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "PREVENTS", + "description": "A treatment or drug prevents a disease.", + "patterns": [ + [ + "Treatment", + "Disease" + ], + [ + "Drug", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "PRODUCES", + "description": "An anatomical structure produces a biomarker or substance.", + "patterns": [ + [ + "AnatomicalStructure", + "Biomarker" + ] + ], + "properties": [] + }, + { + "label": "INDICATES", + "description": "A biomarker or test result indicates a disease.", + "patterns": [ + [ + "Biomarker", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "HAS_SUBTYPE", + "description": "A disease has a subtype or variant.", + "patterns": [ + [ + "Disease", + "Disease" + ] + ], + "properties": [] + }, + { + "label": "SIDE_EFFECT_OF", + "description": "A symptom is a side effect of a treatment or drug.", + "patterns": [ + [ + "Symptom", + "Treatment" + ], + [ + "Symptom", + "Drug" + ] + ], + "properties": [] + }, + { + "label": "ADMINISTERED_TO", + "description": "A treatment or drug is administered to an anatomical structure.", + "patterns": [ + [ + "Treatment", + "AnatomicalStructure" + ] + ], + "properties": [] + } + ] +} \ No newline at end of file diff --git a/docs/benchmark/ontology_novel.json b/docs/benchmark/ontology_novel.json new file mode 100644 index 00000000..46b1fe9e --- /dev/null +++ b/docs/benchmark/ontology_novel.json @@ -0,0 +1,267 @@ +{ + "entities": [ + { + "label": "Character", + "description": "A person or personified being — a fictional character, a historical figure, an author, or a real individual named in the text.", + "properties": [ + { + "name": "aliases", + "type": "LIST", + "description": "Other names, titles, or epithets used for this person." + }, + { + "name": "role", + "type": "STRING", + "description": "Their role or occupation (e.g. narrator, doctor, king)." + }, + { + "name": "gender", + "type": "STRING", + "description": "Gender if stated or clearly implied." + } + ] + }, + { + "label": "Location", + "description": "A named place: country, region, city, building, or natural feature.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Type of place: country, city, region, building, natural feature, etc." + } + ] + }, + { + "label": "Organization", + "description": "A named group, institution, society, company, family line, or faction.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Type of organisation: society, company, family, government, etc." + } + ] + }, + { + "label": "Work", + "description": "A named creative or scholarly work: book, poem, play, essay, or publication referenced in the text.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Type of work: novel, poem, play, essay, treatise, etc." + }, + { + "name": "author", + "type": "STRING", + "description": "Named author of the work, if given." + } + ] + }, + { + "label": "Object", + "description": "A notable physical object, artifact, substance, or material referred to in the text.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Category of the object or substance." + } + ] + }, + { + "label": "Event", + "description": "A discrete happening, action, or episode in the narrative (journey, battle, death, discovery, ceremony).", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Type of event: journey, battle, death, discovery, meeting, etc." + }, + { + "name": "time", + "type": "STRING", + "description": "When it occurred, as stated in the text." + } + ] + }, + { + "label": "Concept", + "description": "An abstract idea, theme, belief, scientific principle, or topic discussed in the text.", + "properties": [ + { + "name": "kind", + "type": "STRING", + "description": "Category of the concept: theme, scientific principle, belief, topic, etc." + } + ] + }, + { + "label": "TimePeriod", + "description": "A named point or span of time: a date, year, era, season, or historical period.", + "properties": [] + } + ], + "relations": [ + { + "label": "INTERACTS_WITH", + "description": "One character speaks to, meets, or otherwise interacts with another.", + "patterns": [ + [ + "Character", + "Character" + ] + ], + "properties": [] + }, + { + "label": "FAMILY_OF", + "description": "A family or kinship tie between two characters.", + "patterns": [ + [ + "Character", + "Character" + ] + ], + "properties": [] + }, + { + "label": "MEMBER_OF", + "description": "A character belongs to an organisation, group, or family line.", + "patterns": [ + [ + "Character", + "Organization" + ] + ], + "properties": [] + }, + { + "label": "LOCATED_IN", + "description": "A character, organisation, or event is situated in a location.", + "patterns": [ + [ + "Character", + "Location" + ], + [ + "Organization", + "Location" + ], + [ + "Event", + "Location" + ] + ], + "properties": [] + }, + { + "label": "TRAVELS_TO", + "description": "A character moves or journeys to a location.", + "patterns": [ + [ + "Character", + "Location" + ] + ], + "properties": [] + }, + { + "label": "PART_OF", + "description": "A location or organisation is a part of a larger one.", + "patterns": [ + [ + "Location", + "Location" + ], + [ + "Organization", + "Organization" + ] + ], + "properties": [] + }, + { + "label": "PARTICIPATES_IN", + "description": "A character takes part in an event.", + "patterns": [ + [ + "Character", + "Event" + ] + ], + "properties": [] + }, + { + "label": "OWNS", + "description": "A character owns or possesses an object.", + "patterns": [ + [ + "Character", + "Object" + ] + ], + "properties": [] + }, + { + "label": "AUTHORED", + "description": "A character (author) created a work.", + "patterns": [ + [ + "Character", + "Work" + ] + ], + "properties": [] + }, + { + "label": "ASSOCIATED_WITH", + "description": "A character or work is associated with a concept or theme.", + "patterns": [ + [ + "Character", + "Concept" + ], + [ + "Work", + "Concept" + ] + ], + "properties": [] + }, + { + "label": "CAUSES", + "description": "One event causes or leads to another.", + "patterns": [ + [ + "Event", + "Event" + ] + ], + "properties": [] + }, + { + "label": "PRECEDES", + "description": "One event happens before another in time.", + "patterns": [ + [ + "Event", + "Event" + ] + ], + "properties": [] + }, + { + "label": "OCCURS_DURING", + "description": "An event takes place during a named time period.", + "patterns": [ + [ + "Event", + "TimePeriod" + ] + ], + "properties": [] + } + ] +} \ No newline at end of file diff --git a/docs/index.md b/docs/index.md index f1372f01..ff849fbf 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,7 +6,7 @@ GraphRAG SDK builds knowledge graphs from documents and answers questions over t ## Key Highlights -- **#1 on GraphRAG-Bench Novel** — 63.73 ACC on 2,010 questions ([benchmark](benchmark.md)) +- **#1 on GraphRAG-Bench** — 66.09 ACC on Novel and 76.87 on Medical ([benchmark](benchmark.md)) - **Simple API** -- `ingest()` + `completion()` with sensible defaults - **100+ LLM providers** via LiteLLM (OpenAI, Azure, Anthropic, Cohere, Ollama, and more) - **Fully modular** -- swap chunking, extraction, resolution, retrieval, and reranking strategies diff --git a/docs/ingestion.md b/docs/ingestion.md index 1bb0b6c9..f9d5b680 100644 --- a/docs/ingestion.md +++ b/docs/ingestion.md @@ -165,7 +165,7 @@ For a detailed explanation of the extraction process, see [extraction.md](extrac **Alternative: DescriptionMergeResolution** - Groups by `(normalized name, label)` — same name but different labels stay separate (e.g., Person "Paris" vs Location "Paris") - Merges descriptions (concatenation or LLM summarization) -- Used in the benchmark-winning pipeline +- Useful when the same entity is described differently across documents For details on resolution strategies, see [strategies.md](strategies.md). diff --git a/docs/strategies.md b/docs/strategies.md index 3c3c4c60..c0faf017 100644 --- a/docs/strategies.md +++ b/docs/strategies.md @@ -428,7 +428,7 @@ resolver = DescriptionMergeResolution( ) ``` -**When to use:** Multi-document ingestion where the same entity appears with different descriptions. Used in the benchmark-winning pipeline. +**When to use:** Multi-document ingestion where the same entity appears with different descriptions. Useful when the same entity is described differently across documents. --- @@ -476,7 +476,7 @@ retriever = LocalRetrieval( ### Built-in: MultiPathRetrieval -Production-grade retrieval with RELATES edge vector search, 2-path entity discovery, 4-path chunk retrieval, and cosine reranking. This is the **default** and the benchmark-winning strategy. +Production-grade retrieval with RELATES edge vector search, 2-path entity discovery, 4-path chunk retrieval, and cosine reranking. This is the **default**, and the strategy used for the [GraphRAG-Bench results](benchmark.md). ```python from graphrag_sdk import MultiPathRetrieval diff --git a/graphrag_sdk/README.md b/graphrag_sdk/README.md index babf40ef..f23b5e6a 100644 --- a/graphrag_sdk/README.md +++ b/graphrag_sdk/README.md @@ -152,16 +152,17 @@ Every algorithmic concern is a swappable strategy behind an abstract base class: ## Benchmark -**#1 on [GraphRAG-Bench](https://graphrag-bench.github.io) Novel** — 63.73 ACC, ahead of MS-GraphRAG (50.93) and LightRAG (45.09). +**#1 on [GraphRAG-Bench](https://graphrag-bench.github.io) Novel** — 66.09 ACC, ahead of AutoPrunedRetriever (63.72), MS-GraphRAG (50.93) and LightRAG (45.09). | Metric | Value | |--------|-------| -| **Novel ACC** | 63.73 (#1) | -| **Fact retrieval** | 65.22 | -| **Complex reasoning** | 58.63 | -| **Contextual summarization** | 69.54 | -| **Creative generation** | 57.08 | +| **Novel ACC** | 66.09 (#1) | +| **Fact retrieval** | 65.49 | +| **Complex reasoning** | 59.26 | +| **Contextual summarization** | 75.42 | +| **Creative generation** | 64.21 | | **Questions** | 2,010 across 20 novels | +| **Medical ACC** | 76.87 (#1) | See [docs/benchmark.md](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/docs/benchmark.md) for methodology and reproduction. @@ -171,7 +172,7 @@ See [docs/benchmark.md](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/docs/ |---|---------|-------------| | 1 | [`01_quickstart.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/01_quickstart.py) | Minimal ingest & query | | 2 | [`02_pdf_with_schema.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/02_pdf_with_schema.py) | PDF with custom schema | -| 3 | [`03_custom_strategies.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/03_custom_strategies.py) | Benchmark-winning pipeline | +| 3 | [`03_custom_strategies.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/03_custom_strategies.py) | Composing ingestion strategies explicitly | | 4 | [`04_custom_provider.py`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/04_custom_provider.py) | Custom LLM/Embedder | | 5 | [`05_notebook_demo.ipynb`](https://github.com/FalkorDB/GraphRAG-SDK/blob/main/graphrag_sdk/examples/05_notebook_demo.ipynb) | Interactive notebook walkthrough | diff --git a/graphrag_sdk/examples/03_custom_strategies.py b/graphrag_sdk/examples/03_custom_strategies.py index 3299afa0..274b2261 100644 --- a/graphrag_sdk/examples/03_custom_strategies.py +++ b/graphrag_sdk/examples/03_custom_strategies.py @@ -1,12 +1,12 @@ """ -GraphRAG SDK -- Custom Strategies (Benchmark-Winning Pipeline) +GraphRAG SDK -- Custom Strategies ================================================================== -Demonstrates the full pipeline configuration that achieved 84.8% accuracy -on the 20-document novel benchmark. Uses: +Demonstrates composing the ingestion strategies explicitly rather than relying +on the defaults. Retrieval is left at the default. Uses: - GraphExtraction (GLiNER2 NER + LLM relationship extraction) - DescriptionMergeResolution (LLM-assisted entity dedup) - Post-ingestion finalize (dedup, embeddings, indexes) - - MultiPathRetrieval (default, configured automatically) + - MultiPathRetrieval (the default; not passed explicitly) Prerequisites: pip install graphrag-sdk[litellm] @@ -125,7 +125,7 @@ async def main(): pass # --- Ingestion with custom strategies --- - print("Ingesting documents with benchmark-winning pipeline...") + print("Ingesting documents with custom strategies...") t0 = time.time() for source_id, text in DOCUMENTS: