docs: update GraphRAG-Bench results and correct averaging method - #291
Merged
Conversation
Replaces the published GraphRAG-Bench figures with a full run across both subsets (August 2026): Novel 66.09, Medical 76.87, Overall 71.48. The previous Novel figure of 63.73 was a question-weighted average of the four task categories, while the leaderboard — and every competitor number shown beside it — uses the unweighted mean. Computed consistently, the old run scored 62.62 on Novel, so the comparison was not like for like. Both formulas are now stated explicitly under the table. Configuration also changes: gpt-4o-mini at temperature 0.7 for construction and generation, text-embedding-3-large at 1024 dimensions, text-to-Cypher enabled, scored with the benchmark's own generation_eval.py unmodified. docs/benchmark.md now covers both subsets, all fifteen leaderboard systems, the full configuration, reproduction steps, and the two settings raised above the SDK defaults. Also corrects claims that attributed 84.8% — an internal 100-question ablation scored 0-10 by an LLM judge, documented in docs/retrieval.md — to the 20-document novel benchmark. The example carrying that claim uses GLiNER2 NER, DescriptionMergeResolution and FixedSizeChunking, none of which produced the benchmark results. CHANGELOG.md is left untouched; it records what was claimed at the time.
📝 WalkthroughWalkthroughThe pull request updates GraphRAG-Bench results, methodology, configuration, ontologies, reproduction instructions, and evaluation details. It also revises strategy documentation and examples to describe custom strategy composition. ChangesBenchmark documentation
Strategy guidance and examples
Estimated code review effort: 2 (Simple) | ~15 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refreshes the repository’s published GraphRAG-Bench results to an August 2026 full run across both Novel and Medical subsets, and clarifies/aligns the averaging method with the benchmark leaderboard convention. It also removes (or softens) previously misattributed “benchmark-winning” claims from docs and examples to avoid conflating internal ablations with GraphRAG-Bench results.
Changes:
- Update README/docs to new GraphRAG-Bench scores (Novel 66.09, Medical 76.87, Overall 71.48) and explicitly document the unweighted per-category averaging formula.
- Rewrite
docs/benchmark.mdto include both subsets, all competitor rows, configuration, reproduction notes, and caveats. - Remove/adjust “benchmark-winning pipeline” framing in strategy docs and the custom strategies example.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| README.md | Updates headline benchmark table and documents the averaging formulas + configuration footnote; adjusts example description wording. |
| graphrag_sdk/README.md | Updates benchmark claim and Novel ACC number in the package README. |
| docs/benchmark.md | Replaces prior benchmarking guide with a results-focused GraphRAG-Bench report, reproduction steps, and caveats. |
| docs/index.md | Updates the docs landing page benchmark highlight to reflect both subsets. |
| docs/strategies.md | Removes “benchmark-winning” wording and links MultiPathRetrieval claim to the benchmark page. |
| docs/ingestion.md | Removes “benchmark-winning pipeline” wording from DescriptionMergeResolution section. |
| graphrag_sdk/examples/03_custom_strategies.py | Updates module docstring and runtime messaging to remove “benchmark-winning/84.8%” claim. |
Suppressed comments (1)
graphrag_sdk/README.md:163
- The per-category Novel breakdown is still the old run (it averages to 62.62), but
Novel ACCwas updated to 66.09. This makes the table internally inconsistent; update the four category scores to match the new run (as in docs/benchmark.md).
| **Novel ACC** | 66.09 (#1) |
| **Fact retrieval** | 65.22 |
| **Complex reasoning** | 58.63 |
| **Contextual summarization** | 69.54 |
| **Creative generation** | 57.08 |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/benchmark.md`:
- Around line 7-9: Replace the <REPO_URL> placeholder in the benchmark
reproduction instructions with the actual public repository URL or a valid
checked-in relative link, ensuring readers can access the runner, ontologies,
and pinned dependencies.
- Around line 13-14: Update the benchmark summaries to document both Novel
averaging methods: the question-weighted calculation yielding 63.73 and the
prior unweighted four-category mean yielding 62.62, explicitly identifying which
value drives the published table. Apply this clarification in docs/benchmark.md
lines 13-14 and repeat the same distinction in README.md lines 36-41.
- Line 104: Update the FalkorDB image reference in the benchmark Docker command
to replace the mutable latest tag with the exact tested tag and immutable digest
used for the reported August 2026 scores.
In `@graphrag_sdk/examples/03_custom_strategies.py`:
- Around line 2-5: Update the description in
graphrag_sdk/examples/03_custom_strategies.py lines 2-5 to state that the
example composes only the ingestion strategy, unless the runtime configuration
is changed to pass a custom retrieval_strategy to GraphRAG. Apply the same
corrected description in README.md line 222, keeping both descriptions
consistent.
In `@graphrag_sdk/README.md`:
- Around line 155-159: Update the Novel benchmark table adjacent to the headline
in README.md to use the current category values documented in docs/benchmark.md,
ensuring its reported average matches the headline’s 66.09 result. Preserve the
table structure and other benchmark categories unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 94465d87-e7ec-4264-9702-a3bce4b0a273
📒 Files selected for processing (7)
README.mddocs/benchmark.mddocs/index.mddocs/ingestion.mddocs/strategies.mdgraphrag_sdk/README.mdgraphrag_sdk/examples/03_custom_strategies.py
added 2 commits
August 9, 2026 16:06
The runner package is provided on request instead of published, so the benchmark page no longer points at a repository URL. The configuration and the full reproduction procedure stay documented inline, including where the package has to sit inside a GraphRAG-Benchmark checkout for the paths in the commands to resolve.
Adds both hand-authored ontologies used for the GraphRAG-Bench run. They are the artifact behind the corpus-only claim, so publishing them lets that claim be checked without needing the runner: the files contain domain type definitions only, with no question, answer or evidence text. Requests for the runner go through Issues or Discord rather than a personal address, matching the channels used elsewhere in the repo.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (2)
docs/benchmark/ontology_medical.json (2)
102-120: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winRemove the overlapping
CAUSESpatterns.
CAUSESallows["RiskFactor", "Disease"]at Lines 106-109.RISK_FACTOR_FORallows the same pair at Lines 191-196.CAUSESalso allows["Disease", "Symptom"]at Lines 114-117, which is the inverse ofSYMPTOM_OFat Lines 139-144.The extractor can pick either label for the same fact. Edge labels then vary between runs, and traversals that filter on one label miss the other. This reduces the reproducibility of the published benchmark result.
Keep one label per semantic pair.
♻️ Proposed narrowing of `CAUSES`
{ "label": "CAUSES", - "description": "A risk factor, pathogen, or disease causes a disease or symptom.", + "description": "A pathogen causes a disease.", "patterns": [ - [ - "RiskFactor", - "Disease" - ], [ "Pathogen", "Disease" - ], - [ - "Disease", - "Symptom" ] ], "properties": [] },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/benchmark/ontology_medical.json` around lines 102 - 120, Update the CAUSES ontology entry by removing the overlapping RiskFactor–Disease and Disease–Symptom patterns, leaving only non-overlapping patterns so each semantic pair has a single label.
1-2: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winAdd loader coverage for the published ontology JSON files.
These files match the SDK
Ontology.from_fileshape, so the immediate schema-format concern is resolved. Add a regression test that parses both files throughOntology.from_fileto keep the documented benchmark artifacts valid as the model changes.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/benchmark/ontology_medical.json` around lines 1 - 2, Both published ontology JSON files need regression coverage through the SDK loader. Add a test that calls Ontology.from_file for docs/benchmark/ontology_medical.json (lines 1-2) and docs/benchmark/ontology_novel.json (lines 1-2), asserting both parse successfully and remain compatible with the documented model shape.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/benchmark/ontology_medical.json`:
- Around line 262-272: Update the ADMINISTERED_TO ontology entry so its patterns
match the description by adding a Drug-to-AnatomicalStructure pattern alongside
the existing Treatment pattern. Preserve the current Treatment pattern and
description.
---
Nitpick comments:
In `@docs/benchmark/ontology_medical.json`:
- Around line 102-120: Update the CAUSES ontology entry by removing the
overlapping RiskFactor–Disease and Disease–Symptom patterns, leaving only
non-overlapping patterns so each semantic pair has a single label.
- Around line 1-2: Both published ontology JSON files need regression coverage
through the SDK loader. Add a test that calls Ontology.from_file for
docs/benchmark/ontology_medical.json (lines 1-2) and
docs/benchmark/ontology_novel.json (lines 1-2), asserting both parse
successfully and remain compatible with the documented model shape.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 865980fc-41a9-4fcc-86c4-2148fd3da25f
📒 Files selected for processing (3)
docs/benchmark.mddocs/benchmark/ontology_medical.jsondocs/benchmark/ontology_novel.json
🚧 Files skipped from review as they are similar to previous changes (1)
- docs/benchmark.md
…ands The reproduction commands referenced a package the reader cannot obtain, and the two-virtualenv note is an install detail that only matters once someone has it. Replaced with what the run actually does per stage, what it produces, and what it costs — which is the part that is meaningful without the package.
…el scores Restores the original comparison set in the package README and adds AutoPrunedRetriever alongside it rather than in place of it. Also corrects the per-category breakdown, which still held the previous run's figures while the headline had been updated — they averaged to 62.62, not the 66.09 stated above them. Adds the Medical result, which the table omitted.
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.
Suppressed comments (3)
docs/benchmark.md:47
- The leaderboard comparison table uses method names that are inconsistent with the rest of the repo (e.g., README.md uses “AutoPrunedRetriever” and “G-Reasoner”). This makes it harder to cross-check rows against the published leaderboard and other docs.
| AutoPrunedRetriever-llm | 63.72 | 67.00 |
| G-reasoner | 58.94 | 73.30 |
graphrag_sdk/README.md:163
- The per-category Novel metrics are still the old run’s values (they don’t average to 66.09). They should be updated to match the new August 2026 Novel breakdown (Fact 65.49, Complex 59.26, Contextual 75.42, Creative 64.21), otherwise this section contradicts docs/benchmark.md.
| **Novel ACC** | 66.09 (#1) |
| **Fact retrieval** | 65.49 |
| **Complex reasoning** | 59.26 |
| **Contextual summarization** | 75.42 |
| **Creative generation** | 64.21 |
graphrag_sdk/examples/03_custom_strategies.py:7
- This docstring says GraphExtraction uses “GLiNER2 NER”, but the example actually uses
GraphExtraction(llm=llm)which defaults toGLiNERExtractor(modelurchade/gliner_medium-v2.1). Consider renaming this to “GLiNER NER” (or explicitly naming the default model) to avoid implying a different extractor/version than what runs.
Demonstrates composing the ingestion and retrieval strategies explicitly
rather than relying on the defaults. Uses:
- GraphExtraction (GLiNER2 NER + LLM relationship extraction)
- DescriptionMergeResolution (LLM-assisted entity dedup)
The example composes ingestion strategies only — it never passes retrieval_strategy to GraphRAG, so retrieval runs on the default. The docstring and both README entries said otherwise. Standardizes on 'G-reasoner', the spelling the GraphRAG-Bench leaderboard uses, so the same system does not appear under two names across pages.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Updates the published GraphRAG-Bench figures to a full August 2026 run across both subsets, and fixes an averaging inconsistency in the previous numbers.
Results
Why the old Novel figure was not comparable
The previous 63.73 was the question-weighted average of the four task categories. The GraphRAG-Bench leaderboard — and therefore every competitor number printed beside it — uses the unweighted mean of the four.
Computed the leaderboard's way, that same run scored 62.62 on Novel, not 63.73. This mattered for the headline claim:
The new run leads on both subsets under the leaderboard's own formula, so no composite is needed to make the claim. Both equations are now stated explicitly under the README table.
Verified: every competitor's Overall in the table reproduces exactly from
(Novel + Medical) / 2. Only our own row used a different method.Configuration behind the new numbers
gpt-4o-mini(Azure OpenAI) at temperature 0.7 for both graph construction and generation, GraphRAG-SDK 1.3.0 on FalkorDB,text-embedding-3-largeat 1024 dimensions, text-to-Cypher enabled, scored with the benchmark's ownEvaluation/generation_eval.pyunmodified.Two settings sit above the SDK defaults and are declared as such in
docs/benchmark.md:enable_cypher=True(defaultFalse) andembedding_dimension=1024(default256). Both are single global values chosen before any results were scored and applied identically to both subsets.Ontologies are published
Both hand-authored ontologies are checked in at
docs/benchmark/. They are the artifact behind the corpus-only claim, so publishing them lets that claim be verified without the runner — the files contain domain type definitions only, with no question, answer or evidence text.The reproduction package itself (runner, ontology builders, pinned dependencies) is available on request via Issues or Discord rather than published.
Corrected claims
Several places attributed 84.8% to "the 20-document novel benchmark." That figure is from
docs/retrieval.md— an internal ablation over 100 questions scored 0–10 by an LLM judge, a different corpus, scale and metric. The example carrying the claim (03_custom_strategies.py) also uses GLiNER2 NER,DescriptionMergeResolutionandFixedSizeChunking, none of which produced the benchmark results. The number and the "benchmark-winning" framing are removed rather than replaced, since no GraphRAG-Bench run used that configuration.That example also only composes ingestion strategies — it never passes
retrieval_strategy— so its description is corrected accordingly in the docstring and both READMEs.MultiPathRetrieval's claim was accurate and is kept, now linked todocs/benchmark.md.Files
README.md— table, both formulas and the full configuration in the footnotedocs/benchmark.md— rewritten: both subsets, all 15 leaderboard systems, configuration, what the run does per stage, caveatsdocs/benchmark/ontology_{medical,novel}.json— new, the ontologies usedgraphrag_sdk/README.md— PyPI long description; carried its own stale copy of 63.73, and a per-category table that still averaged to 62.62 under a 66.09 headlinedocs/index.md— same stale claimdocs/strategies.md,docs/ingestion.md,graphrag_sdk/examples/03_custom_strategies.py— misattributed claimsCHANGELOG.mdis deliberately untouched; it records what was claimed at the time.Known caveats, documented in
docs/benchmark.mdNovel-55f0c0e2) with contradictory ground truths under two differentsourcevalues. Reported to the benchmark authors.Review notes
Two bot comments are declined with replies in-thread: documenting the old averaging method in the product docs (it belongs with the change, and is above), and a description/pattern mismatch in
ADMINISTERED_TO. The latter is a correct finding, but that file is the artifact that produced the Medical result and must stay byte-identical to what ran — logged for the next run.Summary by CodeRabbit