Open-source benchmark for evaluating AI code review tools — the datasets, the judge, and the pipeline code. Reproduce our results or evaluate your own tool.
As AI agents write more code, we need systems to make sure the code they generate is good. This has led to the proliferation of AI code review tools.
Without shared evals for these tools, every company grades its own homework. You can't reproduce the results, compare tools on the same dataset, or verify the methodology. With static evals, agents can game the benchmark. By creating an online and offline benchmark that can check each other, this project allows for robust code review evals.
We open-source everything: the PRs, the golden comments, the LLM judge prompts, the evaluation pipeline, and a continuously-updated online benchmark that avoids training data leakage.
50 PRs from 5 major open-source projects, each with human-verified golden comments — the real issues a reviewer should catch.
| Repository | Language | Domain |
|---|---|---|
| Sentry | Python | Error tracking |
| Grafana | Go | Observability |
| Cal.com | TypeScript | Scheduling |
| Discourse | Ruby | Forum platform |
| Keycloak | Java | Authentication |
Each PR has curated golden comments (173 total) with severity labels (Low / Medium / High / Critical) and category tags (bug, security, concurrency, data, api, perf, test_gap, doc_defect, style, speculative). An LLM judge matches each tool's review against the golden comments using three judge models (Claude Opus 4.5, GPT-5.2, Claude Sonnet 4.5). Category-based scoring profiles (Strict / Core / All) control which issue types count toward the score, and F-beta weighting lets users prioritize recall over precision.
Tools evaluated: Augment, Baz, Claude Code, CodeAnt, CodeRabbit, Cubic, Cursor Bugbot, Devin, Gemini, GitHub Copilot, GitLab Duo, Graphite, Greptile, KG, Kodus, Macroscope, Qodo, Sourcery, and more. Running a tool that isn't on the leaderboard takes an afternoon — fork the benchmark PRs, trigger the tool, run the pipeline. Publishing it alongside the others additionally requires meeting the inclusion criteria.
Known limitation: Static datasets risk training data leakage — tools may have seen these PRs during training. That's why we also run the online benchmark.
See offline/README.md for setup and usage.
The online benchmark continuously samples fresh real-world PRs from GitHub where code review bots left comments. Because the PRs are recent, tools can't have memorized them during training.
GitHub Search API
│
▼
┌────────┐ ┌─────────┐ ┌─────────┐ ┌────┐ ┌───────────┐
│Discover│────▶│ Enrich │────▶│ Analyze │────▶│ DB │────▶│ Dashboard │
└────────┘ └─────────┘ └─────────┘ └────┘ └───────────┘
Search API GitHub API LLM 3-step Postgres Interactive
finds merged fetches full extraction & or SQLite filters &
bot-reviewed PR context matching time series
PRs
How analysis works:
- Extract bot suggestions — The LLM reads the diff the bot reviewed and the bot's comments, then extracts each actionable suggestion with its category (bug, security, performance, style, ...) and severity.
- Extract human actions — The LLM reads the post-review commits and identifies what the developer actually fixed after the bot commented.
- Judge matching — The LLM determines which bot suggestions correspond to actual fixes, producing per-PR precision (what % of the bot's comments were useful?) and recall (what % of real issues did the bot catch?).
Bots tracked: CodeRabbit, GitHub Copilot, Claude, Cursor, Augment, Codex, Gemini, Greptile, Graphite, Qodo, Propel, and others.
Dashboard features: Filter by language, project domain, PR type, issue severity, diff size, engagement signals (human comments/commits after bot review), solo-bot PRs, and sample controls. Track performance over time. Adjustable F-beta weighting. See online/FILTERS.md for the full filter spec.
See online/README.md for architecture and setup.
Both benchmarks use an LLM-as-judge approach, but with different methodologies suited to their data:
| Offline | Online | |
|---|---|---|
| Ground truth | Human-curated golden comments (173, categorized) | Developer's post-review fixes |
| Precision | Tool comments that match a golden comment / total tool comments | Bot suggestions matched to real fixes / total suggestions |
| Recall | Golden comments in active profile found by the tool / total golden in profile | Real fixes caught by the bot / total fixes made |
| Scoring | Category-based profiles (Strict/Core/All) + F-beta (0.5–3.0) | F-beta with adjustable weighting |
| Judge input | Golden comment + tool candidate | Full PR timeline: diff, bot comments, post-review commits |
In both cases, the judge prompt asks "do these describe the same underlying issue?" — different wording is fine, only the substance matters.
Judge model variance: Different LLM judges can score differently. We mitigate this by storing results per judge model and reporting which model was used. The offline benchmark has been evaluated with Claude Opus 4.5, Claude Sonnet 4.5, and GPT-5.2 — the top 5 tools are identical across all three judges, with most tools varying by at most 2 rank positions.
├── offline/ # Offline benchmark (fixed dataset)
│ ├── golden_comments/ # Human-curated issues per repo (5 JSON files)
│ ├── code_review_benchmark/ # Pipeline: fork, download, extract, dedup, judge, export
│ ├── analysis/ # Interactive HTML dashboard
│ ├── tests/ # Test suite (no network access required)
│ └── results/ # Evaluation outputs (per judge model)
│
├── online/ # Online benchmark (continuous)
│ ├── etl/ # Python pipeline
│ │ ├── pipeline/ # Discover → Enrich → Assemble → Analyze → Label
│ │ ├── llm/ # Prompts, schemas, async client
│ │ ├── db/ # Database layer (SQLite + PostgreSQL)
│ │ ├── jobs/ # Background workers
│ │ └── dashboard/ # Streamlit dashboard
│ └── api_service/ # Rust API + embedded HTML dashboard
│
└── LICENSE # MIT
cd offline
uv sync
cp .env.example .env # add GitHub token + LLM API key
# Download reviews for all tools
uv run python -m code_review_benchmark.step1_download_prs --output results/benchmark_data.json
# Extract individual issues from reviews
uv run python -m code_review_benchmark.step2_extract_comments
# Deduplicate candidates (prevents false positives from inline+summary overlap)
uv run python -m code_review_benchmark.step2_5_dedup_candidates
# Run the LLM judge (pass dedup groups to avoid penalising duplicate candidates)
uv run python -m code_review_benchmark.step3_judge_comments \
--dedup-groups results/${MARTIAN_MODEL}/dedup_groups.json
# View results
open analysis/benchmark_dashboard.htmlcd online/etl
uv sync
cp .env.example .env # add GitHub token + GCP project + LLM API key
# Discover recent PRs from BigQuery
uv run python main.py discover --all --days-back 7
# Enrich with GitHub API data
uv run python main.py enrich --chatbot "coderabbitai[bot]" --one-shot
# Run LLM analysis
uv run python main.py analyze --all
# Launch dashboard
uv run python main.py dashboardRunning a tool that isn't on the leaderboard is open to anyone:
- Fork the 50 benchmark PRs into a GitHub org where your tool is installed
- Let the tool review each PR
- Add the tool name to the download config and run the pipeline
- Compare the results against the existing tools in the dashboard
See offline/README.md for detailed instructions.
Publishing a tool on the leaderboard alongside the others has two further requirements.
Public usage. Roughly at least 600–1,000 reviewed public PRs, across a spread of orgs, repos, and authors. The offline benchmark is a fixed set of 50 PRs, so on its own it can't tell us whether a score reflects how a tool behaves in practice. We validate it against the online benchmark, which measures how developers respond to a tool's reviews in the wild, and that cross-check needs enough public review activity to be meaningful. Private installs aren't visible to us and can't be counted, so this is a measurement constraint rather than a judgement about a tool's overall adoption.
Attributable reviews. We need to be able to tell from the GitHub API that a review came from the tool rather than from a person — a bot account, a dedicated machine account, or a consistent marker in the comment body all work. Without one of those we can't separate a tool's findings from a human reviewer's comments.
For any tool we publish we also run the pipeline ourselves rather than take submitted results, so every number on the leaderboard is produced the same way.
If a tool doesn't meet these yet, the harness is public and you're welcome to run it and publish your own results.
We welcome contributions — new tools, better golden comments, improved judge prompts, additional datasets. Open an issue or PR.
If you use this benchmark in your research or product evaluation, please cite:
@misc{code_review_benchmark,
title = {Code Review Bench},
author = {Aleksandr Zverianskii and Ashley Zhang and Jacob Clyne and Antía Garcia and Fazl Barez and Shriyash Upadhyay},
url = {https://github.com/withmartian/code-review-benchmark},
year = {2026},
license = {MIT}
}MIT — see LICENSE.