Add gx results

[joelachance]

Adds two entries for gx, an AI code review CLI.

Submission requirements

• Fork all 50 benchmark PRs into a GitHub organization — satoricorp-benchmarks, one repo per PR per entry, under the naming convention step1_download_prs.py parses.
• Let gx review every PR — 100 PRs, 383 review comments.
• Add gx to the download configuration — both entries added to _NON_BOT_TOOLS, since gx posts under a user account rather than a GitHub App.
• Generate dashboard results — candidates, dedup groups, evaluations for all three judges, plus regenerated dashboard JSON and HTML.

Entries

Two entries, following the existing convention for multiple configurations (qodo-v2 / qodo-extended-v2, greptile-v4-1 / greptile-v5):

entry	configuration
gx	full-coverage configuration
gx-pro	higher-precision configuration

Every finding is verified before it is reported.

Results — all three judges

Core profile, the dashboard default:

judge	entry	precision	recall	F1
sonnet-4.5	gx	38.6%	57.0%	46.0
sonnet-4.5	gx-pro	47.7%	46.8%	47.3
opus-4.5	gx	41.2%	57.6%	48.0
opus-4.5	gx-pro	49.7%	44.9%	47.2
gpt-5.2	gx	34.3%	55.1%	42.2
gpt-5.2	gx-pro	43.6%	45.6%	44.6

On the 66 Critical and High severity goldens, gx finds 43 and gx-pro 38.

Reproducible from the forks

benchmark_data.json can be regenerated rather than taken on trust:

uv run python -m code_review_benchmark.step1_download_prs \
    --org satoricorp-benchmarks --tool gx

Repos are named as the parser expects, with each entry's reviews on PR #1:

keycloak__keycloak__gx__PR37429__20260823
cal_dot_com__cal.com__gx-pro__PR8087__20260823

Verified: all 100 repo names round-trip through parse_repo_name → find_golden_url to the correct golden entry, and live comment counts match the submitted data exactly (383/383, zero mismatches).

No repository-specific configuration was used: none of the five benchmark repos carries a REVIEW.md, so every review reflects gx's defaults.

One bug found along the way

find_golden_url matches by substring:

if f"/{original_repo}/pull/{pr_number}" in url:

/discourse-graphite/pull/1 is a substring of /discourse-graphite/pull/10, so a repo named …__PR1__… can resolve to PR 10's golden set. It is correct today only because pull/1 precedes pull/10 in the golden file; reordering that file silently mis-scores. An anchored match would fix it. Not touched here — happy to send it separately.

What's in the diff

Same file set as #20, the maintainer's own tool-addition PR, plus the _NON_BOT_TOOLS entry:

• offline/README.md — one vendor row
• offline/code_review_benchmark/step1_download_prs.py — _NON_BOT_TOOLS
• offline/analysis/benchmark_dashboard.py — TOOL_COLORS and TOOL_DISPLAY_NAMES
• offline/analysis/benchmark_dashboard.{html,json} — regenerated
• offline/results/benchmark_data.json and <judge>/{candidates,dedup_groups,evaluations}.json for all three judges

Both entries went through step 2.5 dedup.

Integrity

• Stripping gx and gx-pro from benchmark_data.json, candidates.json, and evaluations.json reproduces origin/main exactly, for all three judges — verified programmatically.
• Regenerating the dashboard changes no existing tool's metrics. Remaining churn in benchmark_dashboard.json is key ordering, which the generator does not preserve.
• JSON files are written in each file's own upstream serialization style, so benchmark_data.json is an additions-only diff.
• ruff check clean, pytest tests/ 28 passed.

[ashleyzhang01]

Hey @joelachance, thanks for submitting GX.
Before we re-run it on our end to validate the results, I wanted to flag that we gate offline inclusion on some level of
online activity so we can ensure it reflects real usage. To measure that, we need to be able to tell which reviews on a public PR came from the tool. For other tools, it's a bot account we can filter on for PR reviews. Is there a way to identify a GX-authored review today?

[joelachance]

Hey @ashleyzhang01, thanks for the response! If I understand, you need to see a GX review on an open source project, correct?
I'm working on getting some oss projects up and running, I can provide a link soon if that's the ask.

Thanks again, lmk!

[ashleyzhang01]

@joelachance On usage, we're looking for somewhere in the range of at least 600-1,000 reviewed public PRs across a good spread of orgs and repos before we add a tool, so scores aren't dominated by one codebase and we can use online results to validate offline.
Separately, there's an attribution question. We need a programmatic way to tell that a review came from GX rather than from a human reviewer, and right now I'm unsure what we can key on.

[joelachance]

Ok, I won't have 600+ public PRs for a while.
It sounds like you'd need to id the github user for the bot, that's the one way I'd know how to do that.

any other benchmarks I can use? this is a main marketing lever for me. i get you want to keep your bench relevant

[joelachance]

respectfully, i'd suggest adding this requirement to the README @ashleyzhang01. I spent a couple hundred dollars making sure this was done correctly and ethically, and this is a significant setback, and wish I had known.

Thanks for considering.

[ashleyzhang01]

@joelachance i'm sorry to hear that, we'll definitely add it to the readme. you can still publish your results on your own while referencing our benchmark. the offline/analysis folder has a dashboard you can run locally, where you can compare your results to the other code review tools