Add Corbulo results

[ciprian-cgr]

Adds Corbulo to the offline benchmark.

Corbulo is an AI code review tool. Per offline/README.md, adding a tool requires forking the benchmark PRs and collecting the tool's reviews — this PR contains those reviews (steps 0 and 1). No judged outputs are included, so the results can be produced with whichever judge you prefer.

What's here

• 50 review entries in offline/results/benchmark_data.json — 251 comments (201 inline, 50 review bodies)
• One row in the evaluated-tools table

How the reviews were produced

The 50 benchmark PRs were forked with step0_fork_prs into corbulo-martian-benchmark, where the Corbulo GitHub App is installed. Each PR carries exactly one review, posted by corbulo-core[bot]. Everything is public and can be inspected or re-collected with step1_download_prs --org corbulo-martian-benchmark.

Consistency

Each PR's entry matches its GitHub PR exactly — verified per PR, not just in aggregate: 201 inline comments and 50 review bodies on GitHub, the same 201 and 50 in this file, zero per-PR mismatches. No PR has an empty review.

The diff is purely additive. The only removed line is the file's closing brace; no other tool's data, and no golden comment, is modified.

Note on how the review is shaped

Blocking findings are posted as inline comments on the diff. Findings that can't be anchored to a diff line are carried in the review body, along with a summary table of everything in the review — so a reader sees the full set in one place. Since the same finding can appear both inline and in that table, the dedup pass in step 2.5 is what keeps it from counting twice.

Website: https://corbulo.dev/

[ashleyzhang01]

Hey @ciprian-cgr, thanks for submitting Corbulo.
Before we re-run it on our end to validate the results, there's one thing on our inclusion criteria to flag. We gate offline inclusion on some level of online activity so we can ensure it reflects real usage. Searching the GitHub API for PRs
reviewed by corbulo-core[bot] returns 57 all time, and about 50 of those are the benchmark forks. We're looking for somewhere in the range of 600-1,000 reviewed public PRs across a good spread of orgs and repos before a tool goes in. To be clear, this only looks at public repos, so private installs are invisible to us and can't be used to measure. If we've got this wrong and there's OSS activity we're not picking up, let me know.

[ciprian-cgr]

@ashleyzhang01 thanks for the reply! Wish I had that kind of usage, but Corbulo has been in development for a very long time and it's mostly been used on mine and friends private projects. As you can imagine, I have been analyzing a good number of public projects so far, but only offline and for development purposes. Any suggestions on how to move forward? I'm sure Corbulo is not the first app in this situation.

[ashleyzhang01]

@ciprian-cgr Makes sense, and you're not the first to be in this situation. Unfortunately it wouldn't be fair to make an exception for you. You're welcome to run the benchmark yourself and publish yourself how Corbulo scores. And whenever public volume picks up, ping me and I can run the evaluations.