GitHub's ReviewBench crowns Copilot king—but independent benchmark tells a different story

GitHub has introduced ReviewBench, an open-source benchmark for evaluating how effectively AI-powered code-review tools spot problems in pull requests. The announcement arrives as the company continues expanding Copilot's review capabilities, which launched last October and have since evolved to include agentic architecture, approval authority, and integrated billing in GitHub Actions.

Copilot code review claims the top spot on ReviewBench's inaugural leaderboard with a 40.1% grounded F1 score—a metric blending precision and recall. The benchmark analyzed 219 public pull requests across 187 repositories and 19 programming languages, drawn from GitHub's analysis of 103.9 million PRs to ensure representative language and repository distributions.

How the benchmark works#

The evaluation builds a reference set of findings from human review comments, author changes, static-analysis tools, and LLM outputs. Claude Sonnet 5 classifies candidate findings, while a separate LLM matcher determines whether they correspond to genuine issues. GitHub reports 96.6% agreement between human and classifier judgments on true and false positives, with 47 initially misclassified findings manually corrected.

The dataset, methodology, and judging setup are all publicly available, and vendors can submit their own test runs.

A vendor-backed benchmark with caveats#

The leaderboard carries important qualifications: GitHub's own team produced the initial results by running publicly available versions of each competing product. Vendors neither conducted nor verified these tests, and the testing dates span several months—Copilot was evaluated on October 1, while Cubic and Greptile were tested in June. Both products may have evolved since their evaluations, and performance on ReviewBench's corpus may not reflect performance on a company's actual codebase.

A vendor-created benchmark topped by that vendor's product inevitably raises credibility questions. However, GitHub staff argue the benchmark has already validated itself: staff applied engineer Alejandro Carderera de Diego noted that offline ReviewBench results have consistently matched real-world production experiments with Copilot, suggesting the benchmark measures something meaningful.

The independent rival tells a different story#

Meanwhile, Martian, an AI research company positioning itself as a neutral arbiter, launched its own in February. It pairs offline testing with online tracking that measures how developers actually respond to review comments across real open-source projects.