What a Merge Rate Counts: Coding Agents in the Public Record

One measure, four answers

Merged pull requests are the nearest public measure of whether a coding agent's work was accepted in a real repository. Four sources report that measure for the same agents, and they disagree by up to 58 points. GitHub Copilot's merge rate runs from 38.2% in one study to 96.0% in a public tracker. Codex runs from 65.3% to 90.3%.

The disagreement is not a sign that agents are unreliable or that any source is wrong. It is a sign that "merge rate" names four different calculations, and that a figure quoted without its denominator cannot be compared with anything.

What was measured

All four sources are observational counts of agent-authored pull requests (PRs) on public GitHub. None is a controlled trial, and none draws on private or enterprise repositories. Agents are identified by labels such as branch prefix, bot author or a co-author line in the PR body.

Copilot is last in two studies, fourth of five in a third, and the highest valid figure in the tracker. The ranking of agents changes with the counting method.

Two features of the table deserve attention. First, Li et al. and Ehsani et al. both describe their sample as AIDev-pop, yet one contains 7,122 PRs and the other 33,596. The star thresholds differ (over 500 against over 100), and I could not reconcile the two cohorts from the published text. Second, PRarena reports Cursor at 101.27%, with more merged PRs than ready PRs. The numerator and denominator cannot be on the same basis, and the page does not explain why. The tracker's other figures carry the same uncertainty, and it shows no as-of date.

Why the figures may differ

These are possible causes, and none of the sources tests them. The star threshold and time window differ. The academic studies count closed PRs, while the tracker counts ready PRs against merged ones. Mazloomzadeh et al. report model-adjusted probabilities rather than raw proportions. Li et al. note that their data came two months after the agents' public release and that Cursor and Claude Code are mainly used locally, which leaves only 144 and 101 of their PRs respectively. Mazloomzadeh et al. find the model-adjusted merge probability moving only between 63.7% and 68.2% across four quarters, so time alone does not obviously explain the spread.

What the rejected pull requests show

Ehsani et al. sampled 600 non-merged agent PRs, of which 562 were still accessible and could be categorised. As a share of the 600, 38% were abandoned or never reviewed, and 23% were duplicates of another PR. CI or test failure accounted for 17%, incorrect implementation for 3% and incomplete implementation for 2%. By the paper's grouping, reviewer-level and PR-level causes make up about 69% of non-merges and code-level causes about 22%.

That changes how a low merge rate should be read. A large part of the shortfall in this sample reflects maintainers not engaging, or the agent opening a redundant PR, rather than reviewers judging the code wrong. The study does not say which of those two dominates among the unreviewed PRs, and that remains unknown.

Against human contributors

Only Li et al. report a human baseline drawn from the same repositories: 76.8% accepted, against 38.2% to 65.3% for the agents in their table. That is a gap of 11.5 to 38.6 points, calculated from their table. Median turnaround for accepted PRs was 3.9 hours for humans, 0.3 hours for Codex and 17.2 hours for Copilot.

The picture is not uniformly unfavourable to agents. On documentation PRs, Li et al. report 88.6% accepted for Codex and 85.7% for Claude Code, against 76.5% for humans. A success on one task type deserves the same scrutiny as a failure on another: both are observational, both depend on which maintainers and which repositories are in the sample.

Merge is acceptance, not proof of quality. Mazloomzadeh et al. report that agentic PRs show comparable or lower defect proneness than human PRs, with mostly non-significant differences, but they measured this on merged PRs only, which excludes everything reviewers turned away.

What remains unproven

• No source reports reviewer time, CI cost or token cost per merged PR, so the cost of reaching a merge is unknown.
• All evidence comes from public open-source repositories. No builder-reported or enterprise figures were found, and none of the four rates has been independently replicated.
• Xu et al.⁵, using the same 33,596-PR sample as Ehsani et al., report no merge rate. They find that 19.8% of concurrent PR pairs from the same agent conflict when replayed, and call this a lower bound. Agents that open many PRs at once may add review and integration load that a merge rate does not capture.

What a decision-maker should conclude

A merge rate that cannot name its denominator cannot support a staffing or tooling decision. Before using any such figure, whether from a vendor, a tracker or a paper, ask for five things:

1. What is counted as the denominator: all opened PRs, closed PRs or ready PRs, and how drafts and duplicates are treated.
2. The repository set and time window.
3. Whether human PRs from the same repositories and period are the baseline.
4. Whether the figure is raw or model-adjusted.
5. How non-merged PRs split between abandoned, duplicate, failed CI and rejected on the code.

For a pilot, the more reliable route is to measure it yourself. Record those same categories, plus median time to merge, for agent and human PRs in your own repositories over the same period. Until a figure can be stated with those five items attached, treat it as an illustration rather than a benchmark.