Agentic AI
Agentic code review tool CodeRabbit faces skepticism from developers in real-world use
An empirical study of over 31,000 code reviews reveals that AI-assisted code review suggestions are rejected more often than accepted, hampered by false positives and misalignment with developer practices.
Autonomous code review agents are becoming more prevalent in software development workflows, but a new empirical study reveals a sobering picture of their real-world effectiveness. Researchers examining CodeRabbit, a popular AI-driven code review tool, found that developers reject the majority of its suggestions in practice.
The Study's Findings#
The research analyzed data from over 31,000 code review interactions across 10,191 pull requests in 239 GitHub repositories. The results paint a mixed picture: while 36.4% of agentic reviews were accepted by developers, a much larger portion—56.3%—were outright rejected. Only 7.3% of reviews sparked substantive discussion.
Why Developers Are Pushing Back#
The primary culprits behind rejection involve issues with review quality itself. Developers frequently dismissed suggestions that were false positives, redundant, or simply outside the scope of what needed fixing. Another significant factor was a fundamental mismatch between what the AI recommended and what developers actually intended to do, along with suggestions that conflicted with established coding practices within their teams.
Notably, agentic reviews also showed a skew in their focus: they concentrated more on identifying functional problems but were less likely to offer insights on code evolvability—and when they did venture into either domain, their overall accuracy suffered.
Predicting When Reviews Will Fail#
The researchers explored machine learning approaches to forecast which code reviews would be rejected. Using lightweight, learning-based methods, they achieved up to 76% F1 score, indicating that detectable patterns do exist between review content and developer reception. This suggests the problem is not insurmountable.
What's Next#
The study highlights both the potential and the gaps in current agentic code review tools. While autonomous review systems are increasingly popular, their effectiveness remains limited by inaccurate suggestions, misaligned recommendations, and a tendency to miss important context about developer intent. The findings suggest clear opportunities for improvement in how these tools are trained and calibrated to better serve development teams.
