AI
New Benchmark Tests AI Agents on Java Code Modernization Tasks
Researchers release FreshBrew, a systematic evaluation framework for measuring how well AI models handle real-world Java migration challenges, revealing both capabilities and shortcomings of current approaches.
Keeping large codebases up-to-date as software platforms evolve remains a major challenge in development. Researchers have introduced FreshBrew, a new benchmark designed to evaluate how AI agents perform when tackling Java code migrations at the project level—a task traditionally handled through manual work and rule-based automation tools.
The Challenge of Code Modernization#
As technology stacks mature and new platform versions arrive, engineering teams face pressure to migrate aging codebases to stay current. This work is complex, costly, and error-prone, making it a natural candidate for AI assistance. However, evaluating whether AI solutions actually work well in these scenarios requires rigorous testing with clear success criteria. The FreshBrew benchmark addresses this gap by focusing on migrations from Java 8 to Java 17—a realistic, high-stakes scenario that demands both functional correctness and semantic preservation.
How FreshBrew Works#
The benchmark evaluates AI agents using a curated dataset of 228 open-source Maven repositories that run successfully on JDK 8 but fail on JDK 17. Critically, the authors selected projects with high test coverage, ensuring that agent success can be measured not just by whether code changes compile, but whether they maintain the original program's behavior. This design guards against "reward hacking"—where an AI system might appear to solve the problem while actually breaking functionality.
The researchers tested several leading language models against established rule-based migration tools. Results show that Gemini 2.5 Flash achieved the strongest performance, successfully migrating 52.3% of test projects to JDK 17. While significant, this figure also highlights the complexity of the task and the limitations of current AI approaches.
What the Results Reveal#
The empirical analysis uncovers both strengths and failure modes in existing agentic systems. AI agents show promise on certain classes of modernization problems, yet struggle with others—insights that are valuable for developers considering whether to rely on these tools for real migrations. By releasing the benchmark openly, the authors aim to spur progress in trustworthy, automated code modernization while providing the community with a reproducible evaluation standard.
