Software Engineering
New benchmark tool tackles automated performance optimization in Java codebases
Researchers introduce JETO-Bench, a configurable framework for identifying and validating execution time improvements in real-world Java projects, filling a gap in performance-testing benchmarks.
Automated performance optimization has emerged as a key research area, yet most existing benchmarks for testing execution time improvement patches focus on Python, C++, or .NET languages. A new framework now brings this capability to Java, addressing unique challenges posed by the language's runtime environment.
The Challenge with Java Performance Testing#
Java's just-in-time (JIT) compilation and garbage collection mechanisms create significant challenges for consistent execution measurements. Unlike more straightforward performance analysis in other languages, Java execution times can vary considerably across repeated runs. This volatility has made it difficult to build reliable benchmarks for performance patches without specialized tooling designed for Java's specific runtime characteristics.
Introducing JETO-Mine and JETO-Bench#
Researchers have developed JETO-Mine, a configurable tool for automatically constructing reproducible benchmarks of execution time improvement patches in real Java applications. The system operates through a three-phase pipeline:
- Static analysis scans GitHub repositories using customizable filters and AI-powered issue classification to identify candidate patches
- Dynamic analysis isolates patches in Docker containers and applies statistical testing to confirm genuine performance gains
- Evaluation harness enables structured comparison of both patch and test generation tools
Using this pipeline, the team built JETO-Bench, a collection of 660 identified patches and 91 manually verified executable improvements drawn from 174 Java repositories spanning nearly 1.8 million commits over 11 years.
Early Results and Research Opportunities#
Testing OpenHands with GPT-4-mini on the benchmark shows the tool successfully fixes approximately 14.3% of issues—results consistent with performance on other programming languages. The research also uncovered a notable gap: most open-source Java projects lack tests that explicitly demonstrate execution time improvements, highlighting an opportunity for future work in automated test generation tailored to performance scenarios.
Both JETO-Mine and JETO-Bench are publicly available for the research community.