LLM agents achieve 94% success on automated software analysis, outpacing existing baselines

Researchers have systematically evaluated how well large language model agents can handle the practical challenge of setting up and running software analysis tools on real-world projects. The task is harder than it might seem: it requires not just installing an analysis tool, but also resolving environment dependencies, handling tool-specific configuration, and verifying that meaningful analysis results are actually produced.

The benchmark and its findings#

The team created AnalysisBench, a test suite comprising 35 tool-project pairs covering seven different analysis tools applied to ten C/C++ and Java projects. Each pair includes a manually verified reference setup, allowing researchers to distinguish between agents that merely report success and those that genuinely produce working configurations.

When evaluated across four different LLM backends, a custom agent called AnalysisAgent achieved a manually verified success rate of 94% (33 of 35 tasks) on Gemini-3-Flash. This outperformed the strongest baseline, ExecutionAgent, which reached 77% accuracy. The results underscore a key finding: agentic architecture design proved more influential than the underlying language model alone.

Where existing approaches fall short#

Beyond raw performance numbers, the research pinpointed concrete weaknesses in current agents:

• Stage mixing — agents blur the lines between separate phases of the setup process
• Poor error localization — difficulty pinpointing where and why a task fails
• Premature termination — agents declare success before fully validating outputs

The analysis also revealed that certain tasks remain persistently difficult: whole-program static analysis tools and Java-specific utilities showed the lowest success rates, with Java toolchains presenting greater obstacles than C/C++ equivalents. Notably, self-reported success metrics consistently inflated the true success rate compared to manual verification.

Real-world validation#

The setups generated by AnalysisAgent proved functionally robust. Extended runs uncovered two previously unknown bugs in the masscan and radare2 tools, demonstrating that the agent's configurations support genuine downstream analysis work rather than merely appearing functional at the surface level.