New benchmark reveals reverse engineering remains a major blind spot for AI security agents

Researchers have identified a critical gap in AI's cybersecurity capabilities: while language models excel at analyzing software when source code is available, they stumble when forced to reverse engineer binaries—a task that's unavoidable for studying malware, firmware, and proprietary applications. A new benchmark designed to measure this gap reveals the problem is far more severe than previously thought.

The missing link in AI security#

Most cybersecurity work involving AI has focused on scenarios where source code is accessible. But much of the software that matters most to security—malware, embedded systems, closed-source applications—exists only in compiled form. Extracting meaning from these binaries through reverse engineering (RE) is a foundational skill in security research, yet evaluating how well agents perform at it has proven surprisingly difficult.

The core challenge stems from a fundamental methodological problem: to properly test whether models are genuinely analyzing binaries or simply recognizing them from training data, benchmark examples must have never appeared as source code in a model's training set. Simultaneously, these examples must be substantial and realistic—matching the complexity and anti-tampering techniques of real-world software. Until now, no benchmark has satisfied both requirements.

Introducing SRE-Bench#

Researchers created SRE-Bench, described as the first reverse-engineering benchmark that is both clean of training contamination and truly realistic. Built from the ground up by reverse-engineering experts over 5,000 hours, the benchmark includes 19 custom-built programs, each around 16,900 lines of code on average. To make the challenge authentic, the team developed 44 distinct anti-analysis techniques, resulting in 262 different binary variants and 1,572 graded analysis tasks.

Performance gaps are stark#

Evaluation across five leading language models tells a sobering story:

• GPT-5.6-sol (the strongest performer) achieved 61.4% accuracy per instance and fully solved just 31.5% of the binaries
• Claude-Opus-5, GPT-5.5, Grok-4.5, and GLM-5.2 all scored lower
• Performance remained largely flat across compiler optimization levels and linking strategies—areas where human analysts would show significant variation