Research
New Benchmark Exposes Critical Gaps in Binary Code Similarity Detection
Researchers reveal that current models for comparing binary code fail dramatically when tested against real-world diversity, dropping accuracy by up to 30% on firmware and semantic variations.
Binary function similarity detection—the ability to identify when two pieces of compiled code perform the same task—underpins critical security work including vulnerability discovery, malware forensics, and software patch tracking. While researchers have built numerous tools to tackle this problem over decades, they've lacked a common yardstick to measure progress fairly.
The Problem with Current Benchmarks#
Existing datasets used to evaluate binary similarity tools are narrow in scope, typically testing against only a limited set of code transformations or binary types. This has created a false sense of confidence in the field: models that appear to work well in controlled lab settings often fail when applied to the messier complexity of real-world software.
EXHIB: A More Realistic Test Suite#
A research team has introduced EXHIB, a comprehensive benchmark built from five distinct real-world datasets. Each dataset illuminates a different facet of the binary similarity problem, covering everything from firmware to semantically distinct but functionally equivalent code.
Testing nine prominent similarity detection models across EXHIB revealed troubling performance gaps. Models showed accuracy drops of up to 30% on firmware and semantic datasets compared to their performance on standard benchmarks. This gap exposes a fundamental weakness in how the field has been evaluating progress.
The Generalization Problem#
The research uncovered a critical blind spot: robustness to low- and mid-level variations in compiled code—such as different optimization levels or compilation flags—does not transfer to handling high-level semantic differences. In other words, a tool that works well when code is compiled differently may completely fail when the underlying logic itself varies subtly.
This finding suggests the field has been optimizing for the wrong metrics, potentially building tools that solve yesterday's problems while remaining vulnerable to tomorrow's real-world scenarios.