AI
New benchmark method exposes flaws in how LLMs are tested for reverse engineering
Researchers introduce REFORGE, a pipeline that challenges the reliability of current LLM evaluation methods by accounting for compiler optimization and binary-to-source alignment uncertainty.
Large language models are being deployed in live offensive-security workflows for reverse-engineering tasks, but our ability to accurately measure their performance in this domain remains unresolved. A new paper from the arXiv cs.SE community argues that existing benchmarks obscure a fundamental problem: they treat the ground-truth labeling of decompiled functions as a solved step and fail to account for misalignment between compiled binaries and source code.
The alignment problem#
The core issue lies in binary-to-source alignment under compiler optimization. When code is compiled at different optimization levels, the mapping between the original C source and the decompiled output becomes less reliable. Current benchmarks don't measure this uncertainty, creating misleading evaluations of LLM capability.
How REFORGE works#
Researchers developed REFORGE, a provenance-tracked pipeline designed to construct reliable ground truth through a multi-stage process:
- Compilation from C source with DWARF debug information
- Syntactic extraction and alignment across optimization levels
- Decompilation with captured uncertainty
The pipeline operationalizes alignment confidence as an "eight-gate funnel" with three-tier stratification, systematically filtering functions by how reliably they can be evaluated.
What the data shows#
Testing on a controlled micro-benchmark revealed a stark picture. High-confidence function yield drops from 87.2% to 65.9% as compiler optimization levels increase. Comparisons that ignore this stratification appear to show worse LLM performance with higher optimization—but this signal is partly explained by survivorship bias, where only the easiest functions survive as evaluable in optimized binaries.
Why it matters#
The paper demonstrates the approach by evaluating seven contemporary LLMs on function naming, showing how uncertainty-aware evaluation practices could reshape benchmarking in AI-assisted reverse engineering. As these tools move into real security workflows, accurate measurement becomes essential for understanding their actual reliability versus inflated claims.
