Study Raises Concerns About Reliability of Neural Decompilation for Dart Binaries

A new academic study examines how well neural decompilation systems perform on compiled Dart code, questioning both their practical reliability and the metrics used to measure them.

The Core Problem#

Decompiling binary code—converting compiled executables back into human-readable source—is a critical challenge for security researchers, malware analysts, and software maintenance teams. Machine learning has offered promise in this domain, but the study reveals significant gaps between claimed performance and actual functional correctness.

Researchers evaluated several leading adapter-baseline neural decompilation models through execution-based testing. Crucially, they paired their tests across multiple checkpoints and random seeds, applying statistical corrections for multiple comparisons. The results showed functional regressions—meaning the adapted models actually performed worse—in both Qwen3-8B adapter variants at every tested checkpoint. The other four model comparisons produced inconclusive results, suggesting the problem isn't isolated.

Metric Validity Under Scrutiny#

The team also audited what standard evaluation metrics actually measure. On 141 certified reference tasks with valid contracts, they found troubling correlations: the most widely used CodeBLEU score showed only weak-to-modest association with pass@10 performance ($\rho$ = .218–.246), and a compilation-focused metric was even weaker ($\rho$ = .072–.082). Most strikingly, only 21–23% of code samples that compiled actually executed correctly—a huge gap between syntactic and functional success.

How Semantic Cues Shape Results#

A targeted intervention removed semantic information—variable names and related metadata—while keeping types, operation counts, and instruction sequences. This single change reduced task coverage from 42 to just 7 out of 154 tasks, indicating that neural models rely heavily on human-readable naming to infer function behavior. Graph perturbation tests showed no detectable degradation under the semantic contract (six-test Holm p ≥ .750), leaving the role of instruction-use attribution unresolved.

Key Findings on Consistency#

When the researchers sampled the same model five times with different random seeds, consistency was poor. The baseline model solved an average of 4.8 tasks but reached a maximum of 15 across all seeds—meaning it solved different problems each time. Only one task was solved in every single run, raising questions about how reliably these systems work in practice.