LLM-Driven Code Analyzer Uses Execution Verification to Catch Vulnerabilities Across Languages

Researchers have developed a framework that uses language models to detect and fix software vulnerabilities while grounding every action in executable proof—a departure from earlier systems that might act on probabilistic predictions without verification.

The work addresses a core challenge in deploying AI agents: model predictions are educated guesses, not certainties, and errors compound across multiple stages. By requiring execution-based confirmation before attempting any repair, the system reduces false positives and unnecessary code changes.

How the System Works#

The framework combines three reasoning stages: detection of potential flaws, validation through code execution, and guided repair. At its core sits a Universal Abstract Syntax Tree (uAST), which normalizes source code across Java, Python, and C++ into a shared structural format. This enables the model to recognize patterns across different languages.

For analysis, the system fuses two complementary embeddings—structural representations from GraphSAGE and semantic embeddings from Qwen2.5-Coder-1.5B—using learned gating weights. This hybrid approach provides interpretability automatically, showing which inputs influenced each prediction.

Performance and Trade-offs#

Within a single language, the system achieved 89.84–92.02% detection accuracy. When tested on code it had never seen from other languages, cross-language performance reached 74.43–80.12% F1 score, demonstrating meaningful generalization. End-to-end, the framework successfully resolved about 70% of vulnerabilities while maintaining a failure rate of 12.27%.

Ablation studies confirm the design choices matter: removing the uAST layer dropped cross-language F1 by over 23%, and disabling the execution-validation step led to 131% more unnecessary repairs.

Why Execution Grounding Matters#

The key innovation is the strict rule: no fix is applied without execution-based evidence of exploitability. This closed-loop approach prevents the system from chasing phantom bugs or over-correcting based on uncertain predictions, making it safer to deploy in real codebases.

Research paper: arXiv 2604.10800