AI
AI Framework Validates Suspected Bugs with Automated Tests, Finding 121 Real Defects
AnyPoC uses multi-agent verification to separate real bugs from false alarms and generate executable proof-of-concept exploits at scale.
Large language models can now flag potential bugs in code, but their alerts are often unverified hunches—leading to wasted effort on false positives. A new research framework addresses this by automatically generating executable proof-of-concept (PoC) tests that either confirm a suspected defect or expose it as a false alarm.
The Challenge: Unproven Bug Reports#
LLM-based bug detection agents produce vast quantities of candidate issues, but without automated validation, security teams must manually verify each one. This manual step severely limits scalability and creates a bottleneck in the bug detection pipeline. Worse, the LLMs themselves are prone to hallucination—they may generate plausible-sounding but non-functional test cases that trick themselves into believing they've found a real bug.
How AnyPoC Works#
The team behind AnyPoC—a multi-agent framework—tackled this by treating proof-of-concept generation as a test synthesis challenge. The system operates in three stages:
- Fact-checking: Analyzes the candidate bug report and interrogates its claims before investing effort in PoC generation
- Iterative synthesis and execution: Generates test cases, runs them against the target code, and collects execution traces to see what actually happens
- Independent re-validation: A separate agent re-executes and scrutinizes the PoC to catch hallucinations and prevent the framework from rewarding itself for fake results
The framework also maintains and evolves a knowledge base of successful PoC patterns across different software types, allowing it to tackle heterogeneous tasks—from web browsers to databases to multimedia libraries.
Real-World Results#
The researchers tested AnyPoC on 12 major software projects: Firefox, Chromium, LLVM, OpenSSL, SQLite, FFmpeg, and Redis, among others. Compared to baseline LLM agents like Claude Code and Codex, AnyPoC generated 37% more valid PoCs for confirmed bugs and rejected 9.7 times more false positives. Over two thousand noisy bug reports fed into the system yielded 121 genuine new bugs—108 confirmed by developers and 92 already patched. Notably, 46 of the generated PoCs were incorporated as official regression tests in the target projects.
