New benchmark dataset maps hidden security flaws introduced by compiler optimizations

Compiler optimizations can inadvertently introduce security vulnerabilities into programs—a class of bugs that has proven difficult to systematize and study. A new research effort addresses this challenge by releasing a carefully audited dataset designed to support the detection and analysis of compiler-introduced security bugs.

What CISB-Bench Is#

CISB-Bench comprises 429 C-language cases extracted from historical bug reports in GCC and LLVM. Each entry includes a simplified code example, intermediate representations (IR) at various optimization levels (-O0 through -O3), documentation of its original source, and a definitive label indicating whether it represents a genuine security-related compiler failure. The dataset totals 280 confirmed compiler-introduced security bugs (CISBs) and 149 challenging non-bug cases—essential negatives that prevent overfitting in detection models.

How It Was Built#

The researchers took a deliberate approach to labeling. Two independent reviewers classified each case, achieving agreement on 369 entries (86%, Cohen's kappa = 0.662). The remaining 60 disagreements were resolved through arbitration. This dual-review process ensures the dataset's credibility as a research benchmark, since the authors acknowledge that not every report mentioning a security issue necessarily traces to an actual compiler defect.

Why It Matters#

Compiler bugs affecting security are particularly insidious because the evidence is scattered: buried in issue trackers, hidden in test case reductions, tied to specific compiler versions, and only visible in generated binaries. CISB-Bench standardizes this forensic work, making it possible to train and evaluate automated detection tools. The paired artifacts—both source code and compiled intermediate representations—enable researchers to test whether security flaws can be spotted at the source level alone, by analyzing IR, or through joint analysis of both.

Research Applications#

By providing reusable, inspectable cases with transparent provenance, the dataset supports new work in compiler security mining and machine learning-based bug detection. It sidesteps the false confidence that raw IR representations alone capture every historical failure, instead grounding research in carefully audited real-world examples.