New Framework Generates Thousands of Realistic Verilog Debugging Benchmarks from Open-Source Projects

Researchers have introduced VeriBugBench, a systematic framework for constructing rigorous debugging benchmarks in Verilog—the hardware description language widely used for chip design. The work addresses a long-standing gap: while testing and debugging RTL (register transfer level) code is critical to hardware development, researchers have lacked standardized benchmark datasets with realistic, annotated faults and corresponding test cases.

How the Framework Works#

The approach combines three complementary techniques. First, the team analyzed actual bug-fix histories across 45 open-source hardware projects to identify recurring repair patterns. These patterns were then mapped to 19 executable mutation operators—inverse transformations that systematically introduce faults at different structural levels. Second, the framework uses a large language model to generate design-specific test stimuli by analyzing each project's clean design and existing testbenches, then composing these enhanced stimuli with the original tests. Third, only faults whose effects appear at design outputs are retained in the final benchmark, ensuring practical relevance.

What the Benchmark Delivers#

The resulting VeriBugBench-v1.0 contains 2,608 single-fault Verilog variants across the 45 projects, each with executable test configurations and precise ground-truth fault locations. The assembled testbenches improved mean project-level fault observability from 36.01% to 39.54%—meaning more bugs become detectable through testing. The dataset also shows gains in line coverage and execution-trace diversity, making it more useful for evaluating debugging algorithms.

Why It Matters#

RTL debugging researchers now have a versioned, reproducible artifact for benchmarking automated fault detection and localization tools. The empirically grounded approach—building on real bug-fix data rather than synthetic mutations—helps ensure the benchmark reflects challenges developers actually face in hardware verification.