Software Engineering
Boundary-Value Testing Emerges as Most Reliable Method for Catching GPU Kernel Bugs
A systematic comparison of test-generation strategies reveals that sampling edge-case tensor shapes catches bugs in GPU kernels while avoiding false alarms on correct code.
Researchers have put competing approaches to test-input generation for tensor computing kernels to the test, revealing which methods actually work best at catching real bugs without creating false alarms.
The Hidden Assumptions in Kernel Testing#
Most tensor computing projects rely on informal practices: they pick one representative tensor shape and data type, run a fixed-shape validation check, then deploy. But these choices—which shapes to test, what data types to include, what input values to use—remain largely unexamined and ad-hoc. A new empirical study makes these assumptions visible and measures them systematically across different testing strategies.
Methodology: Testing Seven Strategies#
Researchers used the gpuemu fuzzing system to evaluate seven test-generation approaches on a collection of 26 tensor operations (16 correctly implemented controls and 10 buggy variants designed to mimic common transcription mistakes in LLM-generated code). They ran each strategy on an RTX 3060 GPU and measured performance along two critical dimensions:
- Bug recall: percentage of flawed kernels successfully detected
- False-positive rate: percentage of correct kernels incorrectly flagged as buggy
The strategies differed in how they selected tensor dimensions, mixed data types, and chose input values.
The Clear Winner: Boundary-Shape Sampling#
Boundary-only shape sampling emerged as the operationally safest choice, capturing 78% of bugs in the test suite while producing zero false positives on correct code. This straightforward approach samples tensor shapes at edge cases—small values, large values, and boundary conditions—rather than testing only typical sizes.
The contrast with other methods proved stark. An adversarial value-sampling strategy achieved higher raw bug recall (99%) by deliberately injecting NaN and infinity values, but this aggressiveness backfired: it triggered false alarms on 94% of correct kernels. The validator's NaN-detection logic fired indiscriminately whenever any kernel propagated these special values, whether due to a bug or normal computation flow.