Study compares shrinking strategies across Haskell testing frameworks

Researchers have conducted the first systematic, quantitative comparison of shrinking algorithms across three major property-based testing frameworks for Haskell. Rather than focusing solely on bug-finding, this work directly measures how effectively and efficiently each framework reduces failed test cases to minimal, reproducible examples that developers can actually debug.

What is shrinking?#

Property-based testing works by generating random test inputs and checking whether code behaves as expected. When tests fail, developers need a clear, small counterexample to understand what went wrong—not a large, noisy random input. This is where shrinking comes in: it automatically simplifies failed test cases into minimal versions while preserving the failure, making bugs far easier to diagnose.

Testing the three frameworks#

The study evaluated QuickCheck, Hedgehog, and Falsify across multiple workloads and generator types:

• Type-based generators
• API-based generators
• Correct-by-construction generators

Measurements tracked two key dimensions: effectiveness (how close to the ground-truth minimal counterexample each framework gets) and cost (how long shrinking takes, measured both as absolute time and time per unit of progress).

Key findings#

QuickCheck's structural shrinking approach generally delivered both speed and competitive final result quality across the test workloads. Interestingly, more sophisticated integrated shrinking—where the framework combines multiple shrinking strategies—did not automatically guarantee better performance or more minimal counterexamples, suggesting that integration overhead can outweigh theoretical benefits depending on the problem.

Implications for future work#

The authors emphasize that shrinking has long been treated as a secondary concern in property-based testing despite its practical importance. This structured evaluation provides a foundation for future shrinking algorithm design and benchmarking, potentially informing how testing frameworks should balance speed, result quality, and implementation complexity.