Security detection rules evolve messily, not steadily—study of Sigma and Splunk repositories reveals ongoing tradeoffs

Researchers at Carnegie Mellon University have conducted the first large-scale longitudinal study of how security detection rules change over time, revealing a messier reality than expected: rather than evolving toward a stable, optimized form, rules commonly undergo repeated cycles of expansion and contraction as teams wrestle with fundamental trade-offs.

The study#

The analysis examined 6,859 rule histories from two major public repositories: the community-driven Sigma project and the curated Splunk Security Content (SSC). To measure real changes in detection logic rather than just surface-level edits, the researchers developed a predicate graph intermediate representation—a canonical way of visualizing a rule's logical structure—and paired it with an alignment algorithm to track how the underlying logic shifts between versions.

Key findings#

Around 56% of rules undergo at least one revision to their detection logic during their lifetime. More striking than the frequency of change is its pattern:

• Most rule evolution is non-monotonic: more than half of all rules both add and remove clauses at different points, rather than steadily accumulating conditions.
• Rules commonly revert to earlier states, suggesting that choices are revisited rather than simply locked in once made.
• Between one-quarter and one-third of rules oscillate between two competing goals: expanding detection coverage versus reducing false positives. This cycle repeats rather than converging.

Why it matters#

The findings challenge the assumption that security rules naturally drift toward optimization. Instead, they reflect continuous operational compromise—teams detect a threat, tighten the rule, trigger too many alerts, loosen it, miss attacks, tighten again. This insight has implications for how organizations should think about rule management, governance, and deployment strategies. The study raises open questions about whether better processes, tooling, or standards could help teams escape these cycles or more deliberately navigate the coverage-versus-precision tradeoff.

The full paper, with detailed methodology and LLM-assisted analysis of rule intent alongside human validation, is available on arXiv (cs.SE 2605.05383).