ML system ranks which duplicate BDD test steps deserve refactoring

Behaviour-Driven Development (BDD) test suites often contain redundant step sequences—repeated blocks of test code that could be consolidated. While three established refactoring patterns exist to extract and reuse these duplicates, no automated method previously existed to decide which ones are worth extracting or which pattern fits each case.

The Problem with Duplication#

BDD frameworks like Gherkin organize tests into readable human-like scenarios. Over time, teams accumulate repeated step subsequences across their test files. Three refactoring mechanisms are available: extracting steps into a shared background within a single file, invoking reusable scenarios across a repository, or sharing higher-level steps across organisations. The challenge lies in ranking which recurring patterns are worth the effort to consolidate—a decision currently left to manual code review.

Method and Results#

Researchers analysed a corpus of 339 repositories and discovered over 5 million step subsequences using paraphrase-robust clustering. They trained an XGBoost classifier on a hand-labelled pool of 200 sequences to predict which duplicates merit extraction. The model achieved an F1 score of 0.891, outperforming both a rule-based baseline (F1 = 0.836) and two large language model judges (best at F1 = 0.728).

The study found that:

• 75.0% of test scenarios contain within-file background candidates
• 59.5% harbour within-repo reusable-scenario candidates
• 11.7% include cross-organisational shared-step candidates

Inter-rater agreement among the three labellers was moderate (Fleiss' kappa = 0.56 for extraction-worthiness and 0.79 for pattern selection).

Public Release#

The team released their mining pipeline, trained classifier weights, labelled dataset, and annotation rubric under the Apache 2.0 licence, enabling the BDD community to apply the findings at scale.