Software Engineering
Machine Learning Detects Flaky Tests in Swift with 92% Precision Using Code Vocabulary
Researchers trained classifiers to identify unreliable tests in Swift by analyzing code patterns, achieving strong results where prior work has only focused on Java and JavaScript.
Flaky tests—those that pass and fail unpredictably without any code modifications—undermine confidence in continuous integration pipelines and slow release cycles. A new study shows that machine learning can reliably spot these problematic tests in Swift by learning their distinctive vocabulary patterns.
The Research Approach#
Researchers collected data from 15 open-source Swift projects, isolating 91 flaky tests alongside 22,349 stable ones through repeated test execution and historical commit analysis. They then trained five different machine learning classifiers—Random Forest, Decision Tree, Naive Bayes, SVM, and KNN—using text-based features extracted from test code.
Strong Detection Performance#
The Random Forest model delivered the strongest results, achieving 92% precision and an F1 score of 0.86. The model's performance substantially beat trivial approaches, including a simple rule-based threshold applied to the most revealing vocabulary tokens. This gap (Matthews correlation coefficient of 0.75 versus 0.08 for the best baseline rule) confirms that genuine predictive signal exists in code vocabulary patterns.
Telling Vocabulary Patterns#
Analysis of which words matter most revealed two distinct groups of signals:
- Flakiness markers appear heavily in unstable tests and include asynchronous constructs (
async,await), event-based waiting (expectation,fulfill), exception handling (throws), and explicit timing operations (timeout,wait,now) - Stability markers suggest reliability, primarily straightforward test assertions (
xctassertequal) typical of synchronous, deterministic tests
When the Model Struggles#
The researchers identified important limitations. The approach fails when test flakiness originates in shared code infrastructure beyond the test itself, or when asynchronous patterns are applied deterministically. These cases reveal the fundamental boundary of lexical analysis—code structure alone cannot always expose deeper architectural issues.