Testing & QA
New dataset reveals how web UI tests break when app structures change
Researchers created ReproBreak, a collection of 449 reproducible locator failures across Cypress and Playwright test suites to help developers fix fragile automated GUI tests.
Automated testing frameworks like Cypress and Playwright depend on locators—code snippets that pinpoint and interact with elements on a webpage. When an application's underlying structure shifts, these locators can fail to find their targets, causing test suites to break even though the actual feature still works. Addressing this persistent maintenance headache, researchers have now compiled the first large-scale dataset documenting these failures.
The Problem: Locators and Test Fragility#
Web testing automation relies heavily on locators to identify buttons, forms, text fields, and other interactive elements. A locator break occurs when structural changes to an application render these locators obsolete—the element they're searching for either no longer exists, has moved, or is inaccessible through the same path. This forces teams to manually hunt down and rewrite failing tests, even when the underlying user-facing functionality remains intact.
Introducing ReproBreak#
Researchers examined 359 open-source repositories to uncover commits where locators were modified. After verifying that these changes actually constituted genuine breaks (rather than benign edits), they documented 449 confirmed locator failures from the top 4 projects with the most locator changes. The ReproBreak dataset bundles these failures alongside automated reproduction scripts, allowing researchers and developers to systematically study locator fragility.
Why It Matters#
This resource addresses a gap in test maintenance research. By providing a curated collection of real-world, reproducible failures, ReproBreak enables the community to develop and test new repair techniques, improve test robustness strategies, and better understand what makes locators vulnerable to application changes. The dataset lowers the barrier to building tools that could automatically detect and fix these breaks.

