AI
New benchmark tackles broader defect detection in automated Android app testing
Researchers introduce GUITestScape, an evaluation framework that assesses AI agents on both interaction and display defects across real-world applications.
A research team has developed GUITestScape, a new interactive benchmark designed to evaluate how well AI agents can autonomously test Android applications for bugs without predefined test cases. The work addresses significant gaps in current testing evaluation methods.
The challenge of exploratory testing#
When AI agents equipped with multimodal large language models (MLLMs) perform exploratory GUI testing, they face a unique challenge: they must navigate applications independently and uncover defects through open-ended interaction. This is substantially harder than following scripted test sequences. However, existing benchmarks have fallen short in two key ways. Most focus narrowly on interaction defects — bugs triggered by user actions — while largely ignoring display defects such as rendering issues or visual glitches. Additionally, traditional evaluation approaches rely entirely on preset defect labels, treating the entire testing process as a single pass-or-fail verdict that obscures different types of failures.
GUITestScape and GUIJudge framework#
The new benchmark comprises 61 real-world Android apps and 508 manually annotated defects spanning both interaction and display categories. Alongside it, researchers introduce GUIJudge, an open-set evaluator that breaks down an agent's testing trajectory into distinct, independently measurable capabilities rather than collapsing results into one final judgment.
Experimental testing shows that GUIJudge delivers dependable evaluation that goes beyond predefined labels, substantially surpassing all comparison methods. When applied to existing agent systems without retraining, GUIJudge's diagnostic tools measurably improve defect detection rates.
Key findings#
The benchmark revealed that defect detection remains the primary bottleneck for current models on both interaction and display defects. This finding suggests that future improvements should prioritize enhancing how agents identify potential issues, not just how they interact with applications.

