Agentic AI
Reference Screenshots Help AI Coding Agents Spot UI Bugs Even in Degraded Images
Researchers show that providing AI software agents with clean reference images from their own codebase helps them identify bugs in low-quality UI screenshots.
Coding agents tasked with fixing bugs from UI screenshots often struggle when image quality is poor—low resolution, compression artifacts, and tiny text can make it impossible to identify relevant elements and pinpoint the source files that need changes. Vision-language models lose reliability as image degradation worsens, while text-recognition approaches can amplify errors from corrupted inputs.
The Problem#
When agents receive bug reports with screenshots, visual evidence of the problematic UI may be unclear or unreadable. Neither pure vision models nor OCR-assisted methods reliably recover what's needed for accurate file localization when the visual signal is already degraded. The core challenge: sometimes the bug itself prevents a clear view of the application's state in the original screenshot.
A New Approach: Repository-Rendered References#
Researchers propose augmenting coding agents with clean reference images—crisp screenshots automatically generated directly from the codebase the agent is working with. These references show what the UI should look like, preserving the current (possibly buggy) implementation state. Because they come from the available repository itself, no manual annotation or knowledge of a "correct" version is required. The references can be created on demand for any reproducible application state.
Testing and Results#
The team evaluated this method on a benchmark built from SWE-bench Multimodal, injecting visual bugs into a real web application while varying image degradation levels. They tested whether coding agents could identify the correct source file to modify. Key findings:
- Reference images significantly boosted file-localization accuracy, especially under severe degradation (fourfold downsampling with JPEG compression)
- Text recognition improved by roughly 12 percentage points compared to language-biased OCR operating on the same degraded inputs
Why It Matters#
This approach bridges a practical gap for agentic software development: it helps AI agents understand UI context when visual information alone is unreliable, without requiring extra training data or manual intervention.

