New benchmark tackles conversational UI coding with visual feedback

Researchers have introduced FronTalk, a new benchmark designed to evaluate how well AI models can generate front-end code in response to conversational instructions that include both text and visual elements like sketches, mockups, and annotated screenshots.

The challenge of visual feedback in code generation#

Front-end development inherently relies on visual communication—designers and developers use mockups, wireframes, and marked-up screenshots to convey what a website should look like. Yet most existing code-generation benchmarks treat this crucial dimension as secondary, focusing primarily on textual instructions. FronTalk addresses this gap by studying how AI models respond when given equivalent instructions in both forms simultaneously, asking: can these models interpret and integrate visual feedback alongside written direction?

Benchmark design and findings#

The researchers curated a dataset of 100 multi-turn conversations extracted from real-world websites spanning news, finance, and art domains. Each conversation turn includes paired textual and visual instructions that express the same user intent, allowing researchers to measure how well models handle each modality.

Testing 20 different models revealed two persistent problems:

• Forgetting and overwriting: Models frequently lose track of previously implemented features when receiving new instructions, causing parts of the interface to break or disappear.
• Visual interpretation gaps: Open-source vision-language models especially struggle to correctly parse and act on visual feedback.

Proposed solution#

To combat the forgetting issue, the team developed AceCoder, a method that uses an autonomous web agent to verify each past instruction after implementing new changes. By continuously checking that earlier features remain intact, AceCoder nearly eliminated the forgetting problem and improved overall performance from 56.0% to 65.3%—a gain of 9.3 percentage points.

Broader implications#

The work establishes a foundation for studying multi-turn, multi-modal code generation more systematically, highlighting that conversational UI development involves challenges distinct from simpler code-generation tasks. The researchers have released code and data to support future investigation in this area.