Most AI-Generated Webpages Break on Different Browsers and Devices, Study Finds

Multimodal large language models (MLLMs) have become popular tools for automatically converting visual mockups into functional webpages. Yet a new study from computer scientists shows that these AI-generated sites often fail dramatically when viewed across different browsers and devices—a critical blindspot in how we currently evaluate this technology.

The Compatibility Crisis#

Researchers conducted the first systematic analysis of cross-environment rendering in AI-generated webpages, examining 2,032 instances created by 8 leading AI tools and tested across 9 different browser-and-device setups. The results are sobering: nearly 7 in 10 generated webpages exhibited at least one compatibility issue, revealing pervasive reliability gaps that existing evaluation methods had missed.

The study, published on arXiv, introduces WebCompat, a dataset designed to capture how AI-generated code performs in real-world deployment scenarios rather than just under laboratory conditions.

Dominant Failure Patterns#

Layout-breaking failures dominated the compatibility issues observed. Nearly 9 in 10 problematic pages (88.3%) suffered from whole-page rendering failures—either shrinking dramatically with unreadably tiny text, or creating scale mismatches that cut off content. These are the kinds of glitches that render a site unusable for many visitors.

By contrast, failures confined to individual page elements—such as distorted images or missing components—were far less prevalent at just 13.4%. This concentration of layout-level breakage suggests the root problems lie in fundamental page structure and responsiveness logic rather than fine-grained UI details.

Why AI Models Fall Short#

Paradoxically, the research found that most MLLMs do incorporate responsive design patterns into their generated code. The problem is execution: the models generate responsive structures but fail to implement them correctly. When these designs encounter unfamiliar browser configurations or device sizes, the code breaks down.

A New Detection Tool#

To help identify and remedy these issues, the researchers developed XCompat, a lightweight offline detector that analyzes both visual screenshots and the underlying DOM tree structure. On their test set, XCompat achieved an F1 score of 0.903—outperforming both existing compatibility-checking tools and approaches using large language models.