Researchers Validate LLM-Driven User Experience Simulations Against Real-World Feedback Data

Early-stage product teams frequently face a practical constraint: they want to iterate on user experience decisions, but lack the resources to conduct formal studies with real users. Researchers at arXiv have developed a method to bridge this gap by generating simulated user-experience feedback through large language models, then validating those simulations against authentic customer data.

How the simulation pipeline works#

The approach generates structured feedback by feeding an LLM a combination of UI snapshots, user personas, task descriptions, and iterating prompts. The system outputs walkthrough steps, friction points, and survey-style signals that mimic what a real user study might reveal. This allows teams to test product hypotheses without waiting for actual user research.

Validation through proxy datasets#

The core challenge: without real usability studies to compare against, how do you know if the LLM's simulated feedback is accurate? The researchers tackled this by treating public data as a "proxy" for real user experience—specifically:

• App store reviews from services like Gojek
• Support team tweets and customer service interactions
• Open-source project issues and bug reports

These sources capture genuine friction points and pain points that users encounter in the wild.

Measuring alignment#

The team introduced two metrics to assess whether simulated friction themes matched up with patterns in the proxy data:

• Top-k Jaccard: a measure of overlap between simulated and real themes
• Weighted-Jaccard (W): a distributional metric that accounts for how common each theme is

Testing lexical matching, TF-IDF analysis, and embedding-based approaches, they found that embedding-based alignment consistently outperformed simpler baselines. For example, on Gojek data, embedding W scores (0.128) beat lexical methods (0.000).

However, they also discovered limitations: top-k Jaccard tended to overstate alignment at larger sample sizes, and embedding estimates showed unstable performance when resampled at their data scale.

Scaling up the approach#