LLMs Could Screen Recommendation Updates Without Costly Live Tests

A new approach using large language models (LLMs) could help companies evaluate changes to their recommendation systems before launching expensive live experiments. Researchers from the ArXiv platform explored whether LLMs can reliably judge which sets of recommended items users would prefer, offering a faster way to filter promising candidates.

The Challenge of Testing Recommendations#

Slate recommendation systems—those that present users with ordered collections of items like playlists or product feeds—present a testing dilemma. Traditional evaluation relies on historical user interaction data collected from past system behavior. Off-policy estimators, which assess new systems using old data, only work well when the new system targets the same objectives as the one that generated the training data. This limitation slows innovation and wastes resources on testing weak ideas.

Using LLMs to Simulate Preferences#

The research team tested whether LLMs could express meaningful pairwise preferences between different slates—essentially asking: "which of these two recommendations would a user prefer?" They developed a validation protocol measuring how well synthetic LLM-based preferences aligned with standard recommendation metrics and whether they followed logical preference rules (axioms).

The results showed that when combined with the generalized Rao-Kupper model, LLM-based synthetic testing produced rankings that remained consistent across different utility weightings. In contrast, traditional off-policy methods proved reliable only when the new system's goals matched the behavior patterns in the historical data.

A Practical Screening Stage#

Rather than replacing live A/B testing entirely, the authors position this method as an intermediate screening tool. It sits between initial off-policy evaluation and costly production experiments, allowing teams to eliminate weak candidates and allocate their A/B testing budget toward the most promising options. The approach acknowledges that synthetic preferences from LLMs cannot fully replace real user feedback—but they offer a low-cost way to identify which experiments are worth pursuing.

The methodology examines how LLM pre-training choices and configuration settings influence the quality of preference judgments, providing guidance for practitioners implementing this workflow.