How LLMs recommend mobile apps—and why their logic often diverges from app store standards

When you ask an AI chatbot to recommend a mobile app, the reasoning behind its suggestion is far from transparent. A new empirical study examines how large language models generate and justify app recommendations, uncovering significant gaps between what LLMs prioritize and the metrics that drive traditional app store optimization.

The Research Question#

Researchers conducted a systematic analysis of how both proprietary and open-source LLMs rank mobile applications in response to natural language queries. Rather than relying on traditional keyword-based app store searches, conversational app discovery through LLMs offers flexibility—but it raises critical questions about consistency, accountability, and whether these systems actually align with industry-standard metrics like those used in App Store Optimization (ASO).

Key Findings#

The study identified 16 distinct ranking criteria that LLMs invoke to justify their recommendations, yet these criteria form a fragmented landscape that only partially overlaps with conventional ASO benchmarks. Several patterns emerged:

• Proprietary models are more stable: Commercial LLMs (like those from major tech companies) produced substantially more consistent recommendations than locally deployed open-source alternatives.
• Consistency varies by domain: Different categories of apps showed markedly different levels of agreement across models.
• Web search doesn't help much: Enabling web search to augment LLM knowledge did not meaningfully shift the ranking criteria these models reported.
• Instructions backfire: Counter to expectations, explicitly conditioning recommendations on specific ranking criteria actually reduced cross-model convergence rather than improving it—steering outputs away from a neutral baseline without creating alignment.

Implications#

These findings highlight tensions in the emerging landscape of conversational app discovery. End-users receive recommendations that may reflect idiosyncratic LLM reasoning rather than established quality signals. App developers face an opaque ecosystem where traditional optimization strategies don't fully translate. Researchers building recommender systems must grapple with LLMs' unpredictable and partially explained decision-making.

The researchers also provide a replication package to encourage reproducibility and future work on how language models can be made more transparent and consistent in app recommendation tasks.