AI Judges Poorly Predict How Users View Privacy and Helpfulness in LLM Responses

Large language models are increasingly used for sensitive work—writing emails, processing health information, and summarizing confidential meetings. Yet measuring how well these systems handle private data remains challenging, especially when users must disclose sensitive details to get help.

Researchers have created benchmarks like PrivacyLens to test whether LLMs can safeguard private information while staying useful. The catch: these evaluations typically rely on other AI systems (proxy LLMs) to judge whether an LLM's response properly protected privacy and remained helpful. A new study challenges whether this approach actually reflects users' real judgments.

Testing Real User Perceptions#

Researchers conducted a user study with 94 participants who evaluated 90 scenarios from the PrivacyLens benchmark. Participants rated how well LLM responses balanced two competing goals: preserving privacy and providing useful assistance.

The findings were striking: users showed low agreement with each other when judging the same responses. One person might see a response as protecting privacy while still being helpful; another might view the same output as either too cautious or too revealing.

The Proxy LLM Problem#

Five different AI systems used as "judges" showed high consistency—they largely agreed with one another on how to rate privacy and utility. However, each proxy LLM had low correlation with actual user evaluations. In other words, AI judges agreed among themselves but disagreed with humans.

This mismatch has major implications. Relying on proxy LLM scoring to benchmark privacy-preserving systems could mask real problems or overlook genuine strengths that users actually perceive.

Implications for Alignment#

The results highlight a gap between how AI systems evaluate other AI systems and how people actually experience privacy-utility tradeoffs. Users hold diverse, nuanced views shaped by context and personal risk tolerance—something automated evaluation misses.

The researchers argue for moving toward more user-centered evaluation methods and for better alignment between LLM training and users' actual expectations around privacy and helpfulness in sensitive scenarios.