New benchmark exposes how AI agents struggle with ambiguous real-world policies

Researchers have created a new evaluation framework to test a critical blind spot in AI agents: their ability to navigate inherently ambiguous, real-world policies. Most deployment scenarios for language model-based agents involve policies that admit multiple reasonable interpretations, yet existing benchmarks rarely account for this messiness.

The Problem with Ambiguity#

The gap between how AI systems are tested and how they perform in practice is significant. Current evaluation suites assume policies are unambiguous and well-defined, but in real-world domains like retail, ambiguity is the norm rather than the exception. Customer service guidelines, return policies, and similar rules often contain gray areas that human staff learn to navigate through experience and judgment. When the rules themselves admit multiple valid readings, no single "correct" resolution may exist.

Introducing DRIP-R#

Called DRIP-R (Decision-making and Reasoning Under Real-World Policy Ambiguity in Retail), the new benchmark systematically constructs policy-ambiguous scenarios drawn from actual retail decision-making. Rather than seeking one right answer, DRIP-R evaluates how agents handle the inherent uncertainty.

The framework includes:

• A curated set of customer return scenarios rooted in real policy tensions
• Realistic customer personas to contextualize each interaction
• Two-way conversational simulation with tool-calling capabilities
• A multi-faceted judging system assessing policy adherence, dialogue quality, behavioral consistency, and resolution quality

What the Experiments Reveal#

When researchers tested frontier language models against identical policy-ambiguous scenarios, a striking pattern emerged: the models fundamentally disagreed with one another. This finding confirms that ambiguity is not a peripheral concern but a systematic challenge to how AI agents reason and decide. The disagreement suggests that models trained differently or deployed with different instructions will reach different conclusions on the same case—a potentially serious issue when consistency and fairness matter.