How humans read AI learning—new research reveals what people actually understand about RL agents

When a reinforcement learning agent takes an unexpected action during training, human observers often struggle to understand what's actually happening inside its decision-making process. This gap between agent behavior and human interpretation can lead to poor feedback in collaborative settings—but researchers have just begun systematically studying what people actually perceive.

A new paper from arXiv explores this blind spot by asking a straightforward question: what do humans infer about how an RL agent learns when they watch it behave?

Mapping human mental models#

The team ran two experiments. In the first, nine participants watched RL agents and described their interpretations in interviews. This exploratory phase uncovered four consistent themes in how humans make sense of agent learning:

• Agent Goals — what the agent is trying to achieve
• Knowledge — what the agent knows or understands
• Decision Making — how the agent chooses actions
• Learning Mechanisms — how the agent improves over time

These categories form a human-centered framework for understanding agent behavior—one that doesn't necessarily align with the technical reality of how RL algorithms actually work.

Testing the framework at scale#

The second study brought in 34 participants and tested the framework more rigorously across different scenarios. Observers watched two types of tasks (navigation and object manipulation) controlled by two different RL algorithms (tabular methods and function approximation). The researchers collected and analyzed 816 responses to track how these interpretive themes evolved and interconnected over time.

The paradigm proved reliable, confirming that humans use consistent—if not always accurate—mental models to interpret agent learning.

Why this matters#

The findings have direct applications for designing more transparent AI systems and improving human-robot collaboration. When humans can't read an agent's learning process, they can't effectively guide it. Understanding how people naturally interpret RL behavior opens a path to building agents whose decision-making is more legible to human partners, making collaborative teaching and feedback more effective across real-world applications.

The full paper is .