New dataset reveals how humans intervene in AI web agents—and how to predict it

Autonomous web agents continue to improve, yet human oversight remains crucial for steering these systems toward user goals and correcting course when needed. The challenge is that today's agents often lack insight into when and why people choose to step in—sometimes acting independently past key junctures or pestering users for needless sign-offs.

Researchers have now tackled this problem head-on by studying patterns of human-agent collaboration at scale. Their work introduces a structured approach to anticipating user intervention, alongside a new dataset designed to understand how people actually interact with autonomous systems.

Mapping four styles of human oversight#

The team collected CowCorpus, a dataset comprising 400 real-world web navigation paths furnished by actual users, with over 4,200 interspersed human and agent moves. Analysis of these trajectories uncovered four distinct supervision patterns:

• Hands-off supervision — users monitor but seldom interrupt
• Hands-on oversight — frequent, proactive user involvement
• Collaborative task-solving — back-and-forth refinement between human and agent
• Full user takeover — the person resumes complete control

Training models to anticipate intervention#

Using these interaction categories as a foundation, researchers trained language models to forecast whether and when a user would intervene given the current task state and that user's typical approach. The intervention-aware models achieved a 61.4–63.4% boost in prediction accuracy over baseline models without this context.

The real-world payoff emerged when these smarter models were embedded into live web agents and tested with actual users. Agents tuned to anticipate intervention were rated 36.8% more useful than their non-adaptive counterparts, suggesting that respecting and predicting human preferences breeds stronger human-agent partnerships.

Why it matters#

The findings underscore a fundamental shift in how collaborative AI systems should work: rather than treating humans as occasional nuisances to confirm decisions, systems that understand individual interaction styles and timing become genuinely adaptive partners. As autonomous agents take on more complex, real-world tasks, this kind of behavioral modeling could make the difference between tools that annoy and tools that genuinely augment human capability.