New Benchmark Reveals AI Agents Struggle with Form-Filling, Even with Visual Understanding

Automating paperwork through AI remains deceptively hard. A new benchmark called FormGym highlights just how far current systems fall short when tasked with filling out forms—especially when working from images alone.

The Challenge of Form Completion#

Filling forms is a mundane but cognitively demanding task. Beyond simple text entry, it demands that an AI agent understand visual layouts, retrieve the right information for each field, and interact with form elements correctly. Researchers from the FormGym study evaluated machines on 432 distinct fields distributed across 55 documents, each requiring knowledge of 236 separate user attributes. The constraint is strict: no optical character recognition (OCR), no access to underlying PDF text structure, and no DOM manipulation—just raw images.

Current AI Systems Fall Short#

The results are sobering. Visual language agents (VLAs)—systems designed for multi-modal understanding—achieved less than 1% accuracy on most test cases, hampered primarily by poor localization (pinpointing exactly where information belongs on a form). GUI-based agents, which interact with forms by clicking and typing, performed better, reaching 10.6% to 68% accuracy, but at substantial computational cost and latency.

FieldFinder: A Tool-Based Solution#

To address this bottleneck, the researchers developed FieldFinder, a specialized assistant that helps language models identify the precise locations where text should be entered on a form. When paired with existing models, FieldFinder produced consistent improvements across all test scenarios. Performance gains were dramatic in some cases: one system jumped from 2% to 56% accuracy. Every model tested benefited equally from the tool-based approach, suggesting that field localization, not semantic understanding, is the real limiting factor.

Why It Matters#

As businesses seek to automate administrative workflows, this work exposes a critical gap in current AI capabilities. The findings suggest that building better form-filling systems requires not just smarter language models, but specialized tools that bridge the gap between language understanding and spatial reasoning on documents.