New Framework Improves Document AI by Progressively Focusing on Relevant Regions

A team of researchers has unveiled Doc-CoB (Chain-of-Boxes), a new approach to helping AI systems extract information and answer questions about document images more accurately.

The Challenge with Current Methods#

Many existing document understanding systems face a fundamental tradeoff. Some process entire documents in a single pass, treating all areas equally regardless of relevance to the query. Others zoom into specific regions for detailed analysis but risk losing important broader context. This creates a problem when AI needs to handle the visual complexity and dense information typical of real-world documents.

How Doc-CoB Works#

Rather than jumping straight to small regions of interest, Doc-CoB employs a coarse-to-fine strategy that integrates visual reasoning directly into multimodal large language models. The framework first identifies key layout boxes within a document, then progressively focuses on those areas for deeper analysis using visual prompting. This approach preserves global document information while concentrating computational effort on query-relevant sections.

Training and Validation#

To enable this capability, the researchers introduced two new reasoning tasks: box recognition (identifying relevant regions) and box reasoning (understanding their content). They built an automatic pipeline that generated 249,000 training samples with intermediate visual supervision. Testing across seven benchmarks and four popular language models demonstrated consistent performance improvements, suggesting the technique generalizes well across different systems.

The code and dataset have been made publicly available, pointing toward broader adoption within the document AI research community.