AI
New study tests multi-modal AI models to transcribe handwritten documents across multiple pages
Researchers investigate how large language models combined with OCR can better recognize handwriting by preserving context across entire documents.
Transcribing handwritten documents has long posed difficulties for computer vision systems. A new research paper from arXiv examines how multi-modal large language models (MLLMs) paired with optical character recognition (OCR) technology can improve performance on this task, particularly when documents span multiple pages.
The challenge with multi-page documents#
Most existing work treats handwritten document transcription as a single-page problem, with each page processed independently. This approach discards valuable context — handwritten documents typically maintain consistent handwriting styles and semantic themes across pages. By processing pages one at a time, current methods miss opportunities to use these shared patterns to improve accuracy.
Prior transcription approaches have relied on either fine-tuned models (which demand large amounts of labeled training data) or zero-shot tools like traditional OCR engines and MLLMs. While MLLMs show promise, there has been limited empirical study of how different prompting strategies affect their performance on multi-page handwriting recognition, and little exploration of how to combine OCR and language models effectively.
The research approach#
The team introduces a suite of methods that combine OCR, text-based language model post-processing, and end-to-end MLLM transcription. They evaluate these techniques across a new benchmark assembled from existing datasets, including a newly created dataset called Malvern-Hills.
The core innovation involves two new prompting strategies: OCR+PAGE-1 and OCR+PAGE-N, which pass context across pages to the language model while keeping prompt complexity manageable. These approaches outperform traditional single-page methods by sharing information between pages without overwhelming the model with excessive input.
Why it matters#
This work addresses a real-world problem: handwritten documents in archives, medical records, and historical collections are rarely single pages. Better multi-page transcription could unlock digitization of large document collections without requiring expensive manual labeling or specialized model retraining for each new handwriting style.
