AI
Small AI Models Can Extract Financial Data Accurately, But Verification Remains Critical
A new benchmark reveals that compact open-source language models can read SEC filings with verifiable evidence, though accuracy gaps and reasoning modes show room for improvement.
Researchers have tested whether lightweight language models running on standard consumer hardware can reliably pull financial numbers from SEC filings while providing checkable proof of their answers—a key requirement for trustworthy AI in high-stakes applications.
The Challenge: Beyond Fluent-Sounding Answers#
Large language models produce impressively coherent text, but that fluency does not guarantee correctness. This becomes especially problematic when dealing with regulatory filings where precision matters. The researchers posed a simple question: can smaller models that fit on a single GPU extract specific figures from SEC documents and cite their sources?
Introducing Filing-Evidence Benchmark#
To investigate, the team created a standardized test bed called Filing-Evidence, comprising 1,000 questions drawn from 90 recent 10-K and 10-Q filings across 45 US corporations. The answer keys came directly from inline XBRL tags embedded in the filings themselves—ensuring objective, unambiguous ground truth.
The models tested ranged from 7 to 21 billion parameters and came from ten different developers. All were evaluated on a single RTX 4090 GPU under three conditions: plain operation, a check-and-retry loop, and use of the model's own reasoning capabilities.
Results: Wide Variation in Performance#
Accuracy with verifiable evidence ranged widely:
- Plain setup: 17.1% to 82.7% across the ten models
- Best overall performance: 84.5% (achieved by Qwen3.5-9B)
- Automatic rejection impact: The checker flagged many wrong answers, but between 16 and 167 responses per 1,000 contained incorrect values for answerable questions and still passed all verification checks
Common Failure Modes#
Of the wrong answers that evaded detection, the breakdown was revealing:
- Scale or unit errors: 57%—answers with the right number but wrong magnitude or denomination
- Wrong row or column: 30%—data pulled from a different line item or reporting period but in the correct section

