New benchmark tests AI's ability to reason about economics, timelines, and numbers in long documents

Researchers have created a new benchmark that measures how well AI systems can answer questions requiring careful attention to numbers, dates, and precise evidence in lengthy institutional documents. The challenge mirrors real-world tasks that demand exact grounding rather than approximate topical understanding.

The problem being addressed#

Answering questions about long, densely written reports involves more than finding relevant passages. A system must distinguish between similar figures appearing across different time periods, countries, and forecast scenarios. When the same economic indicator appears repeatedly with different values for different years or nations, small errors compound into significant misunderstandings. Existing benchmarks have not adequately tested this combination of pinpoint accuracy and temporal reasoning at the scale of full institutional documents.

What AfriEconQA contains#

The benchmark draws from 220 World Bank economic reports focused on African economies, a corpus where claims are explicitly tied to fiscal years, specific countries, and projection status. It spans 4,309 question-answer pairs distributed across five reasoning types:

• Factoid questions (1,093 instances) for straightforward data extraction
• List questions (879 instances) requiring recovery of multiple related items
• Multiple choice (944 instances) testing discrimination among options
• Synthesis questions (710 instances) demanding causal reasoning across claims
• Comparison questions (683 instances) involving cross-period or cross-country analysis

Each instance includes supporting evidence and source documentation, built through an automated generation pipeline with human validation of a stratified subset.

How models performed#

The team evaluated three large language models—Qwen 3.6 35B, DeepSeek v4-pro, and Gemma 4 12B IT—under three conditions: zero-shot answering, oracle retrieval (perfect document passage selection), and retrieval-augmented generation combining model reasoning with retrieved passages. On the held-out test split of 862 examples, retrieval significantly boosted performance, but even the best system achieved only 0.545 F1 score. Errors concentrated in list extraction, synthesis reasoning, and temporally constrained comparisons.