New Dataset Tests AI's Struggle With Complex Math Across Scientific Papers

Researchers have created a new benchmark that exposes a critical weakness in how large language models handle numerical reasoning across long scientific documents. The dataset, called SciTrek, asks AI systems to perform calculations and logical operations—counting, sorting, aggregation, comparison—over full-text academic papers, revealing that current frontier models are far less capable than expected at this task.

The Scale of the Problem#

Top-performing models achieve only 46.5% accuracy on exact-match answers when processing 128,000 tokens of context, and their performance drops steadily as documents grow longer. This happens even on questions that appear conceptually simple, suggesting the weakness is not isolated to edge cases but reflects a fundamental limitation in how these systems process and reason over extended textual information.

How SciTrek Works#

The dataset uses a fully automated pipeline to generate questions and verify answers. Rather than relying on human annotation, researchers formulate queries as SQL statements over a database of article metadata—titles, authors, references—then automatically generate questions and execute the queries to obtain ground-truth answers. This approach scales to arbitrary context lengths and dataset sizes with minimal human oversight, and the underlying SQL structure provides a transparent, auditable specification of exactly what reasoning each question demands. It also avoids data contamination issues that plague manually curated benchmarks.

Key Weaknesses Identified#

Fine-grained analysis reveals that models systematically struggle with certain problem types. They perform particularly poorly on citation-related questions and on compound logical conditions, especially when negation is involved. These patterns suggest that models have difficulty tracking relationships and applying multi-step logical filters across large documents.

Path Forward#

Encouragingly, post-training open-weight models on SciTrek data improves their numerical reasoning in ways that transfer to other long-context tasks outside the training domain, indicating that targeted training on this benchmark can yield generalizable gains. The researchers released their work on arXiv.