New Benchmark Reveals Major Gaps in LLM Temporal Reasoning, Even With Web Search

Researchers have unveiled a fresh evaluation framework that exposes limitations in how large language models handle complex temporal reasoning—a capability that's increasingly crucial as LLMs gain access to web search and other external tools.

The Time Puzzles Benchmark#

The new assessment, called Time Puzzles, is designed to test iterative temporal reasoning in settings that more closely match real-world LLM deployment. Rather than presenting isolated date questions, the puzzles layer factual temporal anchors together with calendar constraints across different cultural traditions, often yielding multiple valid solutions. Because the puzzles are generated algorithmically, researchers can deploy them continuously and control their difficulty with precision.

Performance Gaps Exposed#

When tested across 13 different LLMs, the findings were sobering. The best-performing model (GPT-5) managed only 55.3% accuracy without access to tools, even though the underlying facts required for solving were straightforward enough to find through web search. Enabling web search did provide a boost—but the improvement was more modest than one might expect.

A striking detail emerged when researchers reframed the constraints by substituting specific dates for references that required lookup. Under this modified setup, model performance jumped substantially, suggesting that the real bottleneck isn't access to information but rather the ability to navigate the multi-step reasoning chains needed to link facts, convert temporal references, and apply cross-cultural calendar logic together.

What This Reveals#

The findings highlight a critical weakness: while tool use has become a standard feature in modern LLMs, their reliability in leveraging those tools for iterative problem-solving—especially when reasoning chains grow complex—remains far from mature. The gap between models' ability to work with pre-stated facts and their ability to search for and systematically apply facts to solve constrained problems suggests that current approaches to integrating external knowledge with reasoning fall short of what practical applications demand.