Japanese Riddles Expose Critical Reasoning Gap in AI Self-Evaluation

Researchers have developed a novel evaluation framework that uncovers a blind spot in how today's large language models reason and assess their own thinking. The work, based on Japanese children's riddles, identifies a gap between what AI can generate and what it can recognize as valid.

The NazoNazo Benchmark#

The study introduces the NazoNazo Benchmark, a dataset of 201 Japanese riddles designed to test a specific type of reasoning: the ability to restructure understanding and evaluate one's own thought process. Unlike conventional benchmarks that risk contamination from training data, this renewable collection targets reasoning processes that are hard to detect in standard evaluations. Researchers established a baseline from human performance on 120 riddles, where participants achieved roughly 53% accuracy.

Performance Gap Across Models#

When tested on this riddle set, advanced language models showed striking underperformance. Non-reasoning models scored around 7.6% accuracy, while reasoning-oriented systems reached only 17.6%—both far below human capability. The benchmark evaluated 38 frontier models developed between 2023 and 2025 under strict conditions that eliminated retrieval assistance and required zero-shot solving.

The Verification Failure Pattern#

Beyond raw accuracy scores, the research uncovered a distinctive failure pattern called verification failure. Models would generate a correct answer as an intermediate step during their reasoning, yet fail to select it as their final response. Across models with usable reasoning logs, verification failures accounted for 5% to 39% of incorrect answers. This dissociation reveals what researchers term a "metacognitive bottleneck"—a breakdown between the ability to produce candidate solutions and the ability to endorse them.

Implications for Future Improvement#

The finding suggests concrete avenues for development: better confidence calibration, structured verification procedures, and improved stopping mechanisms that know when to commit to a generated answer. By isolating where models lose track of their own valid reasoning, the work provides a framework for diagnosing reliability and points toward more robust reasoning systems.