GPT-6 Astra's 98.6% benchmark score masks deeper questions about testing conditions

OpenAI's GPT-6 Astra has achieved a striking 98.6% score on the ARC-AGI-3 benchmark, a test designed to evaluate whether AI models can solve novel puzzles without relying on training data. This represents a dramatic leap from its predecessor, GPT-5.6 Sol, which scored just 7.8%—yet the headline number conceals important methodological nuances.

What the benchmark measures#

ARC-AGI-3 is built specifically to push models beyond their training. Rather than asking systems to retrieve memorized knowledge, the benchmark places them in unfamiliar interactive environments where they must deduce the underlying logic and adapt their approach. When the test launched in March, frontier AI models largely failed it.

The asterisk in the score#

The major caveat: OpenAI evaluated Astra using its Responses API with two settings adjusted to better represent how the model performs in production use. Critically, the company applied different evaluation setups to the comparison models. Since the testing environment itself influences performance, this methodological difference raises questions about whether the improvement is truly a measure of Astra's capabilities or partly reflects the conditions under which it was tested. OpenAI maintains the configuration changes weren't designed specifically for this benchmark, but the inconsistency in how results were gathered remains relevant context.

Performance across other demanding tasks#

Beyond ARC-AGI-3, Astra's breadth is evident: 97.6% on FrontierMath Tier 4, perfect scores on ExploitBench, and 99.2% on SRE-Bench using four attempts. On Terminal-Bench Science, it jumped from 22.4% to 64.6% compared to Sol. In practical demonstrations, the model operates directly within applications like KiCad, Power BI, and Unity, and can use an experimental feature to maintain notes across extended tasks that exceed a single context window.

The OSWorld 2.0 offline test showed similar gains: Astra achieved 72.6% in roughly 40 minutes per task versus Sol's 65.7% in approximately 75 minutes.

Contribution to open mathematics questions#

OpenAI highlighted Astra's involvement in addressing unsolved problems in number theory. Working with mathematician Julia Stadlmann, the model contributed to research on prime number gaps; one bound dropped from 246 to 186, and the system helped refine a separate constraint that had remained static for over 80 years. However, OpenAI's account lacks clarity on what Astra generated independently versus what emerged from researcher guidance, making it premature to frame this as evidence of AGI-level autonomy.