Testing Claude's Upgrade: Fable 5.1 vs. 5 on a User's Budget

When Anthropic unveiled Claude Fable 5.1 this month, the company highlighted a striking benchmark claim: the new model achieves a 52.6% score on Terminal-Bench-Science, more than doubling Fable 5's 24.7%. But that gap shrinks dramatically when you test both models under conditions actual users might encounter.

The published numbers came from controlled conditions unavailable to most people—eight hours per task with unspecified computational budgets. By contrast, the public leaderboard tested Fable 5 with a $67-per-task budget and found it far less capable than Anthropic's announcement suggested. To see what a typical user would experience, one researcher selected five representative tasks from the benchmark's science domains and ran both models under a realistic $12 budget and 60-turn limit per task.

Testing real-world performance#

The sample included one test from each field covered by the benchmark: symbolic regression (identifying hidden mathematical patterns), Lorenz-96 assimilation (reconstructing atmospheric conditions from sensor data), reactor safety control (programming a chemical system controller), foraging cognitive model (predicting rodent behavior), and nanoindentation (extracting material properties from curve data).

Only one task yielded a passing result: Fable 5.1 successfully completed the symbolic regression problem in 27 turns ($1.96), while Fable 5 exhausted its budget twice attempting the same task and failed both times. On the Lorenz-96 assimilation problem, both models ran out of resources before solving it—Fable 5 hit the cost ceiling at $12.63, while Fable 5.1 used all 60 turns and spent $10.70 without passing.

For the reactor control task, neither model produced a working solution, though Fable 5.1 generated substantially more output tokens (157,710 vs. 121,978). On the foraging model, Fable 5.1 was the only one that explicitly declared completion, but the official grader rejected its output. Both failed the nanoindentation test after burning through output tokens on data parsing.

The broader picture#

Across the five tasks, Fable 5.1 achieved a 20% success rate while Fable 5 scored 0%—a meaningful but far smaller gap than Anthropic's public numbers suggest. The small sample makes drawing firm conclusions difficult, though the direction of improvement aligns with published results, and Fable 5.1 showed an advantage in mathematics, which also represents Fable 5's strongest domain on the independent leaderboard.