Anthropic's Claude 3.5 Sonnet Upgrade Shows Little Edge Over Original in Real-World Testing

Anthropic unveiled Claude Fable 5.1 this week as a major step forward in coding and knowledge work, touting a dramatic leap on agentic research benchmarks. Yet when tested on real-world assignments, the new model proved barely distinguishable from its older sibling.

The headline numbers were certainly eye-catching: on Terminal-Bench-Science—a test involving multi-step research tasks at the command line—Fable 5.1 scored 52.6% compared to Fable 5's 24.7%. That roughly 2x improvement drove enthusiasm from prominent AI practitioners and researchers, and Anthropic positioned 5.1 as the reason to upgrade. Both versions carry the same pricing: $10 per million input tokens and $50 per million output.

Benchmark Promise Versus Practical Reality#

Yet benchmark scores operate under carefully controlled conditions and narrow task definitions. Vendors help shape which tests matter, and models can be tuned to perform well on specific benchmarks without necessarily translating that to everyday use.

To assess whether the upgraded model actually delivered value on work people rely on AI to handle, the author ran four realistic tests:

• Agentic research: Clean corrupted experiment data using lab notes as a guide, then compute batch averages
• Agentic coding: Locate and fix two intentional bugs in a Python project with a failing test suite
• Reasoning: Solve two math problems requiring step-by-step work
• Sensor data audit: Unravel messy sensor readings with known issues including calibration problems, timing drift, and format inconsistencies

Indistinguishable Performance Across the Board#

On research, coding, and reasoning tasks, both models achieved flawless accuracy. Fable 5.1 ran marginally faster and slightly cheaper on the simpler assignments. The results tracked closely with what Anthropic's own benchmark differences had predicted for these narrower problems.

The sensor audit—designed as a harder test to reveal meaningful separation—again produced identical results. Both models correctly handled every trap: recalibrating a swapped sensor at the right moment, filtering corrupted rows, and converting units in the proper sequence. However, Fable 5 completed this most demanding task in fewer turns, consumed 70% fewer tokens, and cost 34% less. Fable 5.1, despite its benchmark superiority, actually underperformed on cost and speed when the work got toughest.