Claude Fable 5 tops Android Bench as Google's Gemini slides to fifth place

Google has refreshed its Android Bench testing platform to evaluate how well large language models handle Android development tasks, and the results present a competitive challenge for the tech giant's own AI tools. In the expanded leaderboard, Claude Fable 5 now leads the field with 84.5 percent accuracy, followed by GPT 5.4 and Claude Sonnet 5—while Google's Gemini 3.1 Pro lands in fifth place.

The Expanding Test Suite#

Android Bench, which launched in March, originally measured how AI agents performed across 100 Android coding problems. Google has now rolled out eight additional models to the test, introducing heavyweight competitors like Claude Opus 4.8, Qwen variants, GLM 5.2, and Kimi K2.7 Code alongside the existing roster. The benchmark also tracks a broader range of performance metrics, including operational cost and efficiency, giving developers more context for choosing tools.

Performance vs. Price Tradeoffs#

While Fable 5 achieves the strongest accuracy score, its success comes with a steep price tag: more than $130 in token costs to run the full benchmark suite. GPT 5.5 similarly demands substantial computational resources exceeding $130 per test cycle. Gemini 3.1 Pro, though trailing in pure accuracy, operates more economically at $87 per run. However, Gemini 3.5 Flash reveals an unexpected inefficiency—despite being marketed as a cost-effective option, it's the most expensive on the leaderboard due to extended runtime requirements, totaling $165 per benchmark cycle with a 28-hour execution window.

A Shift Toward Community Participation#

To keep the benchmark evolving and relevant, Google is transitioning Android Bench to the Harbor framework, which the company describes as a more accessible testing environment. This shift allows individual developers to run custom development tasks, generate results, and propose new test cases for the official benchmark. Recalibrating all prior tests with the new framework has produced slightly adjusted baseline scores, though the underlying evaluation methodology remains largely unchanged. The GitHub repository now hosts the updated dataset and contribution guidelines.

Implications for Google's AI Strategy#

The gap between Gemini and competing models surfaces a meaningful problem as Google increasingly positions AI agents as central to its development vision. The company's commitment to agentic systems has already prompted initiatives to acquire developer source code for training purposes—a sign of how heavily Google is betting on this direction. Gemini's middling performance in a critical use case like Android development underscores tensions between ambition and execution in that strategy.