Google's LLM-Powered Code Optimizer Saves Hundreds of Thousands of CPU Cores at Scale

Google researchers have deployed ECO, an AI-driven system that automatically optimizes source code running across warehouse-scale data centers. The technology combines large language model techniques with continuous profiling to identify and fix performance bottlenecks in production code at unprecedented scale.

How ECO Identifies Targets#

Rather than searching blindly, ECO tackles the challenge of pinpointing what to optimize by layering multiple detection methods. Fleet-wide profiling reveals which code segments consume the most resources, while an embedding-based search engine narrows the field to specific optimization opportunities. The system mines a dictionary of common performance anti-patterns—inefficient coding practices that harm system efficiency—to guide its recommendations.

Ensuring Reliability and Correctness#

Deploying AI-generated code changes to production carries inherent risks. ECO addresses this through a three-stage safety net: automated testing validates functionality, LLM self-review catches reasoning errors, and post-deployment monitoring tracks real-world performance. This multi-layered approach enables confidence that changes will improve systems without introducing bugs or causing service degradation.

Real-World Impact#

ECO has shipped more than 6,400 production commits modifying over 25,000 lines of code across Google's infrastructure. Of all submitted changes, 99.5% never required rollback—an unusually high success rate for automated code transformation at this scale. The optimizations have yielded efficiency gains equivalent to freeing up hundreds of thousands of normalized CPU cores, demonstrating that LLM-based optimization can deliver genuine business value beyond academic proof-of-concept.

The work, published on arXiv, shows how careful system design and verification can make AI-assisted code optimization both safe and practical for hyperscale production environments.