DeepMind's Decoupled DiLoCo trains large models across distant data centers with minimal bandwidth and better fault tolerance

Training frontier-scale language models has traditionally required thousands of identical chips to operate in near-perfect synchronization within a single facility. However, as models grow larger and more complex, maintaining that level of coordination across distant data centers becomes increasingly difficult. Google DeepMind's new Decoupled DiLoCo framework offers an alternative: breaking training workloads into independent compute clusters that exchange data asynchronously, isolating failures to individual islands rather than disrupting the entire training process.

Lower bandwidth, greater resilience#

The architecture builds on two prior advances—Pathways, which introduced asynchronous data flow across distributed systems, and DiLoCo, which minimized bandwidth requirements between geographically distant data centers. By combining these approaches, Decoupled DiLoCo enables models to train continuously even when individual hardware clusters experience outages.

During testing, the researchers used "chaos engineering" to deliberately introduce failures mid-training. The system recovered from entire cluster losses and reintegrated them automatically when they came back online—an improvement over traditional distributed methods where any interruption can stall the entire operation.

Real-world performance gains#

In production tests, the team trained a 12 billion parameter model across four separate U.S. regions using only 2–5 Gbps of bandwidth—achievable with standard datacenter networking rather than custom infrastructure. This setup completed training more than 20 times faster than conventional synchronous methods, since the system batches communication with computation windows rather than forcing parts of the system to wait for each other.

Benchmarking with Gemma 4 models confirmed that Decoupled DiLoCo achieved the same final machine learning performance as traditional training approaches while maintaining higher availability during failures.

Hardware flexibility and reuse#

Another advantage: the architecture can mix different generations of hardware—such as TPU v6e and TPU v5p—in a single training run without performance degradation. This extends the productive lifetime of older chips and addresses a recurring bottleneck: new hardware rolls out gradually across facilities, leaving some data centers with excess older capacity. By treating hardware heterogeneously, teams can now put that stranded compute to work rather than waiting for uniform upgrades.