AI
Google's Zero-TIM Method Eliminates Numerical Drift in TPU Reinforcement Learning
A new technique ensures that sampling and training operations on Google's TPUs produce identical numerical results, stabilizing large-scale language model reinforcement learning.
Google Developers Blog (AI)
Google researchers have unveiled a method to solve a subtle but critical problem in reinforcement learning training: when sampling and training operations use separate execution paths on TPU hardware, they can produce slightly different numerical values for the same computation, causing training instability.
The researchers call their solution Zero Train–Inference Mismatch (Zero-TIM), implemented in Tunix, Google's open-source JAX-based library for language model reinforcement learning. The approach ensures bitwise-identical numerical results across sampling, serving, and training pipelines, even when distributed across 64 TPU v5p chips.
The Core Problem#
Training large language models with reinforcement learning requires true on-policy learning—meaning every action must be generated and evaluated by the exact same model weights. While synchronizing rollout batches ensures weights match, a hidden numerical issue remains: the sampler (which generates tokens incrementally) and the trainer (which evaluates them in batches) follow different code paths that can produce divergent floating-point results.
This divergence stems from floating-point arithmetic's non-associativity. When adding numbers, the order of operations changes where rounding occurs. The effect is particularly visible in lower-precision formats like bfloat16. Consider a simple example: adding the same value multiple times separately yields a different result than grouping them first, because rounding behaves differently depending on the reduction tree structure.
Across separate serving and training stacks, differences in reduction order, intermediate precision, and tensor parallelization create cumulative errors that push the system into an inadvertently off-policy regime—despite identical weights and model equations.
The Solution: A Four-Part Recipe#
Zero-TIM bridges this gap using a generalizable approach that does not require merging sampler and trainer into a single engine. The method has four components:
- Lock within-shard kernel accumulation: Specialized Pallas kernels control exactly how partial sums accumulate within each tensor-parallel rank, maintaining float32 precision throughout before final casting.
- Order cross-shard reductions: Communication (via ppermute) is separated from addition, with every rank summing partial outputs in the same source-rank order. The reduction sequence becomes explicit in the program rather than determined by collective implementations.

