AI
Google releases long-context training tools for TPUs, achieving 70% efficiency on 4M-token sequences
MaxText now supports context parallelism strategies that let AI models train efficiently on sequences up to 14 million tokens by distributing computation across multiple chips.
Google Developers Blog (AI)
Google has announced a new capability for its MaxText training framework that tackles one of the hardest problems in modern AI: training large language models on extremely long sequences. The update enables efficient training—not just inference—at context lengths that reach into the millions of tokens, addressing computational bottlenecks that emerge when sequences grow beyond a single chip's memory.
Why Long-Context Training Matters#
Extending context windows gives language models the capacity to process entire code repositories, hours of video, or complex multi-step reasoning tasks. While native context windows have ballooned from 128K–200K tokens three years ago to 1M–2M today—with models targeting 10M tokens on the horizon—the real challenge is training at those lengths, not just serving them.
At such scales, two critical system bottlenecks emerge: attention computation grows quadratically with sequence length, and the activations required (particularly keys and values for every token) exceed what any single chip can hold.
Context Parallelism: The Core Solution#
Rather than fitting an entire long sequence on one processor, Google's approach splits sequences across multiple chips through context parallelism (CP). The key insight is that keys and values stream between chips during computation, rather than being duplicated on every device—keeping each chip's memory footprint constant regardless of total sequence length.
MaxText now supports four distinct context-parallel strategies on Cloud TPU:
- All-gather: Each chip holds its portion of the sequence and gathers all keys and values at each layer for local attention. Simplest to implement, but memory usage scales with the full sequence length.
- All-to-All (Ulysses): Rotates which dimension each chip is responsible for—trading token chunks for attention heads. Cuts per-chip memory dramatically, but can't scale beyond the number of model heads.
- P2P (Ring Attention): Keys and values circulate around a ring of chips, one hop at a time. No chip ever holds the complete sequence; communication hides behind computation.
- A2A + P2P (Unified Sequence Parallelism): Combines both strategies on different axes of the chip mesh, balancing All-to-All's efficiency gains with Ring Attention's scalability.
All four use the Tokamax Splash Attention kernel, which handles packed sequences and skips fully masked compute blocks.

