Google Shows Ray AI Libraries Now Work at Scale on TPU Hardware

Google has extended Ray's support across its full AI library stack to run efficiently on TPU hardware, eliminating much of the complexity that previously made multi-host TPU setups difficult for developers.

The second part of Google's Ray-on-TPU guide demonstrates how three core Ray libraries handle distributed workloads once the underlying infrastructure—Google Kubernetes Engine (GKE) and Ray Core—manages slice placement. A TPU slice is a fixed group of chips wired together through a high-speed interconnect; keeping multi-host models on a single intact slice is the key constraint that prevents jobs from hanging.

Serving Models with Ray Serve#

Ray Serve has become the entry point for most teams deploying AI models on TPU. Models too large for a single GPU can often fit on one TPU host, and TPU capacity is frequently both cheaper and more readily available than GPU alternatives. Ray Serve provides the familiar features—autoscaling, load-balancing, multi-model serving—and integrates vLLM, a high-performance inference engine optimized for large language models.

Handling tensor-parallel models that span multiple hosts requires only a single configuration addition: declaring the slice topology (such as 4x4 for 16 chips). This topology field ensures workers stay connected through the same high-speed mesh rather than scattering across separate slices. Without it, multi-host deployments deadlock in a perpetual DEPLOYING state, burning TPU hours while appearing healthy.

Google's official tutorials walk through serving Llama 3 8B and Mistral 7B on v5e, Llama 3.1 70B on v6e, and Stable Diffusion, with deployment examples showing full end-to-end setup.

Feeding Data Efficiently with Ray Data#

TPUs process data so quickly that standard data loaders become the bottleneck. Ray Data's iter_jax_batches() method solves this by pre-converting batches to JAX arrays and sharding them directly to device, eliminating the host-side copy that would stall the training step. Batches arrive ready to feed into training loops without intermediate NumPy conversions.

The API also handles ragged final batches—those that don't divide evenly by batch size—with explicit options to drop, pad, or raise an error, preventing cryptic shape mismatches deep into long runs. Developers can apply it as input to JaxTrainer or standalone for offline batch inference across large datasets on a TPU slice. The demonstrate both dataset preparation and batch inference workflows.