Netflix reveals how it balances custom models with off-the-shelf inference tools

Netflix has opened up about the engineering work required to bring large language model inference into production at scale, describing how it combines open-source frameworks with its own infrastructure to serve both standard and custom models while maintaining stability.

The layered approach#

Rather than replacing its entire serving stack, Netflix built on top of its existing JVM-based platform, which continues to handle routing, feature retrieval, candidate generation, and logging. The new LLM path splits workloads by size: smaller models run directly on CPUs within the existing process, while larger requests route to dedicated GPU infrastructure. This separation lets application teams rely on a consistent interface even as the underlying execution environment changes.

Within the GPU layer, Netflix adopted vLLM for inference operations and Triton for the surrounding orchestration—model loading, batching, and scheduling. The division of responsibility is crucial: Triton controls the serving environment and hardware scheduling, while vLLM focuses on the actual inference and custom extension points.

Managing compatibility across frameworks#

Keeping two frameworks in sync proved more complex than expected. Netflix discovered that mismatched versions of Triton and vLLM can prevent deployments from starting entirely, forcing the team to test and pin compatible combinations together. This tight coupling affects reliability, as upgrading one framework requires validating a new pair before production use.

Supporting custom models added another layer of complexity. While vLLM claims Hugging Face compatibility, Netflix's own architectures didn't always align with those defaults, necessitating custom extension code within vLLM to handle proprietary designs and decoding behavior. The team also evaluated how to package models for Triton—choosing vLLM's backend over Triton's Python backend because it allowed models and infrastructure to evolve more independently.

Constrained decoding and state management#

A particularly tricky problem emerged with constrained decoding, a technique Netflix uses to force model outputs into valid formats like JSON by restricting token choices at each generation step. When vLLM pauses a request to free GPU resources and resumes it later, the decoder's internal state can drift from the actual token sequence. Netflix had to add logic to detect these gaps and rebuild the decoder state before generation continues—a reminder that abstraction layers don't eliminate underlying complexity.