AI
OpenAI's GPT-Live Splits Inference and Voice to Slash Latency
The company's new real-time voice system isolates media processing and AI inference from slower application logic to keep conversations responsive.
InfoQ
OpenAI has revealed the architectural blueprint behind GPT-Live, its continuous voice interaction system, showing how the company tackled a fundamental challenge in real-time AI: keeping conversations snappy even when other operations lag.
The Split Between Speed and Everything Else#
The design centers on a sharp division of labor. A live path handles the fastest, most time-sensitive work—the media pipeline and AI inference—and runs in tight, optimized loops. Everything else—delegating tasks, tool use, data persistence, and broader application logic—happens asynchronously behind a remote procedure call (RPC) boundary. This separation lets engineers focus optimization energy on what matters most without performance regressions from slower, less predictable components bleeding into the voice interaction.
As OpenAI detailed in their engineering account, the approach makes it possible to offload tricky pieces like delegation to frontier models and feeding voice data to safety systems, then optimize those pieces in isolation rather than within the critical path itself.
Sessions Stay Put, But Can Move#
Scaling a stateful, continuous conversation posed another puzzle. Each session gets its own dedicated, stateful inference instance that reserves capacity. Yet the architecture remains flexible: new sessions land on instances with available room, and existing sessions can migrate in real time when an instance is draining or a conversation approaches its context limit. This keeps infrastructure responsive to demand without sacrificing the session continuity that continuous voice requires.
WebRTC Gets Tweaked, Not Replaced#
Rather than rip out WebRTC for newer transport standards like RTP over QUIC, OpenAI extended the battle-tested media stack with targeted improvements. The company introduced WARP (WebRTC Abridged Roundtrip Protocol), simplifying the handshake, and added Instant Connect to slash startup latency. These refinements—each deployable independently—preserve what works in WebRTC (error recovery, low-latency fundamentals, congestion control) while trimming excess ceremony.
