New approach lets voice agents respond instantly while tapping powerful reasoning models

Voice assistants have long faced a difficult choice: deploy nimble models that respond instantly but lack depth, or use powerful foundation models that understand complex queries but force users to wait. A team from arXiv has found a middle ground with a technique called conversational infill, which allows lightweight models to generate immediate responses while seamlessly weaving in insights from slower, more capable AI systems as they stream in during the conversation.

How conversational infill works#

The approach pairs a small "talker" model—trained to generate natural speech quickly—with an external "reasoner" model that handles heavier lifting like retrieval, tool use, and complex reasoning. The talker produces an initial response fast enough to meet conversational latency demands (milliseconds), then intelligently integrates streamed knowledge from the reasoner as it arrives, keeping the conversation flowing naturally without awkward pauses or jarring pivots.

The researchers, led by Zachary Englhardt, trained a system called ConvFill on models ranging from 135 million to 1.7 billion parameters using a synthetic dataset of 290,571 examples across six different domains. The training teaches these small models not just to answer questions, but to anticipate and gracefully incorporate additional information delivered mid-response.

Strong real-world performance#

Testing revealed that ConvFill closes the accuracy gap between small and frontier models to within 6.3 percentage points while maintaining millisecond-level time-to-first-response. In a live user study with 18 participants running talker deployments on an Apple M2 chip, users rated ConvFill as comparable to frontier models overall. They especially preferred it for retrieval-heavy tasks and consistently ranked it as significantly more responsive than traditional approaches.

Why it matters#

The technique opens a new trade-off point on the latency-capability frontier. Voice agents no longer have to choose between speed and smarts—they can deliver both by pooling the strengths of small, fast models with powerful but slower foundation models. This is particularly important for on-device voice interactions, where millisecond responsiveness is essential but reasoning depth matters for useful answers.

Code, trained models, and the synthetic training dataset are available online.