A vendor page says 75 ms. Your voice agent, wired to that vendor, makes callers wait over a second for the first syllable. Nobody lied, exactly. That 75 ms is time-to-first-audio for one isolated request on a short string, measured near the vendor’s servers. Your caller hears something else: end-of-turn detection, plus the LLM’s first useful phrase, plus however long your code holds text before handing it to the synthesizer, plus the network, plus a playout buffer on a phone line. The TTS model is rarely the slowest box. The glue around it usually is.
Where the milliseconds go
Most teams target roughly 800 ms from the moment the user stops talking to the moment they hear the agent. Kwindla Hultman Kramer, co-creator of Pipecat, splits that four ways at about 200 ms each: transport, STT plus endpointing, LLM, and TTS. Run those stages back to back and you’ve spent the budget before anything overlaps. The arithmetic only closes if the TTS starts speaking while the LLM is still writing.
That overlap creates the real design question: when do you hand text to the synthesizer?
An LLM streams tokens, roughly word fragments. The TTS wants text it can pronounce with sensible intonation. Hand it one word and you get flat, robotic delivery, because the model can’t see where the sentence is heading. Wait for a full paragraph and you’ve thrown away the overlap. Every voice stack makes this call in a component I’ll call the chunker, and most get it wrong by picking one fixed rule.
The common rules:
Wait for the full response. Best prosody, worst latency: time-to-first-audio becomes total LLM generation time plus synthesis.
Flush on sentence boundaries. “Sure, I can move that appointment to next Thursday afternoon.” is 11 words you must wait for before audio starts.
Flush on clause boundaries (commas, semicolons) with a minimum word count. Fast first audio, but each clause is synthesized as its own utterance, so pitch resets mid-sentence. Listeners hear it.
Adaptive: flush the first chunk as early as possible, then grow chunks as the playout buffer fills.
Adaptive is what I’d ship, and the reason is the playout buffer. Once audio starts, the speaker drains it at exactly real time. With two seconds of speech buffered, the next chunk can wait for a full sentence and nobody notices. With 200 ms buffered, you take whatever text you have. Chunk size should be a function of buffer headroom, not a constant.
The failure that hurts most isn’t slow first audio. It’s the underrun: the buffer drains to zero mid-response because the next chunk isn’t synthesized yet. A 600 ms pause before an answer reads as thinking. A 400 ms hole mid-sentence reads as a dropped call.
Underruns have two sources. The LLM stalls (a tool call, a noisy neighbor on a shared GPU). Or the synthesizer’s real-time factor, synthesis time divided by audio duration, drifts toward 1.0 under load. Past 1.0 you produce speech slower than you play it, and the buffer can only shrink. Your time-to-first-audio dashboard won’t show any of this.


