People forgive a voice agent for many things. They do not forgive a pause. A two-second gap before every reply reads as broken in a way that a slightly wrong answer never does, because humans use timing to decide whether the other party is listening.
The useful thing about latency is that it is made of four separate delays that are usually confused with each other. Fixing the wrong one is why teams grind for weeks with no perceptible improvement.
1. End-of-speech detection
Before anything can happen, something has to decide the person has stopped talking. That decision is made by counting frames of silence — and at 8 kHz telephone audio a frame is 64 milliseconds, so a detector waiting 18 frames has already spent 1.15 seconds before a single word reaches the language model.
This is almost always the biggest single number, and almost always the last one people look at. Reducing it is the cheapest win available. Reduce it too far and you get the opposite failure: the agent cuts people off mid-sentence.
2. The silence window your application adds
On top of the recogniser's own detection, most systems wait a further moment to be sure. The mistake is using one number for every utterance. "Yes" and "I have five years of experience working mostly on Node, though recently more React" deserve different treatment: the first is unambiguously finished, the second may have a thinking pause in the middle.
Use a short window after short confirmations and a longer one after substantive answers. You get snappy responses to "okay" without chopping anyone off mid-thought.
3. Time to the model's first token
The model has to start producing words. Typically 300–900 ms depending on model, prompt size and region.
Two things move it. Keep the static part of the prompt cached, so the system prompt and knowledge base are not re-processed every turn — that alone routinely cuts prompt cost by 85–95%. And start generating during the silence window rather than after it, discarding the result if the person keeps talking. Nothing plays early, and the model's thinking time overlaps a wait you were doing anyway.
4. Time to the first audio chunk
Speech synthesis has to return something playable. A good streaming TTS delivers a first chunk in around 200 ms and is remarkably consistent — in practice this is the most stable link in the chain and rarely the problem.
The trick is not making TTS faster but starting it sooner: synthesise sentence by sentence as the model produces them, rather than waiting for the full reply. The first sentence is playing while the third is still being written.
What the numbers look like together
A tuned pipeline on real calls looks roughly like this:
- End-of-speech detection: ~640 ms
- Application silence window: 250 ms after a confirmation, 700 ms after an answer
- Model first token: 350–950 ms, largely hidden if you speculate during the window
- First audio chunk: ~200 ms
Total felt gap: about 1.9 seconds, and roughly the same whether the person said "yes" or gave a full answer — which is the actual goal. Consistency matters more than the raw number, because an agent that sometimes replies instantly and sometimes takes three seconds feels less reliable than one that always takes two.
The thing that is usually not the problem
Response length. A reply that takes eleven seconds to speak is a much worse experience than one that starts 300 ms later — and it is fixed in the prompt, not the pipeline. Before tuning any of the above, check how long your agent talks for. If the answer is "ten seconds", start there.