Every voice AI demo leads with the voice. It is the easiest thing to show and the easiest thing to be impressed by. It is also not what decides whether the call works.
The thing that decides it is what happens in the half-second after someone starts talking while the agent is still speaking.
Why it matters so much
Real phone conversations are full of interruption. People cut in to correct a wrong assumption, to answer before the question finishes, to say "sorry, who is this?". This is not rude — it is how spoken conversation works, and it happens on a large share of real calls.
An agent that cannot be interrupted forces the human to wait. And because the human does not know how long the agent intends to speak, they do one of three things: talk over it anyway and get ignored, go quiet and lose their place, or hang up. All three are worse than a slightly robotic voice.
Put plainly: a great voice that steamrolls people produces worse calls than a mediocre voice that stops when spoken to.
What has to happen, and how fast
Proper barge-in is not one action. Within a few hundred milliseconds of detecting speech, the system has to:
- Stop the audio already playing to the caller.
- Abort the speech synthesis still being generated.
- Cancel the language model call in flight, so its reply never plays.
- Flush audio already buffered at the telephony provider — this is the one people miss, and skipping it means the agent keeps talking for a second after it "stopped".
- Start listening, treating what the person is saying as the new turn.
Miss any of those and you get the failure everyone has heard: the agent goes quiet, then finishes its old sentence anyway.
The hard part: echo
Here is what makes this genuinely difficult. The agent's own voice comes back down the line — through the speaker, off the room, into the microphone. To a naive detector that is indistinguishable from the caller speaking, so the agent interrupts itself, stops, hears silence, starts again, and stammers.
The practical defence is a short protection window: speech detected within a few hundred milliseconds of the agent's own audio starting is treated as echo and ignored; anything after it is a real interruption and kills the agent's voice instantly.
Tuning that window is a genuine trade-off. Too short and the agent stutters over its own echo. Too long and a real interruption in the first moments of a sentence gets ignored, which is exactly when people interrupt most.
How to evaluate it
You cannot assess this from a demo video, because demos are recorded by people who wait their turn. Do this instead, on your own phone:
- Let the agent start its greeting, then talk over it after about one second. It should stop mid-word.
- Ask a question while it is answering something else. It should abandon the old answer, not finish it.
- Put it on speakerphone in a room with hard walls and let it talk for twenty seconds. It should not interrupt itself.
- Say "mm-hm" while it speaks. Ideally it continues — that is a backchannel, not an interruption.
The fourth is the one nobody passes cleanly, and it is worth knowing how a system behaves rather than assuming.
Create a workspace and try all four against your own number. Ten minutes of deliberately talking over an agent tells you more than any recorded demo.