Suppose a restaurant’s voice agent asks for your order. You say, “One large pizza,” and pause.
The agent has to decide whether that pause belongs to the end of your order or the middle of it. You may be finished. You may also be about to say, “and garlic bread.” At the instant you pause, the words heard so far can be identical in both conversations.
An agent that answers immediately will feel quick when the order is complete and rude when it is not, while an agent that always waits will avoid some interruptions at the cost of placing a pocket of dead air before every reply.
That pocket can begin before the system has written a single word of its answer.
The first clock starts at your last word
A basic detector listens for the presence of speech. When the sound of a person speaking stops, it starts counting silence. After the configured interval has passed, the system treats the turn as complete and lets the rest of the reply pipeline proceed.
Speech services often expose this wait as an endpointing or silence setting. Deepgram’s streaming transcription documentation, for example, lets a developer choose how many milliseconds of silence will cause a segment to be finalized. OpenAI’s Realtime API exposes a silence_duration_ms setting for its server-side voice activity detector. Its documentation gives the tradeoff plainly: a shorter value can make the response begin sooner, while also making the model more likely to jump into a short pause.
The delay is therefore not always a slow model struggling with an answer. Some of it may be a deliberate period in which the system is waiting for evidence that the human has finished.
Silence alone is a crude source of evidence because people pause to remember a date, breathe, choose a word or check something on screen. A duration that feels acceptable in a drive-through order can feel impatient in a medical intake call, so one global timeout cannot account for the purpose of every pause.
The words help, but they leave something behind
A semantic turn detector also considers what has been said because “I need a table for” sounds unfinished even after a pause, while “That is everything” sounds much closer to a handoff.
OpenAI’s Realtime API calls this option semantic voice activity detection. Instead of applying one fixed silence duration to every pause, it estimates whether the thought is complete and adjusts how eagerly the system responds.
Text still does not contain the whole decision. In the pizza order, one speaker can lower their pitch on “pizza” as if closing the sentence, while another can hold or lift it so the same word sounds like the order will continue. Rhythm, pitch and intonation carry information that disappears when speech becomes a transcript.
LiveKit described this limitation while releasing an audio-based turn detector in June 2026. Its documented architecture processes the current speech through two paths, one for what the person is saying and another for timing and delivery, and combines them into an end-of-turn prediction. The company also released a benchmark and datasets, but its performance claims remain the developer’s evaluation of its own system and this article does not reproduce them.
The useful architectural move is narrower. If a decision depends partly on how a sentence was spoken, passing only its transcript to the detector has already discarded relevant evidence.
A turn detector cannot remove every pause
Once the system marks your order complete, other work may still be waiting.
In a common cascaded voice agent, speech recognition finalizes the words before a language model begins the answer and perhaps calls a menu database or payment service. A speech generator then waits for enough text to produce natural audio, which still has to reach the caller and begin playing.
Some of those stages can overlap because a system may start preparing an answer from a partial transcript before the turn is confirmed. A realtime speech model may combine understanding and generation instead of calling separate speech-to-text, language and text-to-speech services, while OpenAI has described a full-duplex design that removes a separate turn detector from the audio response path. The exact sequence therefore depends on the product.
The debugging principle survives those differences: mark the events that your system actually emits and stop treating the silence as one indivisible number.
Put five timestamps on one slow turn
Start with a recorded test call for which you have permission to retain audio and logs, choose one reply that felt late and mark these times:
| Mark | Event |
|---|---|
t0 | The user’s final audible word ends |
t1 | The system declares the user’s turn complete |
t2 | The response model produces its first output, or the first required tool begins |
t3 | The speech generator produces its first audio |
t4 | The caller actually hears that audio begin |
The full period of dead air is t4 - t0. The smaller intervals give it an address.
If t1 - t0 dominates, inspect turn detection, endpointing and final-transcript behavior. If t2 - t1 dominates, inspect the response model, prompt size and tools. If the gap sits between t2 and t3, inspect speech-generation buffering and time to first audio. Cartesia’s streaming documentation gives one concrete example of that last problem: incomplete text chunks can be held until the service receives a clearer sentence boundary or reaches its buffer limit. A final gap between t3 and t4 points toward transport, playback or telephony.
The event names will differ across platforms because LiveKit’s current observability tools expose per-turn and per-stage timings for several of these components, while other frameworks require application-level timestamps. The labels can vary as long as they preserve the order of events well enough to show where the time went.
Run the same request several times before changing anything. A consistently long turn-completion interval suggests a different problem from one occasional tool call that stalls. Keep the input, model and region stable while comparing a setting. Otherwise several variables move and the faster call tells you very little.
On the next slow call, resist the urge to replace the voice or shorten the prompt first, and instead open the trace at t0 because the first unusually distant timestamp after it is where the investigation begins.
Sources
- LiveKit: Understand and improve voice agent latency, 13 April 2026.
- LiveKit: Solving end-of-turn detection, 17 June 2026.
- Deepgram: Endpointing, accessed 7 September 2026.
- OpenAI Realtime API reference, accessed 7 September 2026.
- OpenAI: How we built a realtime system for responsive voice AI in six months, 3 August 2026.
- Cartesia: Stream inputs using continuations, accessed 7 September 2026.