Key takeaways
- ✓A three-model voice agent spends its time in four places: the wait for silence, speech-to-text, the language model's first sentence, and the voice's first sound. The target was first sound within 1 second. None of 20 live calls made it: even with no stalls, the typical steps add up to about 2.6 s (0.5 + 0.64 + 0.86 + 0.57).
- ✓Report the slowest call, not just the typical one. The streamed voice sent no sound at all on 8 of 20 calls: a dead line. A backup request (no sound after 1 s → ask a second voice service too) ended every stall, but the typical call took 4.3 s and the slowest 6.4 s, because the backup starts after a full second of nothing and is slower to start.
- ✓Deciding the caller has finished is a dial with two bad ends. Waiting for 0.5 s of silence cut off 5 of 6 callers who paused mid-sentence; 1.2 s cut off none, and added 1.2 s to every reply. Asking a model whether the words sound finished cut off 1 of 6, in every run.
- ✓One model that hears audio and speaks audio (gpt-realtime-mini) had a typical first sound of about 1.0 s against 3.9 s for the three models: about 4× faster, and its slowest call (1.3 s) beat their fastest (2.1 s). Its clock starts once the caller's audio is sent; the three models' clock includes sending it.
- ✓Fast isn't right. Asked about a return five weeks after purchase, with a 30-day limit, the one model told the caller they were fine in 3 of 3 runs; the three models stated the 30-day rule and said "not sure" twice. Three models give you text at every step to log and check; one model gives speed.
A voice agent has about one second. In conversation, people usually reply well under a second after the other person stops; past a second, the line feels dead. A voice agent built from three models in a row has to listen, think and speak inside that second. This chapter covers where the time goes, why the slowest call matters more than the typical one, how the agent decides the caller has finished, and how one model that hears and speaks compares with three.
The examples come from Lab 08 (Multimodal & Voice), scripts 06_voice_agent.py and 07_speech_to_speech.py, recorded live with the lab's own functions and prompts. The agent is the phone line for a made-up store, Northwind Supplies, answering from a short list of facts: opening hours, a 30-day return limit, free shipping over $50, one store in Seattle. The callers are synthetic voices (gpt-4o-mini-tts), not people. Every time comes from one laptop on one day, so treat them as one measurement, not a benchmark. The lab also covers interruptions and a tool call (at the end of this chapter); word error rate is covered in the Voice AI track.
Where the second goes
The usual design, a cascade, is three models in a row:
- Speech-to-text (
gpt-4o-mini-transcribe) writes down the caller's words. - A language model (
gpt-5.4-nano) writes the reply. - Text-to-speech (
gpt-4o-mini-tts) says it out loud.
Before any of them, the agent has to decide the caller has finished. Here it waits for half a second of silence. The measure that counts is voice-to-voice latency: from the caller's last word to the agent's first sound.
caller stops → wait for silence → speech-to-text → language model (first sentence) → voice (first sound) → caller hears
Streaming overlaps the steps. The agent doesn't wait for the whole reply: it sends the first sentence to the voice as soon as it's written, and plays the first audio as soon as it arrives. So the times that count are the language model's first sentence and the voice's first sound, not the whole reply or the whole audio file.
Here is one call, "What time do you open on Saturday?", step by step:
| Step | Seconds |
|---|---|
| Wait for silence (fixed) | 0.50 |
| Speech-to-text | 0.82 |
| Language model, first sentence | 0.73 |
| First voice service: no sound yet | 1.00 |
| Backup voice service, first sound | 0.81 |
| Silence the caller heard | 3.86 |
Pick a call, and switch between waiting for each step, streaming, and streaming with a backup voice:
no sound after 1 s → also ask a second voice service
Call 3
Caller: “What time do you open on Saturday?”
First sentence of the reply: “We’re open on Saturdays from 8:00 a.m.”
- Wait for silence
- 0.50 s
- Speech-to-text
- 0.82 s
- Language model: first sentence
- 0.73 s
- First voice: silent
- 1.00 s
- Backup voice
- 0.81 s
- Silence the caller hears
- 3.86 s
The wait for silence is a fixed 0.5 s; the other steps were timed live. Each voice setup was a separate request for the same sentence, so a call can stall one way and not another.
Twenty calls, none under a second
Four questions, five runs each: 20 calls. None got its first sound within a second, whichever way the voice was used.
The typical (median) steps add up to about 2.6 seconds even when nothing goes wrong: 0.5 s waiting for silence, 0.64 s speech-to-text, 0.86 s to the language model's first sentence, and 0.57 s to the voice's first sound when it didn't stall. The fastest streamed call took 2.0 s. The budget is gone before the agent says a word.
Network time moves these numbers a lot. The lab's own run, on another day, had speech-to-text alone at 1.6 s and a median of 5.2 s with the backup voice. The point isn't one total; it's that a cascade over public APIs, from a home connection, has no room for a second. The usual next steps are speech-to-text that transcribes while the caller is still talking, and running the models closer to the caller.
Report the slowest call
Sometimes a step doesn't slow down; it stops. On the day of the recording, the voice service often sent nothing at all: a stall. A request with no sound within the 30-second timeout counts as stalled.
| How the voice was used | Calls with no sound (of 20) |
|---|---|
| Wait for the whole audio file | 12 |
| Stream the first sentence | 8 |
| Stream, plus a backup voice after 1 s | 0 |
To a caller, a stall is a dead line, and one dead line ends a call. A median says nothing about it, so report the slowest call and the stall count next to the typical one.
The fix is a backup request (also called a hedged request): if the first voice service has sent no sound after 1 second, ask a second one too (tts-1), and play whichever starts first. The lab's tts_hedged() does exactly this. With it, no call stalled, and the backup won on 16 of 20 calls. But it isn't free: the backup is only asked after a full second of silence, and it was slower to start (a median of 1.45 s after being asked, against 0.57 s for the first voice when it worked). So with the backup, the typical call took 4.3 s and the slowest 6.4 s: slow, but never silent.
The 1-second trigger is a choice. Earlier means fewer seconds lost, and more paid-for requests you throw away.
Has the caller finished?
The half-second wait at the start is end-of-turn detection (endpointing). A voice activity detector marks which moments contain speech, and the agent replies after some amount of silence. The catch: people pause in the middle of a sentence. "My order number is…" pause… "4817."
The lab tests this with six synthetic callers, each with a set pause in the middle (0.4 to 1.0 s):
| Wait for this much silence | Callers cut off (of 6) |
|---|---|
| 0.4 s | 6 |
| 0.5 s | 5 |
| 0.8 s | 2 |
| 1.0 s | 1 |
| 1.2 s | 0 |
A fixed wait is one dial with two bad ends. Short cuts people off when they pause to think. Long cuts nobody off, but then every reply waits 1.2 s after the caller really finished, more than the whole budget.
Wait for this much silence, then reply
Or check the words first: short wait if finished, 1.2 s if not
“My order number ispause 0.6 s4817.”
cut off“I'd like to returnpause 0.4 sthe blue jacket, please.”
replied 0.5 s after the end“Can you checkpause 0.8 sif it has shipped yet?”
cut off“My email ispause 1 sdana dot whitfield at example dot com.”
cut off“Um, sopause 0.5 swhat are your opening hours?”
cut off“Yes.pause 0.7 sAnd can you also send me the receipt?”
cut off
The words check asks gpt-5.4-nano “Have they finished their turn?” Its own time wasn't measured. Fixed waits are computed once from the clips; the words check was asked three times.
Check the words
Silence alone can't tell a pause from an ending. The words can help. Before replying, the agent asks a language model: "Have they finished their turn, or are they likely to keep talking? Reply COMPLETE or INCOMPLETE only." If the words sound finished, it waits the short 0.5 s; if not, the long 1.2 s.
"My order number is" isn't finished, so the agent waited. In every run, the model judged all five unfinished pauses as unfinished. That cut off 1 of 6 callers instead of 5, in all three runs. The one it missed said "Yes.", paused, and then went on: "And can you also send me the receipt?"
The wait after a finished turn depends on what the model says at the end. In runs 1 and 3 it averaged 0.64 s. In run 2 it called four of the five finished callers unfinished, so they waited the full 1.2 s. And "Um, so… what are your opening hours?" was judged unfinished at the end in all three runs. Asking the model takes time too, which wasn't measured here; production systems use a small, fast turn-detection model that runs during the silence.
A shorter "finished" wait of 0.3 s did worse: it cut off 3, 1 and 3 callers, because it sometimes fired at short gaps between words in the second half, as with "if it has shipped yet?" and the spelled-out email address.
One model that hears and speaks
The other design is one speech-to-speech model: it takes the caller's audio in and sends audio out, with no handoffs. Here, OpenAI's gpt-realtime-mini over the Realtime API. A new test, with five questions and three runs each, ran both designs on the same spoken questions, each with the same 0.5 s wait:
| Design | Typical first sound | Fastest–slowest | Under 1 s (of 15) |
|---|---|---|---|
| Three models + backup voice | 3.9 s | 2.1–6.8 s | 0 |
One model (gpt-realtime-mini) |
0.97 s | 0.85–1.32 s | 8 |
About 4× faster, and steadier: every one-model call beat every three-model call. One catch in the comparison: the one model's clock starts once the caller's audio has been sent, while the three models' clock includes uploading the audio to speech-to-text. The three-model median was 4.3 s in the first test and 3.9 s in this one, with the same design: live times vary. The lab's own run found the same gap (1,073 ms against 3,953 ms).
One model (gpt-realtime-mini) · 0.87 s to first sound
Caller: “What time do you open on Saturday?”
Reply: “We open at 8 a.m. on Saturday.”
Both include the 0.5 s wait for silence. The one model's clock starts once the caller's audio has been sent; the three models' clock includes uploading it to speech-to-text.
Fast isn't right · “Can I return something I bought five weeks ago?” · the limit is 30 days
One model (gpt-realtime-mini)
- run 1 · Yes, you can return items within 30 days with a receipt. That should still be fine. Just bring it back to the store in Seattle during our open hours.
- run 2 · Sure, we accept returns within 30 days with a receipt, so you should be all set. Just bring it back to our store in Seattle.
- run 3 · Sure, as long as you have the receipt, returns are accepted within 30 days, so you're still within the window.
Three models
- run 1 · Yes—returns are accepted within 30 days with a receipt. If it’s five weeks ago, I’m not sure it’s within that window.
- run 2 · Yes—if it was purchased at Northwind Supplies, you can return it within 30 days with a receipt. We’re open Monday to Saturday from 8am to 6pm, and closed Sunday.
- run 3 · Yes—if it’s within 30 days and you have a receipt. If it’s been five weeks, I’m not sure it qualifies.
Fast isn't right
The lab's check for each answer was whether it contained the right key fact, and by that check both designs got all 15 right. But one question needed more than a fact: "Can I return something I bought five weeks ago?" The limit is 30 days.
- The one model told the caller they were fine, in all three runs: "That should still be fine", "you should be all set", "you're still within the window".
- The three models stated the 30-day rule every time, and twice added that they weren't sure five weeks qualified. Neither design gave a clear no; every reply started with "Yes" or "Sure".
In this test, only the one model told the caller they were fine. Three replies per design is a tiny sample, but it shows what speed can hide. That's the trade-off between the designs:
| Three models in a row | One speech-to-speech model | |
|---|---|---|
| Speed | Three handoffs add up | Fewer handoffs, faster first sound |
| Control | Text at every step: log it, check it against policy, test it, pick any language model | Audio in, audio out; text only as a side transcript |
| Tone of voice | Lost: a transcript has no tone | It gets the audio |
| Cost per 1,000 turns (lab's estimate, this recording) | about $1.91 | about $2.56 |
On tone, the recording has a small test: the same words said annoyed and said happy, three runs each. The agent was told to open with "I'm sorry" or "Great". Speech-to-text wrote identical words for both tones, so the three models can't tell them apart. For "Well, the package arrived today.", the one model opened with "I'm sorry" in 2 of 3 annoyed runs and "Great" in 3 of 3 happy ones. For "Okay, so I'll just wait then.", it made no difference. Possible, not reliable.
For a support line that has to follow policy, the lab's advice is to start with a well-streamed cascade and use speech-to-speech where natural conversation is the product. If you use one model, check its answers, for example by running checks on its side transcript.
What Lab 08 also covers
The video sticks to time. Script 6 has two more exercises, with numbers from the lab's own run:
- Interruptions (barge-in). While the agent talks, the caller makes a sound. Stopping after 40 ms of speech stopped for "mm-hmm", "okay", a cough and a door; stopping after 400 ms caught a real "Wait, sorry…" only after 1.1 s, and still stopped for a long "mm-hmm". No length works: telling a backchannel from an interruption needs the words. Then cut the agent's message in the chat history to what the caller actually heard, using word timestamps.
- A tool call. "Where is my order 4817?" needed two language-model rounds and a 300 ms tool: first sound after 5.4 s. Playing a pre-recorded "One moment, let me check" when the tool call starts put first sound at 2.5 s. The answer isn't faster; the silence is shorter.
Caveats
- One laptop, one day, home Wi-Fi. Network time moved the totals a lot between runs (the lab's own runs were slower). Your location and providers will give other numbers; the shape (no room for a second, stalls dominate the tail) is the lesson.
- Synthetic callers. Every caller is a text-to-speech voice with set pauses. Real callers have accents, noise and bad lines.
- Each voice setup was a separate request. The whole-file, streamed and backup voices were asked one after the other for the same sentence, so a call can stall one way and not another.
- The wait for silence is fixed, not measured. The 0.5 s is added to every call; the end-of-turn tests use clips with exactly placed pauses, and the words check's own time wasn't measured.
- Different clocks. The one model's time starts after its audio is sent; the three models' time includes the upload.
- Small samples. 20 and 15 calls, six callers, three replies per design for the return question.
The checklist
- Time every step separately: the wait for silence, speech-to-text, the language model's first sentence, the voice's first sound.
- Stream between steps: first sentence to the voice, first audio to the caller.
- Report the slowest call and the stall count, not just the median.
- Add a timeout and a backup request to any step that can stall, and measure what the backup costs in time.
- Test end of turn on callers who pause mid-sentence, and check the words, not only the silence.
- Check the answers, not just the speed, especially for a speech-to-speech model.
Practice: time the calls
Add up a call's silence from its recorded steps, summarise 20 live calls with and without the backup voice (median, slowest, stalls), write the backup request's first-of-two logic, and the lab's end-of-turn detector with and without checking the words. The tests use the 20 recorded calls: 3.86 s for the walked call, 8 stalls without the backup, and 4.3 s and 6.4 s with it.
Quiz
1.What does a three-model voice agent's silence consist of, from the caller's last word to the first sound?
2.What does streaming change in a voice agent?
3.The streamed voice stalled on 8 of 20 calls. Why does that matter more than the median?
4.A fixed 0.5 s wait for silence cut off 5 of 6 callers who paused mid-sentence. Why not just wait 1.2 s?
5.How did checking the words decide when to reply, and what did it still miss?
6.One speech-to-speech model was about 4× faster. Why might you still pick three models in a row?
Check your understanding
Try each question before opening its answer.
Where does a three-model voice agent's time go?
Four places: the wait for silence (here a fixed 0.5 s), speech-to-text, the language model up to its first sentence, and the voice up to its first sound. Typical steps added up to about 2.6 s with no stalls, so none of 20 calls made the 1-second target.
Why report the slowest call, not just the typical one?
A stalled step means silence: the streamed voice sent nothing on 8 of 20 calls. A median hides that, and one dead line ends a call. Report the slowest call and the stall count next to the median.
What does a backup request do, and what does it cost?
If the first voice service has sent no sound after 1 s, it also asks a second one and plays whichever starts first. It ended every stall, but the typical call took 4.3 s and the slowest 6.4 s, because the backup starts after a full second and was slower to start.
Why is a fixed wait for silence a bad dial?
Short waits cut off people who pause mid-sentence (0.5 s cut off 5 of 6); long ones add their full length to every reply (1.2 s cut off nobody, and every reply waited 1.2 s). No setting is fast and safe.
How does checking the words help, and where does it still fail?
A model judges whether the words so far sound finished; the agent waits 0.5 s if yes, 1.2 s if no. It cut off 1 of 6 instead of 5, in every run. It still cut off the caller who said "Yes." and kept talking, and its own time has to fit in the budget too.
Three models or one: which would you pick?
One speech-to-speech model was about 4× faster (about 1.0 s vs 3.9 s), but only it told a caller a five-week-old return was fine. Three models give text at every step to log, check and test. For a line that must follow policy, start with a well-streamed cascade; if you use one model, check its answers.
Next up: Chapter 29 starts the serving series: running an open model on your own machine, why it's slow, and what shrinking it to fewer bits buys and costs.