Testing a voice agent end to end

A voice agent is a chain: speech-to-text, a language model with tools, and text-to-speech. A call can fail while every part looks fine on its own, so testing an agent means following each turn through the chain and finding which part lost what. In the lab's sharpest example, a caller read out a phone number ("nine one nine, five five five, oh one fifty"). Speech-to-text heard all ten digits, twice, yet the agent kept saying it heard only nine; by turn twelve, the call's limit, the caller was adding digits just to please it, and nothing was booked.

The agent books dentist appointments, and both sides of every call in the lab are AI: the caller is an LLM with a synthetic voice.

The agent, and why a chain is testable

The agent is the receptionist of a dental clinic, built as a chain of three parts:

• Deepgram Flux turns speech into text and decides when the caller has finished talking.
• An LLM (gpt-4.1-mini, pinned) decides what to say, and uses six tools to check, book, move or cancel appointments.
• Deepgram Aura-2 speaks the reply.

Every part logs what it got and what it gave, on one clock, so a failure can be traced to one part. A speech-to-speech model, which listens and speaks as one system, can be faster and more expressive, but it's a black box: you can judge its outcomes (task success, latency, turn-taking) and listen to failed calls, not blame a stage.

The caller is an LLM too, with a goal and a way of talking, speaking through TTS. Calls run in real time on an audio clock, so pauses really happen. Eight scenarios, five calls each:

Did it do the job?

Success is decided by code, from the clinic's final state: the right name, phone number, day and time, and nothing extra. Tool-call rules are checked on the log: find_appointment before reschedule_appointment, never book_appointment in a cancel call. An LLM judge is only needed for what code can't decide, like tone or invented facts; this lab needed none.

Each scenario ran twice over: with a perfect transcript (the LLM gets the caller's exact words) and with real audio (the full chain, listening).

• Every failure happened only with real audio. Something on the speech path broke it, but not necessarily speech-to-text.
• Five calls find broken scenarios; they can't rank good ones. 5/5 is anywhere from 57% to 100%. Read the failure descriptions, not the rates.
• Failures cost time before they cost the task. In the failing scenarios, audio calls took nearly twice as many caller turns as text calls.

Who lost the number?

To find the stage, check the key value at each one: did the phone number survive transcription, and did it reach the tool intact? Read the call from the opening, turn by turn:

In numbers_spoken, Flux heard the number exactly in 22 of 22 turns. The LLM passed it to the tools right only 3 times in 11. Flux writes digits as words ("nine one nine five five five zero one five zero"), and the LLM, converting them, added or dropped a digit, then told the caller "I count nine digits". In two book_simple calls the caller said the number 8 and 9 times, and the calls hit their 12-turn limit with nothing booked.

Speech-to-text wasn't perfect everywhere: four 5s in a row ("415 555…") came out as three in 11 turns, and "312" as "three hundred twelve" in 6. But the biggest failure came after a perfect transcription.

Names fail differently. "Haddad" was booked as "Hadad": both are real spellings, and sound alone can't decide. Over the noisy line, "Kowalczyk" became "Kowalski", "Kaluchick" and "Colchic". In the second call above, Flux heard the spelling correctly, letter by letter, and the LLM still booked the misheard name.

WER says little here. Caller WER was 5–19% everywhere, including scenarios that never failed. The first version of the table showed 20–27%, because "415-555-0123" against "four one five…" counted as errors: a formatting artifact, fixed by converting both sides to digits first (Chapter 2's entity checks).

Where the time goes

Voice-to-voice latency runs from the moment the caller stops talking to the moment they hear the reply. In a chain, it's a sum:

caller stops talking
  → end of turn detected         endpointing (Chapter 5)
  → LLM first token              including any tool round trips
  → first sentence complete      sent to TTS at once, not after the whole reply
  → TTS first audio byte         (Chapter 6)
  → playback buffer + any silence at the start of the audio
caller hears the reply

Over 204 agent turns:

• No reply was heard within 1 s; 9% within 1.5 s.
• The LLM is about half of the wait. The first draft of this lab guessed the turn detector; it's about a quarter.
• A tool call more than doubles the LLM's part, from 0.60 s to 1.51 s at the median (voice to voice: 1.86 s without tools, 2.75 s with). The in-memory tool takes milliseconds; the cost is a second LLM call, one to ask for the tool and one to answer with its result. A spoken "let me check", or prefetching, can hide it.
• Sending the first sentence to TTS at once works: it's ready 0.11 s after the first token.

Did the agent let the caller finish?

Speech adds one more problem: timing. In pause_in_number, the caller pauses for 1.6 seconds in the middle of the phone number, to look it up. Switch the widget above to "Pause cut-offs".

• The stricter setting doubled the wait at every turn end, and still cut off half the pauses. The agent's own behaviour matters as much: hearing "my number is 312", it can say "go on" instead of starting the booking.
• Cut-offs don't show in task success, 5/5 either way. Measure turn-taking separately.
• Across all scenarios, 62 of 266 agent turns (23%) answered a turn Flux had ended before the caller finished, and in 37 the reply started while the caller was still talking. Half the cut-off fragments were 1–3 words: a sentence end ("Yes." | "That's correct.") or a filler ("Hi, um,"). The caller is TTS, whose pauses between sentences may be longer than a person's, so check timing with real callers too.

Fix it at its stage, then re-measure everything

Reading version 1's failed calls one by one found three problems, and version 2 fixes each at its stage:

1. Digits: code, not the LLM, now turns spoken digits into a number, between Flux and the LLM.
2. Names: the prompt asks for a spelling of the last name.
3. A fake booking: in an early run, the agent booked an appointment for "test", phone 0000000000, to check availability. It took the 9:30 slot the caller wanted. The prompt now forbids test bookings: an LLM used a write tool as a probe.

• The targeted fix worked where it was aimed: numbers_spoken went from 0/5 to 4/5.
• The spelling fix moved the problem. On the noisy line, the name now failed as letters: "c" heard as "t", spelling cut off mid-name by the turn detector ("k o w a l c"), and one call booked twice under two spellings. Isolated letters over a phone line are among the hardest things to transcribe; a phonetic alphabet or a letter-by-letter read-back is the next experiment.
• Fixes interact. Converting digits early turns a cut-off number into two numbers ("312", then "5550147"). In pause_in_number, v2 asked again for "all 10 digits" 11 times, v1 6 times.
• Overall, v2 isn't clearly better. 29 to 32 of 40 is within chance with five calls per scenario. And every fix costs time: calls went from 87 s to 99 s on average, while voice-to-voice latency didn't change (2.35 s vs 2.26 s).

That loop is the job: a call fails, read the call, find the stage, fix it there, then re-measure everything.

One lesson from building it: in an early version, the caller confirmed and hung up in the same breath ("Yes, that's right, goodbye"), and the harness dropped that last line instead of letting the agent hear it. The agent was blamed for 3 failures that were the harness's. All 80 audio calls were re-run after the fix. Test the test harness too.

Practice: score the calls

Implement the pieces of the lab's scorer: turning spoken digits into a number (the check that found the lost phone number), task success from the final state and the tool log, per-stage latency for one turn, and the Wilson interval that says what 5/5 is worth. The tests use the lab's real calls: the number Flux heard, the "Hadad" booking, the cancel rules, and a real turn's timestamps.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: putting it all together. Chapter 10 builds a model release pipeline, where a candidate passed every offline check and was rolled back 85 seconds into its canary.