Testing text-to-speech

So far every chapter has evaluated machines that listen. This one evaluates a machine that speaks. Text-to-speech (TTS) looks easier to test, because you choose the text, so you always have the right answer. But "good speech" has several separate parts, and each needs its own measure. One line from the lab shows why: asked to read "Use the password x7-Q9 for access.", a voice said "x seven, negative nine, quetzals." The dash became "negative", and Q became the currency of Guatemala. It sounded perfectly natural (a naturalness score even liked it a little more than this voice's correct readings, on average), and it was wrong.

The lab tests six voices, three from Deepgram (Aura-2 thalia, andromeda, apollo) and three from OpenAI (coral and ash on gpt-4o-mini-tts, and alloy on the older tts-1).

Part	Question	Automatic measure
Intelligibility	Can you understand the words?	Round-trip WER
Correctness	Did it say what the text meant?	Text-normalization tests
Naturalness	Does it sound like a person?	Predicted MOS
Speed	How soon does audio start?	Time to first byte, time to first speech
Clean audio	No glitches, odd pauses, distortion?	Glitch scans

A voice can be strong on one and weak on another, and every automatic measure has blind spots, so a TTS evaluation uses several.

Round trip, with two instruments

The simplest automatic check: synthesize a sentence, transcribe it, and compute WER against the text you sent. The catch is that the recognizer has errors of its own. So use two independent recognizers: an error both make in the same way was probably in the audio; an error only one makes was probably that recognizer's.

Two lessons from the lab. First, a ceiling: on 30 plain sentences, one recognizer heard all six voices perfectly, and no error was made by both. Plain sentences can't separate modern voices, so the effort belongs elsewhere. Second, the instrument failed, not the voice: one recognizer returned an empty transcript for a clean clip, every time, and adding 0.3 s of silence fixed it. One recognizer alone would have blamed the voice.

The most common bug: reading written text

Before speaking, a TTS system has to decide how to say written text: "$1,250.50", "Dr.", "St." (Street or Saint?), "3/4" (a fraction or a date?), "14:00", "support@acme.io". This is text normalization, and it's the most common real TTS bug. For a voice agent reading back a total, a date or a phone number, it is the product.

The lab tested 40 items in 14 categories. The same voices that read plain sentences perfectly got 3 to 8 of them wrong. Try a few yourself:

Two practical notes. "Unsure" is a real category: when the two recognizers disagree, someone has to listen before filing a bug, because the judge is two machines with their own formatting habits. And accept the written form back too: recognizers write some readings as digits, so compare digit strings as digit runs (the same trick as Chapter 2's entity checks).

Does it sound natural?

MOS (mean opinion score) is listeners rating clips from 1 (bad) to 5 (excellent). People are slow and expensive, so a predicted MOS model estimates the score from the audio; the lab uses one called UTMOS. Treat the predictor as an instrument and check three things before trusting it: an anchor (what does real human speech score?), a sanity check (does the score fall on obviously damaged audio?), and its blind spot.

The lab's predictor passed the sanity check: noise at 0 dB dropped a voice from 4.43 to 1.28. But every synthetic voice scored at or above real human read speech, which says as much about the predictor (trained on ratings of synthetic speech) as about the voices. And its blind spot matters: readings judged wrong scored the same as correct ones. "Negative nine quetzals" sounds perfectly natural.

How soon does the voice start?

For a voice agent, speed is part of quality. Measure:

• Time to first byte (TTFB): request sent → first audio bytes arrive.
• Time to first speech: TTFB plus any silence at the start of the audio. This is what the listener actually waits for.
• Generation ÷ audio length: below 1, audio is produced faster than it plays, so streaming playback won't stutter.

Switch the widget above to "When is the first word heard?". By first byte, aura-2-andromeda was twice as fast as the OpenAI voices; by first word heard, they tied, because it starts with 0.38 s of silence. And look at the tail. In the main run, one request in ten to gpt-4o-mini-tts waited 5.5 s or more for its first audio (coral), and 12 s or more (ash), while the median was about 0.65 s. Two requests took over 10 minutes with the connection open. For a voice agent, the tail is the silence callers remember: use a timeout and a fallback. (The lab's client now times out at 60 s.)

Streaming matters too: playing audio as it arrives, instead of waiting for the whole file, saved 0.25–1.3 s per reply.

Find glitches without listening to everything

You can't listen to thousands of clips, but you can scan them: speaking rate per clip, long pauses, clipping, leading silence. The lab flagged 14 clips worth a listen, like a phrase spoken at 5.5 words per second. One lesson from building it: the voice activity detector's default settings are built for transcription (0.4 s padding, short pauses merged), so they reported exactly zero leading silence on every clip. For timing, turn the padding off. The same bug had quietly skewed Chapters 3 and 5, and fixing it corrected their numbers.

Practice: score a voice

Implement the pieces: two-recognizer blame, a reading checker that matches spelled-out and written digits, leading-silence detection, and time to first speech. The tests use the lab's real bugs ("negative nine quetzals", the lost cents, the lost "acme") and its real timings.

Quiz

Check your understanding

Try each question before opening its answer.

What are the separate parts of TTS quality, and how is each measured?

Measure intelligibility with round-trip WER, reading correctness with normalization tests, naturalness with MOS/listening, speed with time to first byte and first speech, and clean audio with glitch scans.

How does round-trip WER work, what can't it tell you, and why use two recognizers?

Transcribe synthesized speech and compare it with the input text. It does not measure naturalness and inherits recognizer errors; independent recognizers help distinguish audio errors from one instrument's mistakes.

Why is text normalization a common TTS bug, and how would you test it?

Written forms such as dates, abbreviations, currency, and codes have multiple readings. Build a categorized test set with accepted readings, compare the output through recognizers, and listen to confirm failures.

What's MOS, and how do you decide whether to trust a MOS predictor?

MOS is a 1–5 listener rating; a predictor estimates it. Check an anchor against human speech, a sanity response to damaged audio, agreement with human ratings, and known blind spots.

What's the difference between time to first byte and what the user waits for?

TTFB marks when audio bytes arrive. The listener waits through any leading silence and playback buffering until the first speech, so measure time to first speech too.

How do you find glitches in thousands of clips without listening to all of them?

Automatically scan for unusual speaking rate, long pauses, leading silence, clipping, and recognizer disagreements. Treat flags as a prioritized listening queue, not a verdict.

Next up: turning all of this into a release process. Chapter 7 builds quality gates in CI, including one that blocked a better model, and was right.