Test sets that match real audio

A word error rate is only as meaningful as the audio it was measured on. Chapter 1 showed how to compute WER. This chapter is about what to compute it on: test sets that look like your real traffic, and the measurements a single average hides.

Slicing test sets vs. aggregate WER — evaluating on real-traffic slices, crucial entities, and versioned datasets.

One average describes someone else's product

An overall WER mixes easy and hard audio in whatever proportion your test set happens to have. If your customers send phone calls and your test set is audiobooks, the number describes a different product. The fix is slices: subsets that share one property (noisy, phone, accented, conversational, full of numbers). Report WER per slice, and gate releases on the slices that matter (Chapter 7).

There are three ways to get test audio, and each one can fool you differently:

Use transformations to find where a model breaks, real recordings to decide which model to use, and synthetic speech for content you can't find recorded.

Find the breaking point with a noise sweep

Noise is measured as signal-to-noise ratio (SNR), in decibels: 20 dB is a quiet office, 10 dB a busy café, 5 dB a loud street, and 0 dB means speech and noise are equally loud. A noise sweep runs the same clips at each level, which shows where each model breaks. Here are six models from the lab on babble noise, other people talking:

Three things to notice:

• At 20 dB nobody notices. Between 5 and 0 dB the curves bend and the models spread apart. A single "noisy" slice at one level would miss that shape.
• The noise type matters as much as the loudness. At 0 dB, babble did 2–6 times the damage of steady pink noise. It's made of speech, so models confuse it with the speaker. One model transcribed a background voice outright: "the empty coffin remains", said by someone else in the mix.
• Same WER, different failures. At 0 dB babble, gpt-4o-transcribe had 259 deletions where the others had 40–48: on 12 of 40 clips it wrote the first sentence and stopped. Read the errors, not just the number.

Watch out for one trap. gpt-transcribe degraded 28× from clean to 0 dB, the largest ratio in the table, and was still the best model at every level. Ratios exaggerate when the starting point is tiny.

Clean audio is a floor, not a forecast

The real test is real speech. On actual earnings calls and meetings, every model got 4–25 times worse than on audiobooks. And the transformed slices didn't predict the winner: gpt-4o-transcribe ranked 2nd–3rd on synthetic noise and last on both real slices. Click through the slices and watch the ranking reshuffle:

Also notice the table microphone. Moving from a headset to one mic on the table made every model 1.7–3.9× worse on the same words, and the best headset model wasn't the best far-field model.

The words that matter

WER gives "the" and "4471-2209" the same weight. Entity accuracy counts only the words a customer cares about: account numbers, money, dates, names, drug names. Each kind gets its own comparison rule. For example, "4471-2209", "4471 2209" and "four four seven one two two zero nine" all match a digits entity, and a lost leading zero doesn't.

You can't use the WER normalizer for this, because it's wrong exactly where entities live: it turns 047 into 47 and 1-800-555-0199 into "one 800 555 199". Try the cases from the lab:

In the lab, the two metrics disagreed in both directions. 36 transcripts got every entity right but still had WER errors, mostly from number formatting. And 25 transcripts had exactly one word error, which broke the one entity that mattered: "Siobhan" → "Chavon", "lisinopril" → "lysinopril". The model rankings changed too: whisper-small.en was 4th by WER and tied for 2nd by entity accuracy.

Key-term boosting (giving the model a list of names and jargon) is the cheapest fix for names, and it worked: most missed names came back. But measure it with entity accuracy, not WER. One model's WER fell from 7.0% to 2.2% mostly because the prompt changed its style: it started spelling out numbers, which happened to match the references.

Count the empty transcripts

One failure every table above averages away: clips that come back completely empty. Across 738 clips, two models returned an empty transcript 10 times each, including clear speech like "Revenue grew fourteen percent…". In WER, an empty clip is just a batch of deletions averaged with hundreds of good clips. For the customer, it's total failure. Track it as its own number.

A test set is a versioned product

Treat your test set like code:

• Manifest and card. One file listing every clip, and a card saying what each slice is, how it was made, and what it can't tell you.
• Hold out by group, never by clip. In the lab, a random split put 27 speakers and 39 recordings on both sides of the dev/holdout line. Split by speaker, call or meeting.
• Errata, not edits. Reference corrections live in a reviewed file. When several models agree against a reference, check it: the lab found real errors ("10 perfect price gap" for "percent"), and one false alarm, "TOILETTE", which was correct all along.
• Versions. Applying six corrections moved every model's WER by 0.1–0.4 points, with no model changing. That's the size of a typical "improvement" from a new model version, so scores are only comparable on the same dataset version.

Practice: check the words that matter

Implement the lab's entity checks: digit_runs(), which turns "four four seven one, two two zero nine" into 44712209; an entity match for digits and names; and entity accuracy over a set. The tests use the lab's cases: number formatting, digits spoken as words, the lost leading zero, and the misheard name.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: some failures don't make errors, they make text. Chapter 3 is about models that write words on silence: "30 seconds of silence became 'Meghalaya, India'".