Menu

Blog Learn Videos About Start learning

Episode 01 of 10

Word error rate, explained

How to measure speech-to-text accuracy: line up what was said with what the model wrote, count three kinds of mistakes, and know the four things the number won't tell you.

Lab 01 · Speech-to-Text Evaluation Basics5 min read

Video transcript5:03· read along, or click a time to jump there

0:00Intro

I tested eight speech-to-text models. Every single one got about ninety-eight percent of the words wrong.

They were fine. The way I measured was broken. Here’s how word error rate really works, and how to read it.

0:14The 98% scoreboard

Here’s my scoreboard: eight models, a hundred audio clips. Every one of them, about ninety-eight percent wrong.

Would you ship any of these? Hold that thought. We’ll come back to this number.

0:27Meet Yuki

I’m Yuki, an engineer at Cloudly. We’re adding a phone line to our support assistant.

Callers speak, and a speech-to-text model turns their words into text for the assistant. If it mishears, everything after it goes wrong.

So I have to pick that model. And to pick one, I need a score I can trust.

0:4801 · The test set

First, test audio with a known right answer. I use a hundred short clips of people reading books aloud, a standard set called LibriSpeech.

For every clip, a person wrote down exactly what was said. That’s the answer key.

Half the speakers are clear, and half are harder to understand.

1:0701 · One clip

Here’s one of them.

🔊 “Concord returned to its place amidst the tents.”

On top, what was said. Below, what a free model, Whisper base, wrote. Almost right: it wrote Concorde, with an e.

1:2102 · Counting errors

Now the score. Say someone says: schedule a meeting with John tomorrow. And the model writes: schedule a meeting with Jon tomorrow, please.

Line the two up, word by word. Jon instead of John is a substitution. Please, which nobody said, is an insertion. A word the model skipped would be a deletion.

Add up the mistakes and divide by the number of words actually said: two out of six. That’s a thirty-three percent word error rate.

Zero percent would be perfect. Lower is better.

1:5403 · What WER can’t tell you

Two things this number hides. First, it can go above a hundred percent.

Someone says yes. The model writes: yes, thanks for watching and please subscribe. Six extra words for one word said: six hundred percent.

Second, every word counts the same. Fifty heard as fifteen, in a money transfer, costs exactly as much as to heard as into.

So the score can’t tell a typo from a disaster. Keep that in mind. It comes back.

2:2304 · The 98% mystery

Now, back to ninety-eight percent. Why would every model get almost every word wrong?

Look at that first clip again. The answer key is written in capital letters. The model wrote a normal sentence, with a period at the end.

Compared exactly, every single word counts as a mistake: a hundred percent, for a transcript that’s almost right.

So before counting, both texts get cleaned up the same way: lowercase, no punctuation. Now it’s twelve and a half percent. One real error: Concorde.

That clean-up step is called normalization. And it decides the number.

3:0105 · Clean-up picks the winner

With clean-up, the scoreboard finally makes sense: between three and nine percent.

But the clean-up rule itself matters. With basic clean-up, OpenAI’s mini model beats the free Whisper small.

Switch to the Whisper normalizer, which also unifies numbers and spellings, and the order flips.

The gap is six words one way, and one word the other, out of about seventeen hundred. Too close to call a winner.

3:2806 · Accuracy, delay, price

A model is more than its error rate. On a phone line, delay matters: how long until the text comes back. And so does the price per hour of audio.

Look at the most expensive model, gpt-4o-transcribe. Three cheaper models beat it on accuracy, delay and price at once.

Expensive isn’t the same as best.

3:5007 · Read the errors

But there’s more behind its four percent. Just five clips hold more than half of all its errors. What happened in them?

Listen to one.

🔊 “I know you will, said Dickie, with enthusiasm…”

English, clearly. And the model wrote: German.

It did this on four of the hundred clips: German, Chinese, and something like Italian.

To the error rate, that’s just a few more errors among many good clips. That’s why you read the errors, not just the score.

4:2208 · My pick

So, my shortlist. On this test, gpt-transcribe is the most accurate, and nova-3 is the fastest of the API models.

But these clips are people reading books in a quiet room. My callers will be on phones, in noise, saying names and numbers.

So no winner yet. Next time, I build a test set that sounds like real calls, and see which model is still standing.Yuki: Next: a test set that sounds like real calls. 📞

4:49Summary

Know your answer key. Count substitutions, deletions and insertions. Clean up both sides the same way. And read the errors behind the number.

That’s how to read a word error rate.Yuki: Thanks! 😄

Key takeaways

✓WER = (substitutions + deletions + insertions) ÷ words that were said. Lower is better.
✓WER can go above 100%: one invented sentence on a one-word clip scores 600%.
✓Report corpus WER (total errors ÷ total words), never the average of per-clip WERs.
✓The normalizer is part of the metric. Change it and the same transcripts can score 98% or 4%.

Every speech-to-text evaluation starts with one number: word error rate. It's simple to compute, and it's the number every model card, benchmark, and vendor slide quotes. It's also easy to get wrong, and easy to misread. This chapter covers how it works, then the traps.

Line it up, then count

You need two texts: the reference, what was actually said (written by a person), and the hypothesis, what the model wrote. Line them up word by word using the fewest edits, then count three kinds of mistake:

Error	Meaning
Substitution (S)	A word was written as a different word
Deletion (D)	A word that was said is missing
Insertion (I)	The model wrote a word nobody said

REF:  schedule a meeting with john  tomorrow ******
HYP:  schedule a meeting with jon   tomorrow please
                              S              I

WER = (S + D + I) ÷ number of reference words. Here that's 2 ÷ 6 = 33%. Try it yourself, and edit either sentence to see the alignment change:

Try it · Word error rateEdit either text, or pick an example

Reference · what was saidHypothesis · what the model wrote

Normalizer

scheduleschedule✓

aa✓

meetingmeeting✓

withwith✓

johnjonS

tomorrowtomorrow✓

∗∗∗pleaseI

Top row: reference. Bottom row: hypothesis. S = substitution, D = deletion, I = insertion.

Substitutions

1

Deletions

0

Insertions

1

Words said

6

WER

33.3%

WER = (1 + 0 + 1) ÷ 6 = 33.3%

Four things WER doesn't tell you

The formula is short, and that's why its quirks surprise people:

It can pass 100%. Insertions have no limit. In the lab, a model that heard "yes" and wrote "yes thanks for watching and please subscribe" scored 600% on that clip. (Chapter 3 is about exactly this failure.)
It isn't symmetric. The reference sets the denominator. Swap the two texts and you get a different number. Use the swap button above to see it.
Every word weighs the same. "fifty" → "fifteen" in a bank transfer costs one error, exactly like "to" → "into". A good WER can still hide the one word your product depends on. (Chapter 2 adds entity accuracy for this.)
It hides where the errors are. 5% could mean a few small slips everywhere, or three clips that are completely wrong. Those need different fixes. In the lab, one model had 57% of all its errors in just 5 clips.

Corpus WER, not average WER

A test set has many clips, and there are two ways to turn them into one number:

Corpus WER: total errors ÷ total reference words, over the whole set.
Average WER: compute WER per clip, then average those.

Always report corpus WER. The average gives a two-word clip the same weight as a 40-word one, so a single slip on a short clip dominates the result. In the lab, the same two clips gave 4.8% as corpus WER and 26.2% as an average. The sliders start with a similar pair, a 2-word clip and a 40-word clip, each with one error. Move them to see how the two numbers react:

Try it · Corpus vs. average WERDrag the sliders

Clip AWER 50.0%

words said2errors1

Clip BWER 2.5%

words said40errors1

Average of per-clip WERs

26.3%

Corpus WER (report this)

4.8%

Corpus WER is 2 errors ÷ 42 words. The average lets one slip on a 2-word clip count as much as a whole 40-word clip.

The normalizer is part of the metric

Before counting, both texts are cleaned up the same way: lowercase, drop punctuation, and usually more. Without that, "Mr. Smith paid $25." vs. "MISTER SMITH PAID TWENTY FIVE DOLLARS" is almost every word wrong.

How much does it matter? In the lab, whisper-base.en scored 98% WER with no normalization, 4.0% with basic cleanup, and 4.3% with OpenAI's Whisper normalizer, on the same transcripts. The raw number was measuring capital letters, not hearing. And for two close models, the normalizer even picked the winner: one model won under basic cleanup, the other under the Whisper normalizer, by a margin of a few words out of about 1,700.

So a WER without its normalizer can't be compared with anything. Every model in a comparison must use the same one, and your results should record which one (and which version) was used.

Read the errors, not just the number

The most useful thing you can do after computing WER is look at the worst clips. In the lab, every clip was English and every request set language="en", yet some models translated instead of transcribing: "Where's Milicent" came back as Portuguese, a sentence about alkaloids as German. In WER, each of those is just a few more errors averaged with 99 good clips. They need their own metric, which is where the next chapters go.

Practice: write WER yourself

The best way to really understand WER is to build it. Fill in the three TODOs below: the edit-distance table, the walk back through it, and corpus WER. Then run the tests. They're the same cases as above, from the lab: the misheard name, the swapped words, the 600% clip, the asymmetry, and corpus vs. average. You can also download the files and run them locally with pytest.

Try it · Practice in PythonRuns in your browser · ⌘/Ctrl + Enter to run

wer_practice.py50 lines

"""
Practice: word error rate from scratch.

WER = (substitutions + deletions + insertions) / number of reference words

Fill in the three TODOs, then run the tests:
    pytest test_wer_practice.py        (locally)
or press "Run tests" on the chapter page.
"""

def edit_counts(ref: list[str], hyp: list[str]) -> tuple[int, int, int]:
    """Line up two word lists with the fewest edits and return (sub, dele, ins).

Hint: build a table cost[i][j] = fewest edits to turn the first i reference words
    into the first j hypothesis words. Then walk back from cost[n][m] to count
    which edits were used (match/substitute, delete, insert).
    """
    n, m = len(ref), len(hyp)
    cost = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(1, n + 1):
        cost[i][0] = i  # delete everything
    for j in range(1, m + 1):
        cost[0][j] = j  # insert everything

# TODO 1: fill in the rest of the table
    # cost[i][j] = min(match or substitute, delete a reference word, insert a hypothesis word)

# TODO 2: walk back from (n, m) and count sub, dele, ins
    sub = dele = ins = 0
    return sub, dele, ins

def wer(ref: str, hyp: str) -> float:
    """WER for one clip. Texts are already normalized: just split on whitespace.
    An empty reference gives 0.0 for an empty hypothesis, and infinity otherwise."""
    r, h = ref.split(), hyp.split()
    sub, dele, ins = edit_counts(r, h)
    if not r:
        return 0.0 if not h else float("inf")
    return (sub + dele + ins) / len(r)

def corpus_wer(pairs: list[tuple[str, str]]) -> float:
    """Corpus WER over many clips: total errors / total reference words.

TODO 3: don't average the per-clip WERs. Sum errors and words first.
    """
    return 0.0

↓ wer_practice.py ↓ test_wer_practice.py ↓ wer_solution.py

Quiz

Try it · Check your understanding0 of 6 answered

1.Reference: "the cat sat". The model wrote: "the cat sat on the mat". What's the WER?
2.A test set has a 2-word clip with 1 error and a 40-word clip with 1 error. Which number should you report?
3.whisper-base.en scored 98% WER on raw text and 4% after basic normalization, on the same transcripts. What does that tell you?
4.Two correct WER tools give the same total errors on a clip but different substitution / deletion / insertion counts. Why?
5.A new model version is 1.4 points better in WER. Better clip by clip on 14 clips, worse on 6. What should you do before calling it an improvement?
6.Every model you test gets the same word wrong. What do you suspect first?

Check your understanding

Try each question before opening its answer.

How is WER calculated, and what are S, D and I?

Align the reference and hypothesis using the fewest word edits. WER = (substitutions + deletions + insertions) ÷ reference words.

Why can WER be above 100%, and why isn't it symmetric?

Insertions are unlimited while the denominator is reference words, so a short reference can have many extra hypothesis words. Swapping reference and hypothesis changes the denominator and which edits count as insertions or deletions.

Why report corpus WER instead of the average of per-clip WERs?

Corpus WER divides total errors by total reference words, so each word has equal weight. Averaging clip rates overweights short clips and can produce a very different score.

What does normalization do, why does it matter, and what can go wrong?

It applies the same text policy to reference and hypothesis so formatting differences do not count as recognition errors. Pin and test the normalizer: it can miss equivalent forms, transform text incorrectly, or erase meaningful disfluencies.

What do you need to record so a WER number can be reproduced?

Record the dataset version and audio/reference hashes, normalizer and version, model version and settings, code version, and (for latency) hardware and network conditions.

How do you compare models that differ on accuracy, speed and cost?

Remove dominated models first, then select against explicit product requirements such as a maximum slice WER, latency limit, or budget. Compare on representative product audio, not only an overall score.

Two versions differ by 1.4 WER points. What else do you need before calling it an improvement?

Estimate noise with a paired comparison on the same clips, inspect which slices and clips changed, and verify the result on representative product audio. A point difference alone does not establish a real or useful gain.

Every model gets the same word wrong. What do you suspect, and what do you do?

Check the reference and normalization policy before blaming every model. Confirm corrections by review, record them in versioned errata, and rescore all models against the new dataset version.

Next up: a single WER on clean audiobooks says little about phone calls or noisy rooms. Chapter 2 builds test sets that match real audio.

Learning something useful?

New labs, articles, and videos on evaluating and testing AI systems come out regularly. Follow along so you don't miss the next one.

Subscribe on YouTube