Real upgrades vs. noise

A new version of my speech-to-text model came out.

On my hundred-clip test, it made about thirty percent fewer mistakes.

I almost switched that day.

Then I tested it properly. And the answer was: I can’t tell.

So was the upgrade real, or just luck?

Hands-on voice AI evals, episode four.

Today: how to tell a real improvement from luck.

First, a simple experiment. One model, and the same three hundred clips.

I run it five times. Nothing changes between the runs.

And still, the score moves: from two point eight nine, to three point two percent.

Why? An online model doesn’t give exactly the same answer twice. Each run, it rewrote the transcripts of about forty-five clips. That’s one in seven.

So a score has noise in it, even when nothing changed.

Now think about a release gate: an automatic check in CI that compares the new score with the old one, and can block a release.

A common rule: fail if the score moves, up or down, by more than a tenth of a point. A point here is one percent of all the words.

So let me compare the model with itself. Every pair of my five runs: that’s ten comparisons. Nothing changed, so every alarm is a false alarm.

How often do you think that gate failed?

Six times out of ten.

Stricter rules fail even more often. And a team that keeps seeing false alarms learns to click re-run. Then real problems get re-run away too.

So how much noise is in a score? There’s a simple trick to measure it, called the bootstrap.

Think of the three hundred clips as marbles in a bag.

Draw three hundred marbles, and put each one back after you draw it. Some come up twice, some not at all. That’s a new test set, made from the same clips.

Score it. Then do that ten thousand times.

Here it is for Whisper small, a free model from episode one. It scored four point four percent. Each bar shows how many re-drawn test sets got that score. Ninety-five percent of them landed between three and a half and five point four.

It’s like a poll’s margin of error: four point four, give or take one point.

With a hundred clips, like in episode one, the range gets wider. I picked a thousand hundred-clip test sets from my three hundred. Ninety-five percent of them scored between three point two and five point seven.

So a one-point gap between two models could simply be which clips you happened to pick.

Compare that with the five runs from before, on the same clips. Running the model again moved its score by about a tenth of a point. Picking different clips moves it about three times as much.

And re-running never shows you that, because every run uses the same clips. In LLM evals, people often report the deviation across a few runs. That misses the bigger noise.

Now, back to my upgrade. On three hundred clips, here are both versions, each with its margin of error.

The two ranges overlap almost completely. But comparing two ranges is a weak test.

Both versions ran on the same clips. And a clip that’s hard for one version is usually hard for the other.

So instead, I compare them clip by clip. That’s called a paired test. I re-draw like before, but whole speakers at a time, because one speaker’s clips tend to be hard or easy together. On each draw, I score both versions and take the difference. A hard clip is hard for both, so that part cancels out.

The change: minus zero point zero six points. Its range: from minus half a point, to plus half a point.

The range includes zero. So the honest answer is: can’t tell.

Clip by clip, the new version was better on twenty-six clips. The old one was better on twenty-six. On the other two hundred forty-eight, they did exactly the same.

So a comparison has three possible answers: better, worse, or can’t tell. And can’t tell is a real answer.

Whisper base to Whisper small: clearly better. The whole range is below zero.

My upgrade: can’t tell. Not better, not worse. On three hundred clips, any difference is too small to see. So I’d decide on something else, like price or speed.

So where did my one point four come from? I went back to episode one’s hundred clips.

Listen to this one.

🔊 “Where’s Milicent?”

Where’s Milicent. The old version wrote: Onde é Millicent? That’s Portuguese.

And this one.

🔊 “For the separation of an alkaloid…”

The old version wrote the whole sentence in German. Eleven of its fourteen words, wrong.

Two language switches by the old version. Without those two clips, the gap shrinks to under a point.

So almost half of my upgrade came from just two clips. And the rest didn’t hold up: on my three hundred clips, which don’t include those two, the change was just zero point zero six.

So what should a release gate do instead? Use the paired test.

Fail only when the whole range of the change is above zero. That means: clearly worse.

On the same ten comparisons of the model with itself, it failed once, instead of six times. No test is perfect: even this gate fires by chance now and then, and here it did once.

One more question: how big a problem can a test even catch?

To find out, I took real results and added extra mistakes on purpose. Then I checked, many times over, whether the paired test noticed.

With three hundred clips, one extra point of mistakes was caught every time.

Half a point: only two times in three.

And a quick fifty-clip test? Even a two-point problem slips through about one time in seven.

To catch a problem half as big, you need four times the clips.

So now, before I call an upgrade better, I do three things.

Run the same model twice, to see how much the score moves on its own.

Compare old and new on the same clips, with the paired test.

And accept that sometimes, the answer is: can’t tell.

So far, every test waited for the whole recording. Next time: live speech, where it also matters when each word arrives.Yuki: When does each word arrive? 🤔

One score is not a result. Measure the noise, compare on the same clips, and let “can’t tell” be an answer.

Then when you say better, it’s true.Yuki: Thanks! 😄