Testing streaming speech-to-text

These are live captions, replayed exactly as two speech systems wrote them, while the person was still talking.

🔊 “Sancho got up with pain enough in his bones, and went after the innkeeper…”

Did you catch the name? The speaker said: Sancho.

The top one wrote Sandra, then fixed it two seconds later.

The bottom one fixed boat to bones. But it kept Sandra, all the way into its final transcript.

So how do you test something that changes its mind while you watch?

Hands-on voice AI evals, episode five.

Today: testing speech-to-text that works live, word by word.

Until now, every test sent a whole recording, and got one transcript back. That’s batch.

A live voice agent can’t wait for the recording to end. It streams: it sends the audio in small chunks as you speak, and gets text back while you’re still talking.

So the model has to guess before it hears the end of the sentence. Those guesses are partials, in amber. They can change. A final is a promise: it won’t change.

Here’s one short sentence, streamed to four systems.

🔊 “Do you suppose the miniature was a copy of the same thing?”

They behave differently. Nova-3 sends a guess about once a second. Flux updates four times a second. OpenAI’s live model only adds words, and never takes one back.

And Whisper, re-run on the growing audio every second, wrote: a copy of the miniature. Then it corrected itself: a copy of the same thing.

The first rule of testing streaming is about how you send the audio.

If you send a ten-second recording as fast as your network allows, the service gets all of it in a moment.

Sent that way, OpenAI’s final text came back before the speaker would even have finished talking: on half of ten clips, more than five seconds early. That can’t happen on a real call.

So a streaming test sends the audio at the speed it was spoken: one second of audio, every second.

And it measures each delay from the moment a word was spoken in the recording.

“Fast” can mean four different things. And the systems rank differently on each.

Word on screen: from the moment a word is spoken, to when it first appears. Flux is the fastest: a fifth of a second.

Word final: until that word can’t change anymore. Here Flux is the slower of the two Deepgram systems: over five seconds.

Why? Flux waits until you’ve finished your whole turn, everything you say before you stop for a reply, then locks it in at once. Nova-3 locks in a phrase at a time.

Speaker stops, to final: what a voice agent waits for before it can answer. Nova-3: half a second.

OpenAI’s live model doesn’t detect the end of a turn. The app decides, here after two and a half seconds of silence.

So for live captions, care about words on screen. For an agent that acts on what you said, care about finals.

Partials that keep changing are called flicker. In captions, the text jumps around. And an app that acts on a partial can act on a word that later disappears.

So I counted how often a word on screen was changed by the next update.

OpenAI’s live model: four percent, and nearly all of those are half-written words being finished. It only adds text.

Whisper, re-run every second: about one word in four changed.

You can buy stability. Show a word only when two guesses in a row agree, and never take it back.

Rewrites dropped from twenty-seven percent to two. But words appeared later: from under a second, to over two.

Stability and speed trade against each other, so a test has to report both.

A voice agent also has to decide when you’ve finished talking, so it knows when to answer.

The simplest way is a silence timer. Three hundred milliseconds of silence, and your turn is over.

So I took sentences and put a pause in the middle. Here, eight tenths of a second.

🔊 “The count had thrown himself back on his seat… leaning his shoulders against the partition…”

Where do you think the turn ended?

Right in the pause, after: the count had thrown himself back on his seat. An agent would start answering in the middle of the sentence.

With pauses that long, the three hundred millisecond timer cut off all twelve sentences.

A one-second timer cut off none of them. But replies came more than twice as slowly.

Flux uses a model that listens to what’s being said. By default, it cut off four of the twelve, and ended turns in about a second.

That’s the trade-off: fewer cut-offs, or faster replies. So test it with pauses like the ones your callers make.

Last, I checked every stream I recorded for problems: four hundred and eighty-eight of them.

One failed. A Flux stream, on a clip of an earnings call that ends mid-sentence: our Chief Executive Officer, and…

The screen showed the words as they were spoken. But the stored transcript was empty.

Flux never ended the turn. Two and a half seconds of silence went by, then another second, and it was still waiting, probably because the sentence wasn’t finished. Then my test closed the stream, and nothing was ever finalized.

Remember the empty answers from episode two? The same result for the user, from a different cause.

One more: if the app goes quiet for twelve seconds, mid-stream, the service closes the connection. None of the six transcripts was complete.

Sending a small keep-alive message while quiet fixed all six.

So, for a voice agent, here’s what I’d measure. Send audio in real time. And time the event your app acts on: words on screen, finals, or the end of the turn.

Count how often words change. Test pauses in the middle of sentences. And check that every stream ends with a final.

Next time, the other direction: turning text into speech. Does it say the right thing, and how fast does it start talking?Yuki: Does it say the right thing? 🤔

Streaming adds a new question: not only what was said, but when. Measure both.

Then your live captions keep up, and stay right.Yuki: Thanks! 😄