Quality gates for voice AI

A quality gate is an automatic check that runs on every pull request and blocks a change that makes things worse. This chapter turns Chapters 1–6 into gates: paired comparisons (Chapter 4), slices (Chapter 2), rare failures on non-speech (Chapter 3). Then it proves the gates work: they don't fail when nothing changed, and they catch regressions of a known size. The lab's sharpest case is a gate that blocked a bigger model that was better on every kind of speech, because on a dial tone it wrote "You", and the gate was right.

The lab's system under test is local, whisper-base.en, so the gate runs with no API keys. The evaluation set has 120 clips in four slices:

Write the gate down

Research says things like "no worse on phone calls". A gate needs every choice in that sentence written down:

- name: wer_phone
  kind: relative            # compared with the baseline, clip by clip
  metric: wer
  slices: [phone]
  tolerance: 1.5            # points of WER
  severity: block           # or warn
  why: "Phone lines with background voices: the typical contact-centre call (Lab 02)."

Each field answers a question someone will ask when the gate blocks their PR: which number, on which audio, compared with what, how much movement is allowed, does it block, and why does it exist. The why is what they read at 6 pm. Without it, the gate gets loosened instead of respected. The lab's loader refuses a spec with a missing why, a misspelled kind or a relative gate without a tolerance: a typo must stop the run, not quietly skip the gate.

The lab has nine gates of six kinds:

How a relative gate decides. It pairs each clip with its own baseline result and computes the change in corpus WER, with a 95% interval from a speaker-level paired bootstrap (Chapter 4). It fails only if both hold: the change is bigger than the tolerance, and the interval excludes zero. A change that's real but within tolerance is reported ("worse, within tolerance") and doesn't block.

Why "no new failures" and not a budget. The accepted system already writes text on 8 of 12 non-speech clips, mostly "You". A budget would have to allow 8 just to let it pass, and on the smoke tier's 4 non-speech clips, "8" means nothing. Comparing clip by clip works on any tier and ignores failures that were already accepted.

Seven pull requests

The lab opens seven candidate changes against the same baseline: two other Whisper sizes, another provider, and four deliberately broken versions of the baseline. Pick one, and switch between the smoke tier and the full tier:

• The gate blocked a better model. whisper-small.en improved WER on every slice (−1.9 read, −5.7 phone, −4.6 calls) and fixed 4 non-speech clips. But it wrote "You" on a dial tone and on room tone, which the baseline never did in 5 runs. That's exactly when a person should decide.
• Nova-3 on calls: "can't tell" isn't "fine". +4.1 points passed, because the interval (−3.3 to +15.2) includes zero: 30 clips from 5 calls can't separate it from noise. The fix is more calls in the set, not a looser rule.
• Small real regressions pass by design. Dropping 1% of words made read WER 0.71 points worse, and the interval excluded zero, but it's within the 1.0 tolerance. The tolerance is the loss the team accepts per change; the 10% ceiling stops many accepted steps from adding up. whisper-tiny.en hit it at 10.18%.
• One word, one must-pass failure. whisper-tiny.en wrote "10% cabbage to sales ratio". In WER that's one substitution; for a listener it's the number they needed.
• Failures WER can't see. "Thank you." written on empty outputs moved WER by exactly 0 on every slice. Only invented_text caught it.
• Latency only warns. +1 s per clip more than doubled p95, and the gate warned. Latency depends on the machine, and Nova-3 shows why: one run had calls taking up to 110 s, a rerun's worst was 5 s. A blocking latency gate would have failed a PR over a provider's bad minute.

Red, or couldn't measure

A gate has three results, not two:

The lab returns 2, never 1, when the audio isn't what the baseline was measured on (checked on decoded samples), the evaluation set changed (a fingerprint of every clip's id, samples and reference), the normalizer version changed, a must-pass clip is missing, a key is missing, or the provider failed.

If both show the same red X, people learn that red often means "flaky infrastructure", and start re-running real regressions until they pass. And never fill the gaps by scoring failed calls as empty transcripts: Lab 04 once did exactly that, with an out-of-credit account.

Does the gate cry wolf?

Compare a system with itself. Nothing changed, so every block is a false alarm.

Even the local model isn't deterministic. Three runs from scratch gave identical transcripts on every speech clip, but 2 of 12 non-speech clips flipped. Distant talkers came out as "We'll have our coffee.", then nothing, then three sentences. With a single-run baseline, invented_text failed in 2 of 6 orderings, on pure noise.

The fix is in how the baseline is built: it merges several runs, and a clip that had text in any of them doesn't count as a new failure later. The committed baseline merges five.

With an API model (gpt-4o-mini-transcribe), 58 of 120 transcripts differed between any two runs. A single-run baseline would have blocked one PR in five on noise; a 3-run baseline, none on the full tier. Two more things the A/A test showed:

• Noise can exceed a tolerance. The largest pure-noise change on phone was 1.78 points, above the 1.5 tolerance; the interval kept the gate quiet. The lab's tolerances were set for the local model; the API model would need its own, set from its A/A table.
• A gate that fails every A/A run isn't noise. non_english failed in 5 of 5 runs, on a clip whose audio starts with an interpreter speaking Korean. The model wrote it down, correctly. The clip was excluded, and the machinery worked: the evaluation set's fingerprint changed, run_eval.py exited 2 until a new baseline was recorded, and the must-pass check refused a case pointing at the removed clip.

What can the gate catch?

The other direction: inject regressions of known size into the baseline's own transcripts (30 repeats each, no model calls), and count how often each tier blocks them. Switch the widget above to "When something did".

• The tolerance sets where blocking starts; the tier size sets how sharp that line is. Well below the tolerance, rarely blocked; well above, always; near it, a coin flip in both tiers.
• The full tier reliably blocks a regression about 1 point beyond the tolerance, the smoke tier about 2. Below that, a green check means "nothing this tier can see".
• The smoke tier is blurrier, not just weaker. It sometimes blocks a change that's really below the tolerance, and misses more above it.
• Narrow regressions need slice gates. Dropping 5% of words on phone audio only moved overall WER by 1.3–1.4 points, but the phone gate saw +3.5 to +4.1 and blocked it in 97–100% of repeats.

That's the two-tier design: the smoke tier (45 clips, a fixed third of each slice plus the must-pass clips) on every PR, in about a minute; the full tier (all 120) nightly and before a release. The report posted on each PR says which tier ran and what it can see.

The baseline changes only on purpose

baseline.json holds the accepted system's result for every clip, plus the system, the reason it was accepted, the dataset fingerprint, the normalizer, the machine and the number of runs merged. It changes only in a reviewed PR:

python run_eval.py --tier full --update-baseline --reason "Switch to whisper-small.en: -2 to -6 points WER; accepted 2 new 'You' on tones"

Four rules are enforced in code:

• CI never accepts its own results. The command refuses when CI is set. Otherwise a regression becomes the new normal without anyone deciding it.
• The baseline merges 5 runs, so rare flips aren't new failures.
• A baseline must pass its own blocking gates in every run. A must-pass phrase the accepted system only sometimes gets right would fail PRs at random.
• It must be valid on every machine that runs the gate. That turned out to be the hard one.

Caching follows the same logic. Transcripts are cached by model + settings + language + run + audio hash, and scoring always re-runs from the cached text. A PR that changes a gate or the report re-uses every transcript; a PR that changes the model re-transcribes. The nightly run skips the cache on purpose, so an API model that changes behind the same name gets noticed.

What the first CI runs found

The baseline was recorded on a Mac; the CI runner is Linux on x86. Each problem below was caught by a check built earlier in the lab, and none showed up on the laptop:

1. The rebuilt audio was different. 9 of 30 call clips came out a step apart in a few samples, because resampling in float32 rounds differently on x86 and arm64. The audio check exited 2 before anything was evaluated. Resampling in float64 fixed it, and changed 23 clips by an inaudible amount, which changed 9 transcripts and moved calls WER by 0.5 points. That's why the gate pins audio by its decoded samples.
2. The same model gave different transcripts on different CPUs. In int8, parallel runner jobs landed on 3 kinds of CPU and gave 3 sets of transcripts. On the most common one, a must-pass phrase came out as "10% CAVX to sales ratio": a gate failing on the luck of the runner draw. float32 made every machine agree, and a fresh full-tier run on Linux then matched the Mac baseline exactly.

Latency on the runner was 6.5× the laptop's, so latency_p95 warns on every CI run. That's why it only warns.

A better model fails a gate

Back to whisper-small.en. The gate did its job: it found a trade-off and stopped it for a person to decide. Read which gate failed and its why. If the gain is worth the loss, accept it: update the baseline in the same PR, with the reason, so reviewers see "accepted 2 new 'You' on tones" in the diff.

What not to do is loosen the gate or net the results automatically. "4 fixed, 2 new" looks fine; the same rule would accept "1 fixed, 1 new" when the new one is a hallucinated sentence on a customer's hold music. If a policy like "new failures on tones are acceptable" makes sense, decide it in gates.yaml, with a why, not in the moment of a red build.

Practice: the gate's decisions

Implement the decisions inside the lab's gates: the relative verdict (tolerance and interval), "no new failures" against a multi-run baseline, a whole-word must-pass check, and the 0 / 1 / 2 exit code. The tests use the lab's real results: tiny.en's +3.31, Nova-3's "can't tell" on calls, small.en's 2 new and 4 fixed, and "cabbage".

Quiz

Check your understanding

Try each question before opening its answer.

Next up: what happens after code merges. Chapter 8 is about monitoring voice AI in production: how long until anyone notices a bad release, with no answer key.