A model release pipeline

This chapter puts the earlier chapters together into one release pipeline, and shows why its last stage, a canary on live traffic, matters. In the lab, a new version of the speech service (the same model, with a new audio decoder) passed all seven offline gates, with an identical error rate on every kind of audio, and was rolled back 85 seconds into its canary, at the first check. The pipeline below shows how both can be true.

The pipeline

A release pipeline takes a candidate from "research says it's ready" to "serving customers", with evidence at every step. It's built from the earlier chapters:

candidate (model + serving config)
  → 1. offline benchmark      240 clips: read speech, simulated phone calls, real business calls   (Chapters 1–2)
  → 2. paired comparison      vs the version serving customers now, clip by clip, per slice        (Chapter 4)
  → 3. gates                  any blocking gate fails → STOP, never served a customer             (Chapter 7)
  → 4. canary                 25% of traffic, known-answer probes, monitors on both versions       (Chapter 8)
       → alarm → ROLL BACK
       → enough evidence, no alarm → PROMOTE
  → release report

Every threshold lives in one file, pipeline.yaml, with the reason it exists. A change to a threshold is a change to that file, reviewed like code.

The benchmark is voxbench-v1 (Lab 10's platform): 120 read clips (LibriSpeech), 60 phone clips (LibriSpeech with background talkers, through a phone line), and 60 segments from 5 real earnings calls. A candidate's benchmark always runs fresh; reusing stable's run for a rebuild with the same model name would compare it with itself, which the first version did.

The gates are Chapter 7's kinds: relative WER gates overall (tolerance 0.5 points), on read (1.0), phone (1.5) and calls (1.5), each failing only if the change is beyond the tolerance and its interval excludes zero; a read ceiling of 10%; no clip newly empty; and offline speed, which only warns because a laptop isn't the production machine.

The canary runs both versions side by side on the same service. The candidate serves 25% of traffic, stratified by customer segment. Every 30 canary requests, a check compares the two versions on everything so far, per segment:

Roll back at the first alarm. Promote once every segment has 40 canary requests and nothing fired. If 900 requests pass with no decision, roll back: no evidence, no release.

Test the pipeline

How do you know the pipeline itself works? Test it with candidates whose right answer you already know, written down in pipeline.yaml before running:

All four got the expected decision, and each bad one was stopped by a different stage.

Stopped at the gates

tiny-fast is about twice as fast, which is why it was proposed: to cut serving costs. Overall it got 4.74 more words wrong in every hundred (+2.65 to +6.93), and 15.05 more on phone audio. Three gates failed (wer_all, wer_read, wer_phone); calls was "can't tell" (+2.32, −2.09 to +7.01). The release stopped after 17 seconds, and no customer ever heard it.

The twist: invisible offline

Why was the new decoder identical offline? The benchmark reads the audio files directly. Only the live service runs the decoder. The new decoder ignores the file's sample rate, so phone recordings, made at 8 kHz, are read as 16 kHz and play twice as fast: Chapter 8's decoder bug, shipped as a release.

The canary caught it at the first check, three ways:

1. Impossible event: a contact-centre upload decoded to half its declared length. That never happens normally, so one is enough.
2. Probes: errors on the known-answer clips jumped from 9.2% on stable to 31.6% on the canary.
3. Must-pass: "the head of the patchwork girl", a phone clip that must always come out right, came out wrong.

And the statistical monitors? With only 6 contact-centre canary requests so far, they couldn't tell yet. Two of the six came back empty (4% on stable), and words per second fell from 3.27 to 1.38, but z was +2.1 and −1.5, short of 3.5. Pick the check below, and try the other statistical rules:

Monitors need volume. A rule for something impossible, and test clips with known answers, don't. At a 25% share, they cut the exposure to one check.

No false alarms

Catching bad versions is half the job; the other half is letting a good one through. The identical rebuild went through the whole pipeline four times, with a different draw of traffic and a freshly run benchmark each time. It was promoted all four times, with no alarm in 20 canary checks.

But one monitor got close. In two of the runs, contact-centre words per second came out 33% higher on the canary, with nothing changed: past the 25% margin, at z = 2.9 and 3.0, against an alarm line of 3.5. Switch the widget to "Margin only" and both rebuilds roll back. That's why a canary checked again and again needs both a margin (ignore differences too small to matter) and a strict z (keep repeated looks from turning noise into alarms). Four runs are too few to measure how often the pipeline false-alarms; they show the thresholds don't fire on ordinary noise.

Offline, Lab 10's noise floor showed up too: in three of the four runs, one phone clip differed between identical runs of the same model, moving WER by 0.02 points. The paired interval touched zero, and every gate passed, as it should.

The better model was promoted too. On the probes, its errors dropped from 9.2% to 6.1%, and it was better offline on every slice (−4.67 overall, −10.37 on phone). And its own confidence went up, to z +3.4 on podcasts: close to an alarm line, for an improvement. A rule like "confidence moved" could block a better model, so confidence is reported, never used to block a release.

Limits

The gates can't see regressions the size of their own tolerances. The interval on a paired phone difference was ±4.7 points, so with 60 phone clips from 18 speakers, the benchmark reliably detects only about 7 points; the gate's tolerance is 1.5. Seeing 1.5 would take something like 1,200 clips, from many more speakers.

These widths come from candidates that changed a lot, so they're likely somewhat pessimistic for subtle regressions; Chapter 7's injected-regression table measures that case directly. Either way, a subtle regression can pass the gates. The canary's probes, and monitoring after release, stand behind them.

The rest of the limits:

• This canary ran on a laptop: recorded audio replayed, evenly split by customer type, at a 25% share. Real canaries usually take 1–5% and wait for real traffic, which sets the decision time.
• The canary measures quality only through 11 probes and reference-free signals. A small regression that moves neither passes.
• Audio neither the benchmark nor the traffic contains (Chapter 8's drift) is untested by construction.
• The same machine runs the service, traffic and models, so latency includes contention. The comparison is fair; the absolute numbers aren't production numbers.

The report

Every run writes a release report: the decision and why, a timeline, the evidence from every stage (paired comparison, gates with their reasons, the canary check by check), the service state after the decision, the risks of this run, and the command to reproduce it. The canary never outlives the decision: whatever happens, the service ends up serving one version. That makes a stop or a rollback auditable, and gives the next evaluation a concrete failure to test.

The platform underneath (Lab 10)

The offline benchmark runs on Lab 10's evaluation platform, which has no chapter of its own. Four of its findings matter for any pipeline like this one:

• Where a clip runs can change its transcript. On hard clips Whisper falls back to random sampling, and a worker's random generator carries state from every clip it transcribed before. One call clip went from 4 errors to a 32-error runaway depending on its shard. Reseeding from each clip's id fixed it; the 1 clip in 240 that still differs between identical runs is measured and reported as the noise floor.
• Retries need a keyed store. With workers crashing mid-shard, appending every row counted 21 clips twice and moved WER by 0.75 points. Keying results by (run, clip) makes a repeat harmless, and a run with a missing shard is marked incomplete and refused, not reported as a smaller sample.
• Combine clips, not shards. Averaging 16 shard WERs cut by slice gave 19.91% against a corpus WER of 16.11%, because 60 phone clips weighed as much as 120 read clips.
• Measure speed; don't assume it. On the lab's laptop, int8 was about 25% slower than float32 and changed 75 transcripts with no WER gain.

Practice: the pipeline's decisions

Implement the canary's rules and the gate's limits: a signal alarm that needs both a margin and a strict z, the probe checks, the decision after each check (roll back, promote, or keep going), and what a gate can actually see. The tests use the release reports' real numbers: the rebuild's +33% at z = 3.0, the decoder's 6 phone requests, and the phone gate's ±4.7.

Quiz

Check your understanding

Try each question before opening its answer.

Evaluation platform (Lab 10)

End of the Voice AI Evaluation series. Gates stop versions that are clearly worse offline; the canary catches what only shows up when the service runs; monitoring watches what neither contains. Each stage catches something the others can't, and the pipeline itself gets tested with answers you already know.