Key takeaways
- ✓Without an answer key, watch what you can compute on every request: errors, latency, empty transcripts, words per second, confidence, and decoded vs declared audio length. Each incident moved a different one.
- ✓The wrong model was the quiet failure: WER rose 3–12 points, and only confidence moved, by 0.03–0.05. The traffic monitors needed a median of 65–85 minutes; a canary with known answers caught it in its first round.
- ✓Textbook thresholds gave 263–526 false alarms a week. Set thresholds from a false-alarm budget on normal traffic, check them on other normal traffic, and keep a safety margin.
- ✓Watch each customer separately: incidents confined to one customer were caught 3–6× faster, and the all-traffic series missed the wrong model on one customer in 7 of 12 tries.
- ✓Drift is a ramp nobody deployed. A canary can't see it and there's nothing to roll back: sample real traffic, have a person transcribe it, and add it to the test set.
Chapter 7's gates run before a release. This chapter is about what goes wrong after, when there's no answer key. A wrong model file, a bad config or a decoder bug can ship while the service still returns fluent text: in the lab, the wrong model added three to twelve wrong words per hundred on every kind of audio, and the first half of a sentence still read fine. Monitoring has to notice anyway, from signals you can compute without a reference transcript, and without crying wolf.
The lab runs a real speech service (whisper-base.en, in float32 so its answers are the same on every machine), sends it clips standing in for three kinds of customers, deploys five kinds of mistakes, and simulates weeks of traffic from its real transcripts.
| Customer | Audio | Share of requests |
|---|---|---|
podcasts |
LibriSpeech read speech, plus 3% non-speech | 22% |
contact_center |
Phone audio with background talkers, uploaded as 8 kHz WAV, plus 15% non-speech (hold music, silence, tones) | 64% |
earnings |
Earnings-22 calls, plus 5% non-speech | 14% |
| Bad deploy | What went wrong |
|---|---|
wrong_model |
The release shipped tiny.en instead of base.en |
vad_setting |
A config change turned on voice activity detection with an aggressive threshold |
decoder_bug |
A new decoder ignores the file's sample rate: 8 kHz phone audio is played as 16 kHz, twice as fast |
slow |
+0.8 s per request |
errors |
8% of requests fail |
None of these is visible to a benchmark: it tests one model, on collected audio, on the day you ran it.
What can you see without an answer key?
Real traffic has no reference transcripts, so you can't compute WER. You can compute these on every request:
| Signal | A change may mean |
|---|---|
| Error rate, latency | Infrastructure trouble |
| Empty transcripts | Decoding broke, a VAD setting, a silent model failure |
| Words per second of audio | Runaways or transcribed background speech (up); deletions, dropped audio (down) |
| Model confidence | Harder audio, or a different model |
| Decoded length ÷ declared length | A decoder bug. It never happens normally |
| Hallucination and wrong-language flags (Chapter 3) | A model or setting change |
Pick an incident and a customer. True WER is shown for teaching; a production monitor never sees it.
The release shipped tiny.en instead of base.en. Phone audio with background talkers, uploaded as 8 kHz WAV.
The quiet failure. WER rose 3–12 points, the model still wrote fluent text, and the only signal that moved was confidence, by 0.03–0.05.
Grey bar: the normal deploy. Normal isn’t clean: the accepted model already writes text on more than half of the non-speech clips, so monitors have to learn normal first.
- The wrong model is the quiet failure. WER rose 3–12 points, the text stayed fluent, and the only signal that moved was confidence, by 0.03–0.05.
- The VAD setting emptied half the phone calls, and a fifth of the earnings clips, while podcasts were untouched. The empty rate is the signal, but only in those segments.
- The decoder bug is impossible to miss, if you log the right thing. A phone clip declared as 4.3 seconds decoded to half that. The model heard "I had again been acting under the influence of this man's power" at double speed and wrote "I have a day for time to go to school for the next hour." Every phone clip had the mismatch. A check that costs one division is the strongest signal in the lab.
- Normal isn't clean. The accepted model writes text on more than half of the non-speech clips. Monitors have to learn normal before they can spot abnormal.
- Confidence is a difficulty meter, not a quality meter. Within one audio type it ranks clips moderately (Spearman ρ −0.32 to −0.65 against clip WER), and across types it follows difficulty (phone lowest). But a different model has its own sense of confidence, so "confidence moved" says something changed, not how much quality did.
Alarms that don't cry wolf
The lab watches 8 signals for each customer and for all traffic: 32 series, every 5 minutes, about 22,600 requests a week. The budget is one false alarm a week for the whole system.
Each window's value first becomes a z-score: how many standard errors it is from normal, given how many requests the window had. A quiet 3 a.m. window with 2 requests can swing wildly without meaning anything; one failed request out of one is an "18σ" window. Then one of three detectors reads the z-scores:
| Detector | Idea | Good at |
|---|---|---|
| Shewhart | Alert when one window is extreme: z > k | Big, sudden jumps on busy metrics |
| EWMA | Alert when a smoothed average drifts: e = λz + (1−λ)e | Moderate shifts, with less noise |
| CUSUM | Add up evidence: S = max(0, S + z − k), alert when S > h | Small, persistent shifts no single window reveals |
32 series (8 signals × 3 customers + all traffic), checked every 5 minutes. Budget: 1 false alarm a week for the whole system. Calibrated on 20 normal weeks, measured on 20 other ones.
| Detector | Textbook threshold | Fitted to the budget | Fitted to 50% of it |
|---|---|---|---|
| Shewhart | 526 / week | 1.45 / week | 0.4 / week |
| EWMA | 263 / week | 3.35 / week | 1.65 / week |
| CUSUM | 277 / week | 3.5 / week | 1.1 / week |
64,512 checks a week, so a 1-in-1,000 fluke happens dozens of times. A threshold fitted exactly to the budget is the tightest one that happened to pass, and on new weeks it fails more often: the winner’s curse again (Chapter 4). A margin pays for most of it; EWMA and CUSUM needed more weeks to calibrate.
Thresholds. With the textbook thresholds, a normal week gave 526 false alarms (Shewhart) and about 270 (EWMA, CUSUM). That's 64,512 checks a week: even a 1-in-1,000 fluke happens dozens of times. So each threshold was calibrated on 20 simulated normal weeks and checked on 20 other weeks, the same dev/holdout discipline as Chapter 3. Fitted exactly to the budget, the held-out weeks gave 1.45–3.5 false alarms: a threshold fitted to a handful of rare events is the tightest one that happened to pass (the winner's curse, Chapter 4). Aiming at half the budget, Shewhart stayed under it at 0.4 a week; EWMA and CUSUM, at 1.1–1.7, need more weeks to calibrate.
Delay. Each incident started at 12 random moments, day or night:
- Impossible events make the best alarms. The decoder bug was caught within 5 minutes by every detector.
- The quiet failure is where detectors differ. Shewhart missed the wrong model in 4 of 12 tries. EWMA and CUSUM caught it every time, after a median of 65 and 85 minutes: an hour of worse transcripts for every customer.
- No single detector is best. Shewhart caught the slowdown in 2 minutes but took 13 hours on the error spike, because night windows push a rate's threshold sky-high. Real systems run a fast detector and a patient one side by side.
- Watch the mean of log latency, not a p95. The first version alerted on a 5-minute p95: CUSUM took a median of 4.4 hours to notice +0.8 s, because a p95 over a dozen requests is too noisy. On mean log latency, 20 minutes.
Per customer. An incident that hits one customer is diluted by everyone else's normal traffic. Watching each customer separately caught those incidents 3–6× faster, and caught every one; the all-traffic series alone missed the wrong model on the earnings customer in 7 of 12 tries. Each extra series spends false-alarm budget, so pick segments that matter: big customers, audio types, languages.
The canary
The fix for the quiet failure is a canary: 11 clips the customers never send (read speech, phone audio, call audio, silence, hold music), with known answers and must-pass phrases, sent through the real API every few minutes, from outside, just like a customer. Because the audio and the answers never change, any change in the score is the system's fault.
Here's a real run, with broken versions deployed between rounds:
11 clips with known answers, sent through the real API like a customer. Broken versions were deployed between rounds. Pick a round.
transcripts changed · must-pass
1 must-pass phrase failed.
The accepted deploy gives the same 11 transcripts every round (canary WER 9.2%), so any change means the system changed. The wrong model was caught in its first round; the traffic monitors needed a median of 85 minutes. But at 8% errors, two rounds (22 requests) saw 3, 3, 0 and 1 failures across four runs: measuring a rate takes volume, and only real traffic has it.
- The wrong model: one round, not 85 minutes. 7 of 11 transcripts changed in the very next round, because the canary knows the answers.
- "Transcripts changed" is the bluntest rule, and the fastest. The accepted deploy gave exactly the same transcripts every round. An intended upgrade trips it too; then the golden answers are re-recorded on purpose, like Chapter 7's baseline.
- Measure the wobble before writing the rules. On an external API (Deepgram Nova-3, 15 rounds), canary WER was 5.3% every round with 0 changed transcripts and 0 failed requests in 165, so "any change" is a valid alarm there too. But the local model isn't deterministic on garbled audio: under the decoder bug, canary WER ranged from 32.5% to 37.7% across runs.
- The WER and latency rules wait for a second bad round, so one network blip doesn't page anyone. The first latency rule, recorded on an idle service, paged on ordinary customer load (the canary's median went from 0.30 s to 0.77 s). It now allows 2× + 0.3 s, which still catches +0.8 s.
- What the canary can't see: rates (it saw 3, 3, 0 and 1 failures in four runs of the 8% error incident; CUSUM on traffic needed a median of 22 minutes), and audio it doesn't contain.
Drift: the change nobody deployed
In a second simulated week, the share of contact-centre calls with loud background talkers grows from 0% to 50%. Same model, same deploy.
| Day | Noisy share (hidden) | True WER (hidden) | Confidence | Empty | Words/s | PSI, confidence |
|---|---|---|---|---|---|---|
| 1–7 | 0% | 24.2–25.8% | 0.593–0.595 | 5–7% | 2.22–2.35 | ≤ 0.009 |
| 9 | 9% | 32.9% | 0.574 | 6.4% | 2.33 | 0.056 |
| 11 | 21% | 44.2% | 0.545 | 6.3% | 2.36 | 0.227 |
| 14 | 39% | 61.5% | 0.499 | 7.5% | 2.38 | 0.660 |
- A ramp, not a step. Confidence slides day by day; words per second creeps up (the extra words are the background talkers); the empty rate barely moves. The population stability index (PSI, where 0.1 is a moderate shift and 0.25 a large one) shows the whole distribution moving.
- It doesn't stop alerting. CUSUM first fired on confidence on day 9.5, at a noisy share around 10%, then 54 times in the rest of the week. After each reset, normal has moved, so it fires again. An incident stops when you roll it back; drift has nothing to roll back.
- The canary can't see it. Its 11 clips never get noisier.
- Measure it. A person transcribes 80 random requests from one day: 24.2% WER (19.6–29.2) on day 3, 60.0% (47.2–72.9) on day 14. No reference-free signal can give you that number. The audio becomes a new slice of the test set (Chapter 2), and Chapter 7's gates test the next model on it.
Don't reset the reference period automatically: that would absorb a slow real regression as the new normal. Update it after the drift is understood and accepted.
Page, or dashboard?
Page someone only for what needs action now and is rarely wrong: the canary failing, an error spike, audio of an impossible length. Everything else goes on a dashboard, reviewed daily: slow trends, drift, confidence shifts, how each customer is doing. Slow alarms, like the confidence one, can open a ticket instead of waking anyone. Every alert names the clip, the customer and the version, and something watches the watcher: a CanaryNotRunning rule pages when the canary stops.
The lab's last step wires this into Prometheus and Grafana and deploys the decoder bug under live traffic:
| After the deploy | Alert |
|---|---|
| +25 s | AudioDecodedToWrongLength, contact_center: from customer traffic, before the canary's round finished |
| +35 s | AudioDecodedToWrongLength, canary |
| +47 s | CanaryTranscriptsChanged, CanaryMustPassFailing, CanaryWERHigh |
After the rollback, the canary alerts cleared on the next round, and the traffic alert 5 minutes later, when its window moved past the last bad request.
One lesson from building it: the service first ran the model directly inside its async request handler, so every other request waited behind it, /health and /metrics included. Under the slow deploy, Prometheus lost the service at exactly the moment it mattered. The model now runs in a worker thread. Monitoring has to survive the incident it's watching.
Practice: build the monitor's pieces
Implement the per-request signals (including the length check that catches the decoder bug), a sample-aware window z-score, a CUSUM detector with a reset and a cooldown, and the canary's alert rules. The tests use the lab's real cases: the half-length phone clip, the 18σ failure at 3 a.m., and the canary rounds of the wrong model.
Quiz
1.A release ships the wrong model file. The service still returns fluent text. In the lab, which reference-free signal moved?
2.A new audio decoder reads 8 kHz phone uploads as 16 kHz. What's the cheapest signal that catches it?
3.32 series, checked every 5 minutes, with a textbook 3σ Shewhart threshold. How many false alarms did a normal week produce?
4.Thresholds calibrated to exactly 1 false alarm a week gave 1.45–3.5 a week on other normal weeks. Why?
5.8% of requests start failing. Why is a canary of 11 clips the wrong tool to measure that?
6.More and more contact-centre calls come from loud rooms. The confidence alarm fires 54 times in five days. What do you do?
Check your understanding
Try each question before opening its answer.
What can production break that an offline benchmark can't see?
Deploys (a wrong model file, a wrong setting), infrastructure (decoders, latency, errors), changes behind a model name you don't control, and customers' audio changing. The lab's decoder bug only hit 8 kHz phone uploads, audio a benchmark of 16 kHz files would never contain. CI tests the change you made; monitoring is the only test that runs when nobody deployed anything.
What's a canary, what makes a good canary set, and what can't a canary see?
A small fixed set of clips with known answers, sent through the real production path on a schedule, from outside like a customer. A good set is small, covers the key audio types, and includes must-pass phrases and non-speech clips. It can't see audio it doesn't contain (drift, new customers) or rates too small for its handful of requests (8% errors).
Which signals can you monitor without references, and what does each catch?
Errors and latency catch infrastructure; empty transcripts catch dropped audio and VAD settings (50% of phone clips); words per second catches runaways and deletions; confidence catches harder audio or a different model (the wrong model, by 0.03–0.05); decoded vs declared length catches decoder bugs (0% to 100%); hallucination and language flags catch model or setting changes.
How do Shewhart, EWMA and CUSUM differ, and how do you choose their thresholds?
Shewhart alerts on one extreme window, EWMA on a smoothed average, CUSUM on accumulated evidence. Turn each window into a z-score that accounts for its request count, then calibrate thresholds to a share of the whole system's false-alarm budget on normal traffic, with a margin, check them on different normal traffic, and measure detection delay on injected incidents.
Why monitor per customer segment?
An incident on one customer is diluted by everyone else's normal traffic. Per-segment series caught such incidents 3–6× faster and caught all of them. Every series spends false-alarm budget, so choose segments that matter.
What's data drift, how do you tell it from an incident, and what do you do about it?
The audio changing while the system stays the same: a ramp, not a step. Detectors keep firing after every reset (54 times in a week), and distribution checks show many signals moving. Sample the traffic, have people transcribe it (60% WER vs 24%), and add it as a versioned test-set slice. Never update the reference period automatically.
What deserves a page, and what goes on a dashboard?
Page for what needs action now and is rarely wrong: canary failures, error spikes, impossible events. Dashboards get trends, drift, confidence shifts and per-customer comparisons. Every alert carries the clip, segment and version, and a rule pages when the canary stops running.
Next up: so far, every chapter tested one model on its own. Chapter 9 tests a whole voice agent, where speech-to-text, a language model and text-to-speech hold a conversation.