Key takeaways
- ✓Broken search raises no error. Over two hours on Wednesday, 38 of 40 questions got “I don't know” and the error chart stayed flat. Watch the “I don't know” rate (and other signs of quality), not only errors and latency.
- ✓A day blurs a two-hour problem. Wednesday had 46 “I don't know”s against about 20 on a normal day, but the day can't say when, or how bad. By the hour it's plain: 19 of 21, then 19 of 19.
- ✓A threshold is a number someone picks. “Alert when more than 30% of an hour's answers say I don't know” caught the broken search in its first hour, and fired once more: Saturday 7 p.m., 5 of 16, all five correct. The articles don't answer those questions.
- ✓A share means more with more tries. 4 or more heads in 6 flips happens about 1 time in 3; 40 or more in 60, less than 1 time in 100. A baseline rule learns normal (15% “I don't know” on Monday and Tuesday) and alerts only when an hour this size would look this bad by chance less than once in 1,000 hours.
- ✓Scored against the planted week, both rules caught the broken search in the first hour. False alarms: the threshold 1, the baseline 0; over four kinds of number, thresholds 3, baselines 1. A fixed 40% line would also have stayed quiet this week; the baseline sets the bar for each hour's size, so nobody has to guess it.
- ✓Check every hour: checked once a day, the same alert came at midnight, 15 hours after search broke. And when an alert fires, open the traces from that hour: the alert says where, the trace says why.
A dashboard shows a system's numbers over time; an alert watches them for you, and tells someone when one looks wrong. Every alert can fail two ways: it stays quiet during a real problem, or it fires when nothing is wrong, and after a few of those, people stop reading it. This chapter watches a week with problems planted on purpose, picks the number that shows a broken search, and scores two kinds of alert on it: a threshold, a number someone picks, and a baseline, which learns what's normal.
The source is Lab 14 (Observability), scripts 02_traffic.py (the week), 03_dashboard.py (the dashboard) and 04_alerts.py with alerts.py (the rules and their scores), recorded in October 2026. The video follows one problem and one number; the lab's other rules and problems are in a section of their own below, marked as lab-only. Lab 14's other scripts are the other observability chapters: tracing (40), finding causes in traces (42), and user feedback and turning traces into tests (43).
A fire drill: a week with planted problems
To know whether an alert would have caught a problem, you need a week where you know what the problems were. The lab replays one, Monday 12 to Sunday 18 October 2026 (virtual time):
- 1,055 questions from 140 simulated learners to an assistant that answers from 7 short articles on AI engineering. It searches the articles for passages that match the question, and a model writes the answer from those passages only, or says “I don't know”.
- Every answer is a live model call (gpt-5.4-nano), and every request leaves a trace (chapter 40).
- Four problems, planted on purpose:
| When | What | |
|---|---|---|
| Search broken | Wed 09:00–11:00 | search returns passages from the wrong articles |
| A release | Thu 00:00 on | v2 ships to half the users: one sentence, no citations, longer history |
| API slow | Fri 12:00–14:00 | 40% of model calls take 4 s longer |
| API errors | Sat 10:00–12:00 | 60% of model calls get a 503 |
The video follows the first. Because it's a drill, the right answer to every question is known, so the lab can check which alerts fired, how fast, and which fired for nothing.
Watch the right number
One row per day, from 02_traffic.py:
| Day | Questions | Errors | “I don't know” | Judged good (online) | Right (known answers) |
|---|---|---|---|---|---|
| Mon | 151 | 0 | 25 | 98% | 100% |
| Tue | 151 | 0 | 21 | 99% | 100% |
| Wed | 150 | 0 | 46 | 91% | 79% |
| Thu | 150 | 0 | 20 | 97% | 99% |
| Fri | 152 | 0 | 17 | 98% | 99% |
| Sat | 150 | 15 | 13 | 90% | 90% |
| Sun | 151 | 0 | 15 | 98% | 100% |
Wednesday is higher: 46 “I don't know”s, against about 20 on a normal day. But the day can't say when it happened, or how bad it was. By the hour it's plain: in the two broken hours, 38 of 40 questions got “I don't know” (19 of 21, then 19 of 19). And the error chart is flat: broken search doesn't crash anything. So the number to watch here is the “I don't know” rate.
The lab's dashboard (03_dashboard.py, one offline HTML page) charts six numbers per hour with the problem windows shaded: slow end of latency (p95), errors, “I don't know”, answers judged good, citations by version, input tokens by version. Each problem shows on a different chart. Every request in the table below the charts expands to its question, the passages found with their scores, the answer, and its spans.
A threshold alert
Nobody watches a chart all day. An alert is a rule, checked on a schedule: here, every hour. The simplest kind is a threshold:
Rule("refusals over 30%", "refused", "threshold_rate", 0.30) # min_n=5: skip hours under 5 questions
Alert when more than 30% of an hour's answers say “I don't know”. Checked every hour from Wednesday to Sunday, it fired three times:
| Hour | “I don't know” | |
|---|---|---|
| Wed 9 a.m. | 19 of 21 (90%) | the broken search, in its first hour |
| Wed 10 a.m. | 19 of 19 (100%) | the broken search |
| Sat 7 p.m. | 5 of 16 (31%) | nothing broken |
An hourly rule can't speak before the end of the hour the problem started in, so catching it in the first hour is as fast as it gets.
The alert says where. The trace says why.
The alert points at Wednesday, 9 a.m. To see why, open a trace from that hour. One learner asked: what does parameter-efficient fine-tuning (PEFT) do instead of updating all model weights?
| Passages search found | Match scores | Answer | |
|---|---|---|---|
| Monday | fine-tuning, fine-tuning, fine-tuning | 0.767, 0.549, 0.522 | PEFT injects small trainable matrices and freezes the rest… |
| Wednesday | transformers, RAG, AI agents | 0.767, 0.549, 0.522 | I don't know based on the provided documents. |
The planted fault keeps the match scores (how close each passage is to the question) the same, the way a mixed-up search index can look confident. Given passages from other articles, “I don't know” was the honest answer. The problem is search, and only the trace shows it.
Saturday: an alert for nothing
Saturday 7 p.m.: 5 “I don't know”s out of 16, 31%, just over the line. Was something broken? No. All five were questions the articles don't answer:
- How does reinforcement learning from human feedback (RLHF) work?
- How much does Pinecone cost per month?
- Which companies use Qdrant in production?
- How do you deploy a model to Kubernetes?
- How long does it take to fine-tune a 7B model?
“I don't know” was right every time; all 16 answers in that hour were right. The trouble is the size of the hour. Normally about 15% of answers say “I don't know”, about 2 of 16. In a small hour, two or three more is enough to cross 30%.
Same share, different evidence
Think of a fair coin. Flip it 6 times and get 4 heads: you shrug, because 4 or more happens about 1 time in 3 (0.34). Flip it 60 times and get 40 heads: two thirds again, but now you'd check the coin, because 40 or more happens less than 1 time in 100. Same share; more tries make it stronger evidence. Hours work the same way.
A baseline rule uses that. First it learns what's normal from the clean days: on Monday and Tuesday, 46 of 302 answers said “I don't know”, 15%. Then, every hour, it asks: if nothing were wrong, how often would an hour this size look this bad? That's a binomial test, and the rule alerts only when the answer is less than once in 1,000 hours:
Rule("refusals above normal", "refused", "rate_above", 0.001)
# fires when binom_sf(k, n, 0.15) < 0.001: k or more of n is that unlikely at the normal rate
| Hour | “I don't know” | By chance, if nothing were wrong | Baseline |
|---|---|---|---|
| Sat 7 p.m. | 5 of 16 | about 1 hour in 12 (p = 0.083) | quiet: normal noise |
| Wed 9 a.m. | 19 of 21 | practically never (p < 10⁻⁹) | alert |
So the bar depends on the hour's size: in an hour of 16, the baseline needs 9 or more “I don't know”s (56%); in an hour of 60, 20 (33%). A threshold uses the same 30% for both.
Questions in the hour
“I don't know”: 5 of 16
If nothing were wrong, an hour of 16 would usually have about 2.4 “I don't know”s. 5 or more happens by chance about 1 hour in 12 (p = 0.083). The baseline alerts below 1 in 1,000, so for 16 questions its bar is 9 or more (56%).
Solid line: the share the baseline needs before it alerts, for each hour size. It falls as hours get busier, and it's above the 30% line everywhere up to 60 questions. The dot is this hour.
Saturday 7 p.m.: the five “I don't know”s
- How does reinforcement learning from human feedback (RLHF) work?
- How much does Pinecone cost per month?
- Which companies use Qdrant in production?
- How do you deploy a model to Kubernetes?
- How long does it take to fine-tune a 7B model?
The articles answer none of them, so each “I don't know” was right.
The scoreboard
Both rules, checked every hour from Wednesday to Sunday:
| Rule | Broken search caught | False alarms |
|---|---|---|
| “I don't know” over 30% (threshold) | first hour | 1 (Sat 7 p.m.) |
| “I don't know” above normal (baseline) | first hour | 0 |
The lab tried the same two kinds on other numbers too (errors, the slow end of latency, answers a grading model called bad). Over the four thresholds and their four baseline twins: thresholds 3 false alarms, baselines 1. Every planted outage (search, the slow API, the errors) was caught within the hour by a threshold and by its baseline twin.
To be fair to thresholds: a fixed line at 40% would also have stayed quiet this week. The highest hour outside the broken search was 31% (Saturday), and the broken hours were 90% and 100%. But 40% is a number that happens to fit this week. The baseline's bar was at least 42% in every hour it judged, set by each hour's size from what's normal, so nobody had to guess it.
And check every hour. Checked once a day, Wednesday's 46 of 150 (31%) only just crosses 30%, and the alert comes at midnight: 15 hours after search broke.
Try the rules on the real week. Lower the threshold and small, healthy hours start crossing it; switch to once a day and the alert moves to midnight:
Alert when “I don't know” is…
Check
Threshold: alert when more than 30% of the window's answers say “I don't know”. Windows with fewer than 5 questions aren't judged.
Wed 9 a.m. · 19 of 21 (90%) · the broken search
Wed 10 a.m. · 19 of 19 (100%) · the broken search
Sat 7 p.m. · 5 of 16 (31%) · nothing broken: a false alarm
Saturday's 5 of 16 is 31%: just over the line, with nothing broken. In a small hour, two or three extra “I don't know”s are enough.
Lab-only: the other rules and problems
The video follows one number. 04_alerts.py scores 12 rules against all four planted problems. A rule is credited only with problems that can move its number (citations can't see a slow API); any other alert is a false alarm. Hours with fewer than 5 requests are skipped.
- The release was slow to show. v2 went to half the users at midnight on Thursday. It drops citations, so the citation rules caught it, but only 7 hours later: the first hour with 5 v2 questions ended at 7 a.m. Its token growth took 16 hours, because history grows only as conversations get long. Why it cost more is chapter 42.
- A release needs rules per version. Half the traffic changing barely moves totals; the v2-only rules see it as soon as there's enough v2 traffic.
- The baseline's one false alarm was real. Both latency rules fired on Thursday 9 p.m. (p95 7.9 s) with nothing planted. The trace shows one embedding call that took 6.65 s while the week was replayed, in an hour of 6 requests. With 6 requests, the slow end (p95) is just the slowest one: a percentile needs more requests per window than a rate.
- The online judge saw less than the known answers. The grading model, which sees only the question, the passages and the answer, called 91% of Wednesday good; against the known answers, 79% were right. Inside the broken hours it passed 27 of 40, and only 8 were right: refusing from the wrong passages looks fine to a judge that can't tell they're the wrong passages. Its alert rules still caught the broken search within the hour.
- Daily checks: most rules caught the same problems at the end of the day, 12–15 hours late (4 days for the token rule), and “judged bad over 15%” missed both of its problems.
| Rule | Search broken | v2 release | API slow | API errors | False alarms |
|---|---|---|---|---|---|
| Tp95 over 5 s | – | – | 1 h | – | 1 |
| Terrors over 5% | – | – | – | 1 h | 0 |
| T“I don't know” over 30% | 1 h | – | – | – | 1 |
| Tjudged bad over 15% | 1 h | – | – | 1 h | 1 |
| Bp95 over 2× normal | – | – | 1 h | – | 1 |
| Berrors above normal | – | – | – | 1 h | 0 |
| B“I don't know” above normal | 1 h | – | – | – | 0 |
| Bjudged bad above normal | 1 h | – | – | 1 h | 0 |
| Bcitations below normal | 1 h | 7 h | – | – | 0 |
| Binput tokens above normal | – | 16 h | – | – | 0 |
| Bcitations below normal, v2 only | – | 7 h | – | – | 0 |
| Binput tokens above normal, v2 only | – | 16 h | – | – | 0 |
“1 h” is the best an hourly check can do: it speaks at the end of the hour the problem started in. Every outage was caught that fast, by thresholds and baselines alike. The release was slow to show: v2 went to half the users at midnight, and the first hour with 5 v2 questions was 6 a.m., so the citation rules spoke at 7 a.m.
T: a threshold someone picked. B: a baseline learned from Monday and Tuesday. A rule is credited only with problems that can move its number; any other alert is a false alarm. The baseline false alarm (p95 over 2× normal, Thursday 9 p.m.) was one real 6.65 s embedding call in an hour of 6 requests, where p95 is the slowest one.
Caveats
- A drill, not production. The week is replayed in about 7 minutes from simulated learners; the problems are planted and big. Real problems are often smaller and slower, and there the difference between rules shows more.
- One week, a handful of alerts. “3 false alarms vs 1” is a few events. It shows the mechanism, not a rate you can plan on.
- The baseline assumes normal stays normal. It learned 15% from two clean days. If traffic changes (new kinds of questions, a new product), relearn it, or it will alert on the new normal.
- Hours are independent coin flips only roughly. One learner asking several unanswerable questions in a row makes “I don't know”s cluster, so real noise is a bit bigger than the binomial test assumes.
- The 40% line worked this week. The lab's case for the baseline is that it needs no guess and adapts to each hour's size, not that a well-chosen threshold must fail.
The checklist
- Watch a quality number (“I don't know”, citations, judged good), not only errors and latency. Broken search raises no error.
- Chart by the hour. A day blurs a two-hour problem into its 150 questions.
- Check alerts hourly. Once a day means alerting at midnight.
- Skip windows that are too small (here, under 5 questions), and use more requests for a percentile than for a rate.
- Alert against a baseline: learn the normal rate on clean days, then ask how unlikely this window is, given its size.
- Score every rule against known problems: what it caught, how fast, and false alarms per week.
- Alert per version for releases that change part of the traffic.
- When an alert fires, open the traces from that window. The alert says where; the trace says why.
Practice: score the rules
Write the lab's rules from scratch: the 30% threshold, the binomial chance P(X ≥ k), the baseline rule and the bar it sets for an hour of each size, and a scorer that says how fast a rule caught the broken search and how many times it fired for nothing. The tests run them on the recorded week, hour by hour: the threshold fires at Wednesday 9 and 10 a.m. and Saturday 7 p.m., the baseline only on Wednesday, and a daily check alerts 900 minutes late.
Quiz
1.Search starts returning passages from the wrong articles. Which chart moves?
2.Wednesday's 46 “I don't know”s against about 20 on a normal day. What can't the daily number tell you?
3.On Saturday at 7 p.m., 5 of 16 answers said “I don't know” (31%), and the 30% threshold fired. What was broken?
4.What does the baseline rule ask, every hour?
5.This week, a fixed 40% line would also have caught the broken search with no false alarm. So why use a baseline?
6.An alert fires for Wednesday 9 a.m. What do you open first?
Check your understanding
Try each question before opening its answer.
Search breaks and starts returning passages from the wrong articles. Why doesn't the error chart move?
Nothing crashes. Search still returns three passages with confident-looking scores, and the model, given passages that don't hold the answer, correctly says “I don't know”. In the two broken hours, 38 of 40 answers were “I don't know” and there were 0 errors. Watch the “I don't know” rate.
Why chart by the hour instead of by the day?
A day blurs a two-hour problem. Wednesday's 46 “I don't know”s of 150 says something changed, not when or how badly. By the hour: 19 of 21, then 19 of 19. A daily alert also comes late: at midnight, 15 hours after search broke.
Saturday 7 p.m.: 5 of 16 answers said “I don't know”, and the 30% threshold fired. Was something wrong?
No. All five were questions the articles don't answer (Pinecone's price, Kubernetes deployment…), so each “I don't know” was right. Normally about 2 of 16 say it; in a small hour, two or three more crosses 30%. By chance, an hour like that happens about 1 hour in 12.
What does a baseline rule do differently from a threshold?
It learns the normal rate from clean days (15% here), then asks of each hour: if nothing were wrong, how often would an hour this size look this bad? It alerts only below once in 1,000. So a small hour needs a bigger jump: 9 of 16, but 20 of 60.
A fixed 40% line also caught the broken search with no false alarm. Is the baseline worth it?
This week, 40% happened to fit: the highest healthy hour was 31%, the broken ones 90% and 100%. The baseline gets there without anyone guessing the number, and sets the bar from each hour's size; its bar was at least 42% in every hour it judged. Over four kinds of number, thresholds raised 3 false alarms, baselines 1.
An alert fires. What do you open, and what did it show here?
A few traces from that window. Same question, Monday and Wednesday: Monday's search found three fine-tuning passages; Wednesday's found transformers, RAG and AI agents, with exactly the same scores. The alert said where; the trace said why: search, not the model.
Next up: Chapter 42, finding causes in traces: a new version that barely changed the average cost, and the traces that show why long chats got more expensive.