A test gate you can trust

Every change to an AI assistant can quietly break answers that used to be right: new wording in the prompt, a new model, more paragraphs sent. Nobody rereads every answer after every change, so you write an exam of test questions and have it run automatically, before a change goes live. A test gate grades the answers and blocks the change if they got worse. This chapter covers what a gate checks, why it was red on day one, how to see that it catches a real break, and how to tell a real failure from the model's random wobble.

The code is Lab 05's eval suite, run unchanged: its RAG pipeline (gpt-5.4-nano), its checks, its gpt-5.4-mini judge and its baseline gate (evals/gate.py), all wrapped in run_evals.py, which CI runs on every pull request. The help center and the 21 questions are made up for the video and written before any run; the lab's own docs and 18 questions work the same way. Every answer and judge verdict is a real recorded run: the first run, five reruns of unchanged code, two changes that break it on purpose, and five runs of the fix.

How the assistant answers

1. Search: find the 3 paragraphs in the help center that best match the question.
2. Answer: hand them to the model with instructions: answer only from them, name the article in square brackets, and if the answer isn't there, reply exactly "I don't know based on the provided documents."
3. The reply: "How long does an invite link last?" gets invitations expire after seven days, tagged [teams].

An exam, graded one check at a time

21 questions, written before anything ran. 16 have an answer in the help center, and for each the answer key names the paragraph that holds it. 5 have none, like "Do you offer a student discount?", where the right reply is "I don't know", not a made-up discount.

Each answerable question is checked on search and on the answer separately, so a failure points at the part that broke:

"Why is my account locked for 15 minutes?": found ✓, first ✓, sticks to it ✓, names its source ✗. A score is the share of questions passing one check: on day one, 9 of 16 answers named their source.

The gate

The gate compares each new score with the baseline, the scores saved from an earlier accepted run. Think of a bathroom scale: you compare with last week, not with a perfect number. A fixed target ("faithfulness ≥ 0.9") either blocks every change or lets real drops through.

• Allowance (tolerance): a scale wobbles, so the answer scores may dip by up to 0.10. With 16 answers, one miss (0.0625) is fine; two (0.125) is not.
• Search gets none. It returns the same results every run, so any drop is real.
• Critical questions must pass every check, every run, however good the scores are. Here: the student discount and the refund.

Updating the baseline is a deliberate step a person reviews (--update-baseline, committed in the pull request). CI never does it on its own.

Read the red

On day one there was no earlier run, so the run was compared with itself and nothing could drop. It was blocked because a critical question failed. In all, 8 of 21 questions failed a check, in three kinds:

The gate was right to block. Read every failure before touching the gate, the prompt or the allowance.

Break it on purpose

A fire drill: does the gate catch a real break? Deleting the "say I don't know" instruction (one of the lab's Exercise 2 changes):

Without the rule, most unanswerable questions got polite replies in the model's own words, and "Can I pay with Apple Pay?" got a flat "No", though the help center never mentions Apple Pay either way. The check counts only the exact sentence, so a program can tell "no answer" from an answer. The gate blocked it, and the student-discount question, which is critical, failed too.

Meanwhile the source-tag score went up: without that rule, the model tagged almost every answer. One score looked better while the assistant got worse at saying "I don't know". That's why the gate watches every score at once. (One of the judge's four new flags was itself wrong: it called PayPal unsupported, but the billing paragraph lists it. The video leaves that drop out for that reason.)

The lab's other break, sending 1 paragraph instead of 3, dropped search's "found it" from 14 to 12 of 16 and was blocked too (recorded, not shown in the video).

Measure the wobble

The opposite test: will it block a change that did nothing? The same code, five times, without the lab's saved responses (EVAL_NO_CACHE=1, the lab's Exercise 4):

The model doesn't write the same answer twice. "Why is my account locked?" got no tag in run 1, and in run 2 nearly the same answer, one sentence longer, with the tag. Three questions flipped like that, so the score wobbled by two questions (0.125), more than the 0.10 allowance. Save the lucky run (11) as the baseline, and the next run of the same code (9) is a drop of two: blocked, for nothing. The lab's own gate.compare marks it "regressed".

So first measure the wobble by rerunning code you haven't changed, then set the allowance from it, and save a typical run as the baseline, not the lucky one. (In normal CI runs the lab replays saved responses for unchanged inputs, so only the parts a change touched can wobble.)

The fix, and a red that stays

The refund paragraph ranked 4th, so the fix was to send 4 paragraphs. Every answer now sees an extra paragraph, so the whole exam was rerun, five times:

The three questions that flipped stopped flipping in these runs, though the judge wobbled once. The refund question now gets the right answer: a full refund within 14 days of your first purchase.

The gate still says blocked. The refund paragraph is found but ranked 4th, and a critical question must pass every check, including "ranked first". That red is fair: the answer only arrives because a 4th paragraph is sent, and one slip in search would lose it again. The next fix is search itself.

The checklist

1. Read every red before touching the gate, the prompt or the allowance.
2. Break it on purpose and check the gate blocks; watch every score, since one can rise while the assistant gets worse.
3. Rerun unchanged code and check the gate passes.
4. Set each allowance from the wobble you measured, and none for scores that don't wobble, like search.
5. Save a typical run as the baseline, update it only on purpose, and keep critical questions for what must never break.

Practice: the gate

Write the gate's comparison, its critical-question rule and its decision, then measure the wobble. The tests use the scores and checks recorded in the video: day one blocked by refund alone, the fire drill's "improved" source tags, the lucky baseline that blocks unchanged code at 0.10 but not at 0.15, and the 4-paragraph fix that improves two scores and is still blocked.

Quiz

Check your understanding

Try each question before opening its answer.