Traces to tests: what feedback misses, and a test too small to tell

Users tell you some of what goes wrong, and traces hold the rest: every question someone asked, with the answer and each step behind it. This chapter covers both. First, feedback saved with the trace of the answer it's about, and what a week of it can and can't show. Then a test set sampled from those traces, a gate that checks what production showed matters, and how to read a result that misses the bar by one answer.

The source is Lab 14 (Observability), scripts 06_feedback.py (a chat page with feedback linked to traces) and 07_traces_to_evals.py (traces → tests), recorded in October 2026. The assistant is gpt-5.4-nano, and gpt-5.4-mini grades the answers. The course site, Quill Academy, is made up, and the week is a replay: simulated learners and problems planted on purpose, but every answer is a real call to the model. Chapter 40 covered recording the traces, chapter 41 the dashboards and alerts on them, and chapter 42 finding causes in them. This chapter, the last on observability, turns those traces into tests.

The assistant and the bad release

The assistant answers learners' questions from 7 short articles about AI engineering, split into 28 paragraphs: it searches for the 3 closest paragraphs and answers from them, citing each source. Every request leaves a trace: a record of each step inside it (search, the model call, the answer), like an itemized receipt.

On Thursday a new version, v2, went out to half the learners. Someone had made it "more concise": one sentence, no sources. And by mistake it dropped the cap on conversation history, so every question carried the whole conversation with it.

Feedback, saved with its trace

The lab's chat page streams each answer, shows its sources as links to the paragraph they came from, and shows "I don't know" as "not sure" instead of a confident guess. Under each answer: helpful, not helpful, retry, edit, or ask for a person. Each click is stored with the trace id of the answer it's about: the receipt's number. The feedback endpoint refuses anything it can't link to a real trace (204 for a real trace id, 400 for an unknown one or an unknown kind).

In the lab's recorded test of the page:

• "What does LoRA change in a model?" The answer cited [fine_tuning], the paragraph search ranked first. Marked helpful.
• "How much does Pinecone cost per month?" "I don't know based on the provided documents." Marked not helpful.

Open that complaint's trace and it says why. Search's top paragraph (score 0.25) begins "Pinecone is a managed cloud vector database with automatic scaling", and none of the three gives a price. That's not a bug in the assistant: it's a gap in the articles. A total of "not helpful" clicks couldn't tell you that; the trace can.

A week of feedback

The lab replays a week of traffic, 1,055 requests, with four problems planted: search returning the wrong passages on Wednesday morning, v2 from Thursday, slow model calls on Friday, and API errors on Saturday. The learners' feedback is simulated from rules written down in the script:

So what this week shows is how little feedback there is and what it can see, not how people behave.

• Most learners said nothing. 90 feedback events landed on 85 of the 1,055 requests: 1 in 12. 41 requests got a complaint (a "not helpful", a retry, an edited question or asking for a person), and 44 were marked helpful.
• Complaints rose on the bad days. Wednesday, when search broke (6.7% of requests, with 21% wrong or failed), Friday, when answers were slow and got retried (5.3%), and Saturday, when the model's API failed (8.7%). Across the 7 days, the complaint rate followed the real failure rate with r = 0.70 (the lab's online judge: 0.89).

Switch the view to see what feedback tracked, and what it didn't:

What feedback can't see

From Thursday, v1 cited a source in most answers and v2 in none. Did learners complain more about v2?

No: fewer. The simulated learners react only to errors, "I don't know", wrong answers and slow ones, so this result is built in. But the argument carries over to real people: they can't miss a source they never saw, and the same goes for an answer from the wrong source or a bill that grew. Feedback catches what users notice. For the rest, you need a test.

Traces → a test set

Traces have a second use: each one holds a question a learner really asked (here, a simulated learner), in the conversation it came from. So the test cases come straight from the week's traces.

07_traces_to_evals.py samples 50, from clean traces only (no planted incident, no error):

• 10 each of four kinds of first question: what, how, list, and ones the articles don't answer.
• 10 long conversations (turn 8 or later), each with its 7 to 17 earlier turns rebuilt from the conversation's traces.

Why not 50 at random? The week's traffic is 40% how and 32% what questions, 12% unanswerable, and only 14% at turn 8 or later. Fifty at random would hold about 6 unanswerable questions and a handful of long chats, and long chats are exactly where v2 went wrong. Stratify by what can fail.

Every case keeps the trace id it came from, so a failing test leads back to what a learner saw.

The gate

Then a gate: checks every new version must pass before it ships. This one checks what the week showed matters:

1. Right: at least 95% of answers right, by mini's grade against the gold answer.
2. Sources: at least 90% of answers cite one, counted on the 38 questions the articles can answer. The other 12 (10 unanswerable first questions and 2 long chats about topics the articles don't cover) have nothing to cite.
3. Long chats: we pay for every token sent to the model. v1 sends about 500 on a long chat, so the bar is 700 on average.

The fix, v3: still concise, two sentences at most, each with its source, and the history cap back. The same 50 cases against all three:

v2 got every answer right, so a test that only checked answers would let it through. Checks 2 and 3 catch both of its problems: what the week's traces showed went wrong became what the gate checks.

v3 got every answer right too, and its long chats are back to their old size. But it cited 34 of 38, and the bar needs 35. One more cited answer, and it would pass. Click any case to see the question, the trace it came from, and what each version did with it:

Is the fix broken, or the test?

Think of free throws. A player who truly makes 9 shots in 10 takes 38 shots a round. Every round lands somewhere else: in the video's eight simulated rounds, the counts were 35, 31, 33, 37, 34, 34, 36 and 37. 34 or fewer happens in 53.5% of rounds, for a player who truly makes 90%.

The 38 answers are one round. So this test fails a fix that's exactly at the bar about every other time. Play some rounds:

Turn it around. If v3 answered thousands of questions, it would likely cite a source in 76% to 96% of them: that's its likely range (a 95% Wilson interval). v1's, 83–99%, overlaps it. On the same 38 questions, v1 cited where v3 didn't 4 times, and v3 where v1 didn't twice. So this test can't tell the fix from the old version. The fail might be bad luck, or a small real drop; 38 answers can't say which. (The lab's own README calls it noise; the data allows either.)

Two ways out:

• More cases, so the range shrinks. At the same rate, even 380 cases give 86–92%: telling 89.5% from 90% takes thousands.
• Fail only when the whole range sits below the bar, like v2's (its top is 21%). v3's isn't. The cost: a real small drop gets through more often, so pair it with a warning when the bar falls inside the range.

Either way, set the bar knowing how much the score wobbles. Chapter 13 measured that wobble by rerunning unchanged code, and Real upgrades vs. noise covers intervals and paired comparisons in depth.

Judging a sample of traffic

The lab's script 07 has one more exercise, not in the video: grading a random 10% of the week's traces with the online judge instead of all of them. That was 108 judge calls instead of 1,055 (the README says 105, the planned tenth). The full judge found the two worst days, Wednesday (14 of 150 judged bad) and Saturday (15 of 150). The sample missed both: 0 of 14 on each, a range of 0–22%. Its only bad answer was Thursday's, so by the sample Thursday looked worst.

A two-hour incident leaves only a handful of sampled traces. Sample on purpose: every error, every "I don't know", every trace with a complaint, plus a random slice of the rest.

What to do

• Save every feedback event with its trace id, so a complaint comes with what the user saw and why.
• Treat feedback as sparse and partial. Most users say nothing, and the ones who do react to what they notice.
• Don't rely on feedback alone. Missing sources, an answer from the wrong source and a growing bill don't get complaints.
• Build tests from traces, stratified: every kind of question, the unanswerable ones, the long chats. Keep each case's trace id, and refresh the set from recent traces.
• Gate on what production showed matters, not only right answers.
• Read the range, not only the bar. A fail by one answer on 38 cases doesn't prove the fix is broken.

Caveats

• Simulated learners. The feedback comes from rules written in the script, so its volume and timing are assumptions. v2's lower complaint rate is built in, since the rules never mention sources. The lab also alerted on feedback and caught the two big incidents within the hour, but that speed comes from those same rules; with fewer people speaking up, it would be slower.
• A replayed, planted week. The course site and its traffic are made up; the incidents were planted so we know what happened. The answers are real model calls.
• Model grading. "Right" is gpt-5.4-mini's judgement against short gold answers written by a model.
• One run per version. Each version answered the 50 cases once; another run would land somewhere else, as the free throws show.
• Small strata. 10 cases per kind can show that something broke, not small differences within a kind.

The checklist

1. Link feedback to traces: store each click with the trace id, and refuse feedback you can't link.
2. Read the traces behind complaints before reading the totals.
3. Compare feedback with what really happened on known bad days, and list what it can't see.
4. Sample a test set from clean traces, stratified by question kind and conversation length, each case with its trace id.
5. Gate on several checks: answers, sources, and the cost of long chats.
6. Put a likely range on every rate, and size the set so the range is narrower than the difference you care about.
7. Grade a purposeful sample of traffic, not only a random one.

Practice: from traces to tests

Write the lab's pieces: tell a complaint from other feedback, sample a stratified test set from a week of traces (each case keeps its trace id), the Wilson range, the gate with both rules, and the free-throw odds. The tests use the recorded results of v1, v2 and v3 on the 50 cases: 36, 3 and 34 of 38 cited.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Chapter 44 starts on safety and security: removing personal data. Traces hold every prompt and answer, so what users type ends up in them; the next chapters keep it out of places it shouldn't be.