Key takeaways
- ✓One average can hide two changes at once. v1 and v2 ran side by side on the same days: $0.139 against $0.142 per 1,000 answers. Split by turn, v2 was 21% cheaper on first questions and 44% pricier from turn 8 on.
- ✓Splitting means one number per group instead of one for everything. Split by the labels a change could interact with: version, conversation turn, kind of question, user type. Put the version and the turn on every trace, so you can split later.
- ✓Split by kind of question, the planted worry didn't happen: lists were right every time in both versions (36 of 36, 50 of 50), and v2 missed 2 answers of 292. What changed was citations: v1 cited its source 185 times in 193 answers, v2 never.
- ✓Writing costs more than reading: for gpt-5.4-nano, one written token costs as much as about 6 read ones. On a first question v1 read about 308 tokens and wrote 55, and the 55 cost more. v2's one-sentence answers are about half as long, so first questions got cheaper.
- ✓To follow a conversation, the model is sent what was said before. v1 sends the last 3 turns; the same release dropped that cap, so v2 sends every turn. From turn 8 on, v2 read about 856 tokens per question against v1's 505.
- ✓The averages matched because of the mix: by chance, 23% of v2's answers came at turn 8 or later, against 6% of v1's. With v1's mix of turns, v2 would have averaged about 12 cents, 13% below v1. When two groups have a different mix, compare like with like.
An average is one number for everything. When a release changes two things that pull in opposite directions, the average can stay put while both changes are real. This chapter takes one week of traces from a course assistant, where a new version cost the same as the old one on average, and splits the numbers until the two changes show up: what each one did, why the average hid them, and how to compare two versions fairly.
The source is Lab 14 (Observability), script 05_analysis.py, Exercise 1 (breakdowns), on the lab's replayed week, recorded in October 2026. The video's figures were recomputed from the same trace store. Recording each step as a trace and the replayed week's dashboard and alerts are chapters 36 and 37; user feedback and turning traces into tests are chapter 39. Script 05 also groups failed traces and tests an LLM's explanations; the video leaves those out, and they're summarized briefly near the end.
The assistant and its traces
The assistant answers learners' questions from 7 short articles about AI engineering. For each question it searches the articles, takes the 3 best passages, and has gpt-5.4-nano write an answer from them, citing the article with a tag such as [ai_agents].
Every request leaves a trace: a record of each step, with its timings, tokens and cost. Each trace also carries labels:
- version: which version of the assistant answered (
app.version); - turn: how far into the conversation the question came: a learner's first question is turn 1, their eighth is turn 8 (
app.turn); - kind: what kind of question it was: "what", "how", a list, or one the articles don't answer. This one comes from the generated test set; a live request wouldn't have it.
The week is a replay: 1,055 requests from 140 simulated learners, Monday to Sunday, each a real call to the model. Problems were planted on purpose, so the causes are known. The breakdowns use Thursday to Sunday outside the hours of the other planted problems: 509 answers, 217 from v1 and 292 from v2.
The release
On Thursday, v2 went to half the learners. Someone made it "more concise". Its instructions gained one line:
- Be concise: answer in one short sentence, and leave out citations, they clutter the answer.
Here's one list question, answered by both on a first turn:
| Answer | Words | |
|---|---|---|
| v1 | Evaluation of agents needs to cover loop efficiency (steps to completion) and graceful failure handling, in addition to success rate. [ai_agents] |
21 |
| v2 | It needs to cover loop efficiency (steps to completion) and graceful failure handling besides success rate. | 16 |
The same points in fewer words, and no source. Shorter answers should cost less. But the averages, side by side on the same days, were $0.139 per 1,000 answers for v1 and $0.142 for v2. So split them: one number per group instead of one for everything.
Split by
Number
The same answers, split by turn: v2 is 21% cheaper on first questions and 44% pricier from turn 8 on. One release pushed cost both ways.
Right answers graded by gpt-5.4-mini against a reference answer. Cost includes the search embedding, at list prices. v1's turn-8-or-later row is 14 answers from 3 conversations; the +44% has a 95% interval of +30% to +59%.
Split by kind of question
The worry was lists: one sentence isn't much room. gpt-5.4-mini graded every answer against a reference answer.
| Kind | Answers (v1 / v2) | Right | Cites a source | Words per answer |
|---|---|---|---|---|
| what | 60 / 102 | 100% / 99% | 98% / 0% | 31 / 22 |
| how | 97 / 108 | 100% / 100% | 97% / 0% | 41 / 27 |
| list | 36 / 50 | 100% / 100% | 89% / 0% | 26 / 21 |
| not in the articles | 24 / 32 | 100% / 97% | – | 8 / 9 |
Lists were right every time in both versions. v2 missed 2 answers out of 292, a "what" question and one the articles don't answer, against none for v1 (99% vs 100%, p = 0.22 in the lab).
What changed was citations. When v1 answered, it cited its source 185 times out of 193; v2 never did, in any kind. The answers were still right, so nothing complained: a quality chart, or a user, sees a correct answer. That's the problem chapter 39's tests have to catch.
Split by turn
Now the same answers, split by turn:
| Turn | Answers (v1 / v2) | Tokens read (v1 / v2) | Tokens written (v1 / v2) | Cost per 1,000 (v1 / v2) | v2 vs v1 |
|---|---|---|---|---|---|
| 1 | 103 / 77 | 308 / 326 | 55 / 30 | $0.130 / $0.103 | −21% |
| 2–3 | 68 / 72 | 404 / 400 | 48 / 29 | $0.141 / $0.117 | −17% |
| 4–7 | 32 / 75 | 521 / 542 | 42 / 29 | $0.157 / $0.145 | −7% |
| 8 or later | 14 / 68 | 505 / 856 | 36 / 31 | $0.146 / $0.210 | +44% |
One release pushed cost both ways: cheaper early in a conversation, pricier late. v1 had only a few long chats here (14 answers, from 3 conversations), so +44% is rough: its 95% interval is +30% to +59%. But the code shows why it happens, and that's the next two sections.
Why first questions got cheaper
A model is billed by the token, a piece of a word. It reads tokens (its instructions, the question, the passages, and any earlier conversation) and writes tokens (the answer). Writing costs more: for gpt-5.4-nano, $1.25 per million tokens written against $0.20 per million read, so one written token costs about as much as 6 read ones.
On a first question, v1 read about 308 tokens and wrote 55:
- read: 308 × $0.20 per million = 62 millionths of a dollar;
- written: 55 × $1.25 per million = 68 millionths.
Those few written tokens cost more than everything it read. v2's one-sentence answers are about half as long, 30 tokens, so first questions got 21% cheaper.
Why long chats got pricier
To follow a conversation, the model has to be sent what was said before. Take a learner's eighth question, with seven turns behind it:
- v1 sends only the last 3 turns with the new question (
HISTORY_TURNS = 3in the lab's assistant). - v2 sends all 7, every time. The same release dropped that cap, by mistake.
So from turn 8 on, v2 read about 856 tokens per question, against v1's 505. Its answers were about as short as on turn 1, but reading is now most of the bill, and the longer the chat, the more v2 pays. Move the slider to see what each version sends:
v1 sends the last 3 of 7 earlier turns, then the question, its instructions and 3 passages
v2 sends all 7 earlier turns, then the question, its instructions and 3 passages
Measured, turn 8 or later: cost of one answer, in millionths of a dollar
From turn 8 on, the two answers are about as long, but v2 reads about 850 tokens against v1's 500. Reading is cheap per token, but there's a lot more of it.
The turns sent follow assistant.py (v1: the last 3 turns; v2: no cap, dropped by the same release). Token counts are the measured averages for the turn group. Bars leave out the search embedding, a fraction of a millionth of a dollar per answer, so they read slightly below the cost per 1,000.
How the average hid it
So why did the averages show nothing? Think of two taxi companies. Company B is cheaper on short rides but pricier on long ones. If B's riders happen to take more long rides, B's average fare can come out the same as A's, even though most of its rides got cheaper.
That's what happened. Learners were split between the versions by a hash of their id, and by chance v2's had more long conversations: 68 of v2's 292 answers (23%) came at turn 8 or later, against 14 of v1's 217 (6%). 224 of v2's answers were in the cheaper turn groups, but its expensive long chats weighed far more in its average. Mixed together, the cheaper short chats and the pricier long ones cancelled out.
Compare like with like and the picture changes. With v1's mix of turns, v2 would have averaged about 12 cents ($0.121), 13% below v1. On v2's own mix, v1 would have cost $0.144, a little more than v2's $0.142. Try other mixes:
Average each version over
v1's answers by turn
at measured costs 13.0¢ · 14.1¢ · 15.7¢ · 14.6¢
v2's answers by turn
at measured costs 10.3¢ · 11.7¢ · 14.5¢ · 21.0¢
What the dashboard showed: each version averaged over its own answers. By chance, 23% of v2's answers came at turn 8 or later, against 6% of v1's, so v2's average carries far more of its expensive long chats.
Per 1,000 answers. Long chats: v1 6%, v2 23%. Only “its own answers” was measured as a whole; the other averages reweight the measured cost of each turn group. Learners were split between versions by a hash of their id, so the different mix was chance.
The reweighted averages are computed from the measured cost of each turn group; only each version's average over its own answers was measured as a whole.
The rest of script 05
The lab's analysis script goes further than the video. In brief, from the lab's run:
- Outliers, scored with the median and the median absolute deviation so the outliers don't inflate the yardstick: 24 traces read far more tokens than usual, and 23 of them were v2 conversations at turn 8 or later. The same cause, found another way.
- Grouping the 67 failed traces (k-means, 5 groups). Their causes were wrong search (32), the judge calling a correct "I don't know" bad (17), API errors (15) and wrong answers (3). Grouped by trace features (error, refused, top search score, cited, length, judged bad, and whether the question was answerable, which only a test set knows), groups were mostly one cause each (purity 0.79, with one group holding all 15 errors); grouped by the answers' text, 0.63, because 49 of the 67 answered exactly "I don't know based on the provided documents.", whatever the cause.
- "Find traces like this one": for a bad trace, the 10 nearest by features shared its cause; the 10 nearest by text mostly didn't. Search by meaning finds traces that say similar things, not traces that broke the same way.
- An LLM's explanations are hypotheses. gpt-5.4-mini read 20 failed traces and offered three causes. Each was tested: replaying the broken-search requests with search working took them from 20% to 100% right (confirmed), and judging the 17 refusals again still called 16 of them bad (a real judge problem). None of its guesses blamed the API errors.
What to do
- Never stop at one average. Split every number by version, and by whatever the change could interact with: conversation turn, kind of question, user type.
- Label every trace with the version and the turn (and the user type) when it's recorded, so you can split later. You can't split by a label you didn't record.
- Split more than quality. Here right answers didn't move; citations, length and tokens did.
- Compare like with like when two groups have a different mix: first questions with first questions, or reweight one group's costs by the other's mix.
- Then explain each change from the traces: what was sent and what was written. Here, the prompt line and the dropped history cap.
Caveats
- A replayed week. The learners are simulated and the problems were planted, so the causes are known; in production you'd confirm each cause with a replay (the old prompt, the old history cap) on the same requests.
- Small groups. v1 has 14 long-chat answers from 3 conversations, so +44% has a wide interval (+30% to +59%). The direction is backed by the code; the size is rough.
- Grading is a model's. "Right" means gpt-5.4-mini judged the answer to match a reference answer.
- The citation count is a little low for v1. The lab counts tags like
[ai_agents], and misses a few written as[source_name: rag]. That only undercounts v1, so "almost always cited" holds. - List prices. Costs are at October 2026 list prices and include the search embedding.
- Kind isn't on live traffic. The question's kind came from the test set; in production you'd need a classifier or another label.
The checklist
- Record labels on every trace: version, conversation turn, user type, anything a release could interact with.
- After a release, split each metric by version, then split the changed version by those labels.
- Chart what a release could change, not only quality: citations, length, tokens read and written, cost per group.
- Check the mix before comparing averages: how many of each group does each version have?
- Compare like with like: per group, or reweighted to one mix.
- Open a few traces from the group that moved and compare what each version sent and wrote.
- Confirm the cause with a replay, and add a check for it to the release gate.
Practice: split it before you trust it
Price an answer from its tokens, put each trace in a turn group, split traces and average each group, send history the way each version does, and compare the versions like with like. The tests use the week's recorded costs and counts per turn group.
Quiz
1.v1 cost $0.139 per 1,000 answers and v2 $0.142, on the same days. What can you conclude?
2.The release told the model “answer in one short sentence, no citations”. The worry was list questions. What did splitting by kind show?
3.Why were v2's first questions cheaper?
4.Why did long conversations get pricier with v2?
5.Why did the two averages come out about the same?
6.How do you compare the two versions fairly when their mix differs?
Check your understanding
Try each question before opening its answer.
What does "splitting" a number mean, and what do you split by?
One number per group instead of one for everything. Split by version first, then by the labels the change could interact with: conversation turn, kind of question, user type. Here, splitting v2's cost by turn showed −21% on first questions and +44% from turn 8 on.
Right answers didn't move. What did the release change, and why did nothing complain?
Citations: v1 cited its source in 185 of 193 answers, v2 in none, in every kind. Answers also got about a third shorter. The answers were still right, so neither a quality chart nor a learner would flag them.
Why can 30 written tokens save money when the model reads 300?
A written token costs about 6× a read one ($1.25 vs $0.20 per million for gpt-5.4-nano). On a first question v1's 55 written tokens (68 millionths of a dollar) cost more than its 308 read (62 millionths), so halving the answer cuts the bill by about a fifth.
Why did v2's long conversations cost 44% more?
The same release dropped the cap on history. v1 sends the last 3 earlier turns with each question; v2 sends all of them. From turn 8 on, v2 read about 856 tokens per question against v1's 505.
How did the averages come out the same?
The mix. By chance 23% of v2's answers came at turn 8 or later, against 6% of v1's, so v2's pricier long chats weighed much more in its average and cancelled its cheaper short ones, like a taxi company that's cheaper on short rides but whose riders take more long ones.
How do you compare the two versions fairly?
Like with like: compare the same groups, or reweight one version's per-group costs by the other's mix. With v1's mix of turns, v2 would have averaged about 12 cents, 13% below v1's 13.9.
Next up: Chapter 39 closes observability by turning these traces into tests: user feedback linked to traces, a test set built from them, and a release gate that has to be big enough to tell a fix from noise.