Key takeaways
- ✓A model reads the whole prompt first, then writes the answer one token at a time. So total time ≈ the wait for the first token + output tokens ÷ writing speed. Here it usually wrote about 169 tokens a second.
- ✓Streaming sends each token as soon as it's written. The first words came after 0.43–0.66 s, but the whole answer took about as long either way (medians 1.94 s streamed, 1.97 s not). It shortens the blank screen, not the answer.
- ✓Answer length sets most of the total: one sentence (about 40 tokens) took under a second, five paragraphs (about 480) 3–3.7 s. A shorter answer is the cheapest speed-up. Prompt length delays the first word: a 60,000-token prompt took about twice as long (median 1.54 s vs 0.69 s).
- ✓Sixty identical requests averaged 1.06 s, and 19 of 60 were slower than that. p50 (half were faster) was 0.89 s and p95 (only 1 in 20 slower) 1.81 s. With 60 requests, p99 is just the second slowest. Set targets on p95, not the average.
- ✓Two more runs gave nearly the same p50 (0.90 and 0.86 s) but a different slow end: p95 1.19 and 1.14 s, slowest about 1.3 s instead of 3. A timeout at run 2's slowest would have cut off 10 answers in run 1. Measure the slow end over many requests, more than once, and set timeouts well above the slowest seen.
- ✓Thinking off was fast and wrong 60 of 60 on a counting puzzle; low thinking was right 59 of 60 and almost a second slower. The slow end moved too: p95 sat 0.48 s past p50 with thinking off and over a second past with it on. Turn thinking on where it buys right answers, then check p95.
The same question, sent to the same model sixty times, came back in about a second on average. Two out of three answers took under a second. The slowest took three. This chapter is about latency, how long you wait for a model's answer, and how to describe it honestly: what streaming fixes, what sets the total time, and why the average is the wrong number to watch.
It starts a new section on speed and cost, and goes back to the start of Lab 12 (Inference & Serving). Chapters 29–31 covered the lab's second part, serving your own model; this one covers its first two scripts, 01_streaming_latency.py and 02_latency_distribution.py, about calling a model through an API. The numbers are from the run recorded for the video: live calls to gpt-5.4-nano (snapshot gpt-5.4-nano-2026-03-17) on November 2, 2026, from a laptop over home internet, every test three times in a row, for about $0.20. The lab's README describes an earlier run from October; where it differs, it's under "Also in the lab".
How a model writes
A model reads the whole request first. Then it writes the answer one token at a time: a token is a word or a piece of a word. Asked for about 200 words on how a fridge works, it wrote 185 words, which was 240 tokens.
Those are two phases with different costs:
- Reading the prompt happens all at once, and decides when the first token appears. Longer prompts take longer.
- Writing the answer happens one token at a time. Longer answers take longer.
So, roughly: total time ≈ the wait for the first token + output tokens ÷ writing speed. Everything below is one part of that sum.
What sets the wait
Pick a tab. Every bar and dot is one recorded call:
Streamed, the first words came after 0.43–0.66 s. The whole answer took about the same either way (medians 1.94 s and 1.97 s). Streaming shortens the blank screen, not the answer.
Streaming. Normally the API waits until the whole answer is written, then sends it all at once. With streaming, it sends each token as soon as it's written: the first sip while the cup is still filling. Over 9 tries, the first words arrived after 0.43–0.66 s. The whole answer took about the same either way: medians 1.94 s streamed and 1.97 s not. Streaming makes the wait feel shorter; it doesn't make the answer faster.
Answer length. The same question asked three ways:
| Asked for | Output tokens | Total time |
|---|---|---|
| One sentence | 39–44 | 0.77–0.84 s |
| One paragraph | 129–167 | 1.23–3.68 s |
| Five paragraphs | 459–486 | 3.01–3.74 s |
Once it starts, the model usually writes at a steady speed: a median of 169 tokens a second, and 8 of 9 tries between 129 and 186. Once it stalled: a one-paragraph answer of 167 tokens took 3.68 s, 55 tokens a second. Still, more tokens means more time, so the cheapest speed-up is a shorter answer. Ask only for what you'll use.
Prompt length. The lab put more and more background text in front of a tiny question, 9 tries per size:
| Prompt | Median time to the first word |
|---|---|
| 44 tokens | 0.69 s |
| 3,768 tokens | 0.72 s |
| 14,970 tokens | 0.85 s |
| 59,769 tokens | 1.54 s |
Up to 15,000 tokens, the first word usually came in under a second. At 60,000 it took about twice as long as a tiny prompt. But the single slowest first word, 2.87 s, came at 15,000, not 60,000: a slow surprise can hit any request, long or short.
Why averages lie
Back to the sixty identical questions, sent five at a time, like five people asking at once. Say you tell users "answers take about a second", because the average was 1.06 s. 19 of the 60 were slower than that. And the average can't say how much slower: 1.1 seconds, or 3.
A better way: line all sixty up from fastest to slowest.
- p50, the 50th percentile: the one in the middle. Half were faster. Here, 0.89 s.
- p95: near the end. Only one in twenty was slower. Here, 1.81 s.
- p99: one in a hundred. With only sixty requests, that's just the second slowest (2.20 s). To trust a p99, you need many more requests.
The lab computes these by nearest rank: sort the times and take the one at position round(p% × n). Here p95 is the 57th of 60.
Set speed targets on p95, not the average: it shows how long the slower answers take. A timeout is when your app gives up waiting; set it well above the slowest answer you've measured, so it rarely cuts off an answer that's only slow. Try one:
No answer in run 1 is cut off.
Same question, same model, five at a time. Each dot is one answer. Percentiles use the lab’s nearest-rank method: p95 is the 57th of 60 from the fastest.
Run to run
Was that 3-second answer bad luck? The lab sent the same sixty twice more:
| Run | Average | p50 | p95 | Slowest |
|---|---|---|---|---|
| 1 | 1.06 s | 0.89 s | 1.81 s | 3.04 s |
| 2 | 0.91 s | 0.90 s | 1.19 s | 1.26 s |
| 3 | 0.89 s | 0.86 s | 1.14 s | 1.31 s |
The middle barely moved. The slow end did. So yes, partly luck, and that's why one run isn't enough. Say you'd measured only run 2 and set your timeout at its slowest answer, 1.26 s: in run 1 that timeout would have cut off 10 answers. Across all 180, p95 was 1.29 s, 9 were over 1.3 s and only 1 over 2.5 s. Measure the slow end over many requests, and more than once.
Thinking adds to the wait
With thinking on, the model writes hidden working-out before its answer. You never see it, but you wait for it. The lab's puzzle: how many four-digit numbers have digits that add up to nine? (165.) Sixty answers per setting, 20 in each run:
| Thinking | p50 | p95 | p95 past p50 | Right |
|---|---|---|---|---|
| Off | 0.62 s | 1.10 s | 0.48 s | 0 of 60 |
| Low | 1.47 s | 2.65 s | 1.18 s | 59 of 60 |
| Medium | 1.60 s | 2.65 s | 1.05 s | 60 of 60 |
Thinking off was fast because it only writes a short number, and it was wrong every time. Low thinking first wrote about 124 hidden tokens (the median), almost a second more, and was right almost every time. Medium was about the same as low here.
Thinking slowed everything down, the slow end too: p95 sat half a second past p50 with it off, and over a second past with it on. Part of the reason is that how much it thinks varies: 73 to 362 hidden tokens on this one puzzle, about 5×. But the two slowest answers (5.05 s and 3.80 s) thought an ordinary amount, 143 and 159 tokens. Something else held them up.
Also in the lab
From the lab's README and checklist answers, and from the recorded run where noted:
- The October run differed. The README's run had p50 1.10 s and p95 1.77 s on the sixty requests, and its 60,000-token prompts were unpredictable: median first token 5.13 s, from 1.37 s to 17.79 s. In the November recording, 60,000 tokens was slower but steady (1.16–1.86 s). Same scripts, different day: your numbers will be your own.
- The ratio didn't grow. The README describes thinking as stretching p95 ÷ p50 (1.5× off, about 2× on). In the recording the ratio held about level (1.77, 1.80, 1.66 pooled); the slow end moved further out in seconds, not in ratio.
- Parallel calls (recorded, cut from the video). 20 requests one at a time took 18.6–20.6 s; 20 at once took 1.9–2.4 s, 8–11× faster overall. Each request didn't get faster: its p50 went from 0.90–1.02 s to 1.10–1.38 s. Parallel calls raise throughput (requests finished per second), not per-request speed, until you hit your rate limit.
- Agents feel slow for a reason. An agent that makes 5 calls in a row has a worse p95 than any single call, because each call has its own chance of being slow (checklist answer 3).
What to do
- Stream anything a person reads, so the first words show up fast. When code needs the whole output (JSON, a tool call), streaming can still drive a "working…" state.
- Ask for shorter answers. Every token takes time to write.
- Watch p50 and p95, not the average, over many requests, and more than one run.
- Set timeouts well above the slowest answer you've seen, not at the average or one run's slowest.
- Turn thinking on only where it buys right answers, then check what it does to p95.
Caveats
- One model, one laptop, one day.
gpt-5.4-nanoover home internet; network and provider load move every number here. - Small samples. 60 requests per run gives a rough p95 and a p99 that is just the second slowest. 9 tries per prompt size and 3 per answer length show the shape, not precise speeds.
- One question. The sixty requests are the same short question; a real workload mixes lengths, which spreads the times further.
- Targets and timeouts are guidance. "Set targets on p95" and "timeouts well above the slowest seen" are advice from these results, not something the lab measured.
The checklist
- Measure the first token and the total separately. They have different causes.
- Stream user-facing answers.
- Cap and shorten answers where you can.
- Report p50, p95 and the slowest, with how many requests, not the average.
- Repeat the measurement on another run or day before setting a target.
- Set timeouts well above the slowest seen, and count what a candidate timeout would have cut off.
- Re-measure p95 when you turn thinking on or raise its level.
Practice: percentiles and timeouts
Write the lab's nearest-rank percentile, count the answers slower than the average, check a timeout from one run against another, estimate an answer's total time, and measure how far thinking pushes the slow end. The tests use all three recorded runs of sixty and the 180 thinking answers, and expect the recorded numbers: p50 0.89 s and p95 1.81 s, 19 slower than the average, 10 cut off, and a slow end of 0.48, 1.18 and 1.05 s.
Quiz
1.What does streaming change?
2.An answer is slow. Which change usually saves the most time?
3.Sixty identical requests averaged 1.06 s. Why is the average a poor speed target?
4.With 60 requests, what is p99?
5.Run 1's slowest answer took 3.04 s; runs 2 and 3 topped out at about 1.3 s. What should you take from that?
6.Thinking on a counting puzzle: off was right 0 of 60, low 59 of 60. What did low thinking cost?
Check your understanding
Try each question before opening its answer.
What are the two parts of a model's response time, and what makes each one longer?
The wait for the first token, which grows with the prompt (the model reads all of it first), and the writing, which grows with the answer (one token at a time, about 169 a second here). Total ≈ first token + output tokens ÷ writing speed.
Your app parses JSON from the model. Does streaming help?
Not with the total: the whole answer takes about as long either way (1.94 vs 1.97 s here). It can still show a "working…" state, or let you parse fields as they arrive.
Why did 19 of 60 identical requests take longer than "the average"?
Response times have a long slow end. A few slow answers (up to 3.04 s) pull the average up to 1.06 s, above most answers' 0.8–0.9 s, and the average then says nothing about how slow the slow ones were.
Why is p99 from 60 requests not worth much?
Nearest rank makes it the second slowest answer. One more slow request moves it a lot. You need hundreds of requests or more before p99 means anything.
Run 2's slowest answer took 1.26 s. Why not set the timeout there?
Run 1 had 10 answers slower than that. The slow end changes from run to run, so a timeout from one run's slowest cuts off answers that were only slow. Set it well above the slowest you've measured over several runs.
Thinking fixed the puzzle. What do you check before shipping it?
p95, not just p50. Low thinking added almost a second at p50 and moved p95 from 0.48 s past p50 to 1.18 s past it. If the product target is on p95, that's the number that decides.
Next up: Chapter 33 makes each call cheaper: prompt caching, how one timestamp broke it, and the Batch API's half price for work that can wait.