Key takeaways
- ✓Batching uses one read of the model's numbers to write a token for every reply in the batch. On Qwen3-4B (4-bit), one reply alone got 138 tokens a second; 32 at once got 482 in total (3.5×), but only 15 each, so one 128-token reply took 8.4 s instead of 0.9 s. Throughput goes up, each reply gets slower.
- ✓With 40 simulated customers arriving at about 4 a second, a server that answers one at a time finishes about 1.08 a second, so the line only grows. The typical customer waited 13.8 s for the first word (the longest 27.6 s), then got the whole reply in 0.81 s. At 1 a second, the same server's typical wait was 2.2 s.
- ✓Static batching (a bus that leaves with 8 requests or after 0.5 s, and runs every reply to the end) cut the typical first word to 4.5 s. Continuous batching, where new requests join at the next step, cut it to 0.43 s: about 30× sooner than one at a time.
- ✓Continuous batching mostly cuts the wait to start. The typical whole reply went from 14.6 s (one at a time) to 6.4 s, about 2×, not 30×, because writing took 5.97 s instead of 0.81 s on a crowded, shared chip. Static batching was nearly as quick to finish (6.7 s).
- ✓Prefix caching: every email-sorting prompt started with the same 1,777 tokens, then about 87 of its own. Making the model's notes on the shared start (the KV cache) once and copying them cut the typical first word from 2.96 s to 0.21 s (14×) on Qwen3-8B. 19 of 20 answers were the same.
A model you serve yourself rarely has one user. Customers send requests at random times, and each one waits for the server to get to it. This chapter covers how one model serves many people at once: batching, the three ways a server can let new requests in, and prefix caching, which skips re-reading the start that every prompt shares.
The examples come from Lab 12 (Inference & Serving), script 09_batching.py and the prefix-caching exercise of 11_kv_cache.py, run with the open Qwen3 models on one laptop (Apple M4 Max) with MLX. The model writes replies to made-up support emails for an outdoor gear shop. For the load test, the video reran the lab's own exercise (same model, same 40 simulated arrivals, same scheduler) and kept every request's timing, which the script only summarises; its typical first-word times are within 10% of the lab's recorded run. The arrivals are simulated; the model and the timings are real. Chapter 29 covers why a model is slow on your own machine and what shrinking it costs; the lab's API half (streaming, prompt caching, the Batch API) and its speculative-decoding exercise aren't covered here.
Why one reply wastes the chip
A model writes one token (about three quarters of a word) at a time. For every token, the chip reads every number in the model from memory. That reading is the slow part: the arithmetic done with each number is tiny, so most of the chip's arithmetic sits idle, waiting for memory. (Chapter 29 measures this: a rough speed limit of memory speed ÷ model size.)
So here's the idea: while each number is read for one reply, use it for other replies too. That's batching. One pass through the model writes one token for every reply in the batch. Think of a bus instead of a taxi: one trip, many passengers.
More in total, slower for each
The lab ran Qwen3-4B, shrunk to 4 bits, on batches of 1 to 64 replies that all start together, up to 128 tokens each. Tokens a second here means tokens written ÷ wall time, including reading the prompts.
| Replies at once | Tokens/s, in total | Tokens/s, each reply | Time for every reply |
|---|---|---|---|
| 1 | 138 | 138 | 0.93 s |
| 2 | 227 | 114 | 1.13 s |
| 4 | 239 | 60 | 2.15 s |
| 8 | 292 | 37 | 3.50 s |
| 16 | 345 | 22 | 5.94 s |
| 32 | 482 | 15 | 8.42 s |
| 64 | 462 | 7 | 17.57 s |
Thirty-two replies at once made 3.5× the tokens of one, from the same chip. But each reply got only 15 tokens a second, so one full reply took about 8.4 s instead of 0.9 s. Batching does more work in total, and each reply is written more slowly. A server's knob is how many replies it runs at once, which trades each user's speed for total throughput.
At 64 the total fell. The 4-bit model has to unpack its numbers for every multiply, and with many replies that unpacking is real arithmetic, so it has less spare capacity to batch into. The lab's full-size (bf16) copy of the same model gained 8.6× from batching (51 → 439 tokens a second at 64), against 3.5× here, and at 64 both served about the same. Quantization helps the single user most; batching helps the server most; together, they don't multiply.
32 replies at once made 3.5× the tokens of one, from the same chip. But each reply got 15 tokens a second instead of 138, so all 32 took 8.4 s, not 0.93 s.
Tokens a second = tokens written ÷ wall time, including reading the prompts. Each batch is one run; the times are from one laptop.
A rush, one at a time
Real customers don't arrive in neat groups. The load test sends 40 simulated requests at random times, about 4 a second (all 40 within 8.65 s), with the same arrival times for every server design.
One at a time (a taxi): the next request starts when the last one has finished. The laptop finishes about 1.08 replies a second, but four arrive every second, so the line only grows.
- The typical customer (the median, the one in the middle) waited 13.8 s for the first word. The longest wait was 27.6 s. All 40 were done at 37.1 s.
- Each reply itself was written in 0.81 s (typical). Almost all of the time was waiting in line.
At lower traffic the same server looks fine: at about 1 request a second, its typical first word came after 2.2 s; at 2 a second, 9.7 s. That's why a load test has to go past your normal traffic. In the lab's own run, at 0.5 requests a second, the three designs were within a fraction of a second of each other.
The waiting bus and the hop-on bus
Static batching (the waiting bus) collects requests and leaves when 8 are on board or the oldest has waited 0.5 s. Then it writes every reply to the end before it takes anyone new. The typical first word came after 4.5 s: better, but anyone who arrives while the bus is out waits for it to come back.
Continuous batching (the hop-on bus) never goes back to the depot. It writes one token for everyone on board; then, before the next token, new requests hop on and finished ones hop off. Each of those rounds is a step. The typical first word came after 0.43 s, about 30× sooner than one at a time (32× unrounded).
At about 4 requests a second:
| Server | Typical first word | Longest wait for it | Typical writing time | Typical whole reply | All 40 done |
|---|---|---|---|---|---|
| One at a time | 13.8 s | 27.6 s | 0.81 s | 14.6 s | 37.1 s |
| Static batches (8 or 0.5 s) | 4.5 s | 7.8 s | 2.38 s | 6.7 s | 17.7 s |
| Continuous batching | 0.43 s | 1.1 s | 5.97 s | 6.4 s | 13.7 s |
Each column is its own median, so the columns needn't add up.
So do whole replies finish 30× sooner too? No. One at a time, a customer waited about 14 s, then got the reply in under one. With continuous batching they waited under half a second, but the reply took about six, because the bus is crowded (up to 33 replies at once, of the scheduler's 64 seats) and every passenger shares the chip. That's the batching trade from before. The whole reply was about 2× sooner (14.6 s → 6.4 s), and the waiting bus was nearly as quick (6.7 s).
Continuous batching mostly cuts the wait to start. That matters in a chat, where the reply appears word by word: the first word is when the customer stops staring at a blank screen. It's what mlx_lm.server, vLLM and TGI do.
Pick an arrival rate and a server, and see where each customer's time went:
a taxi: the next request starts when the last one has finished
All 40 arrived within 8.7 s and were done at 37.1 s; the server finished 1.08 requests a second. Click a row to see one customer.
Same 40 arrival times for every server; up to 128 tokens per reply. “Typical” is the median, and each column is its own median, so they needn’t add up. One run each, rerun for the video: within 10% of the lab’s recorded first-word times at 4 a second.
The lab's own run
The lab's recorded run of the same test (before the video's rerun) reports the median and 95th percentile (p95, the time 95% of requests beat):
| Requests/s | Server | First word p50 | First word p95 | Whole reply p50 | Whole reply p95 | Tokens/s served |
|---|---|---|---|---|---|---|
| 1 | one at a time | 2.13 s | 3.61 s | 2.9 s | 4.4 s | 132 |
| 1 | static | 0.70 s | 1.04 s | 1.6 s | 2.0 s | 145 |
| 1 | continuous | 0.13 s | 0.21 s | 1.1 s | 1.6 s | 145 |
| 2 | one at a time | 9.38 s | 16.4 s | 10.2 s | 17.2 s | 145 |
| 2 | static | 0.93 s | 1.76 s | 2.0 s | 3.1 s | 261 |
| 2 | continuous | 0.20 s | 0.34 s | 2.1 s | 2.9 s | 270 |
| 4 | one at a time | 13.2 s | 25.1 s | 14.0 s | 26.0 s | 141 |
| 4 | static | 4.61 s | 7.83 s | 7.0 s | 10.3 s | 281 |
| 4 | continuous | 0.46 s | 1.02 s | 7.0 s | 8.2 s | 390 |
One at a time can't serve more than about 140 tokens a second however many requests come. Continuous batching served the most at 4 a second (390). An earlier run gave numbers within about 20% of these, with one difference: at 2 requests a second, continuous batching's median whole reply was worse than static's (4.0 vs 3.0 s). With 40 requests, the finishing-time gap between those two is within noise; the first-word advantage held in both runs, at every rate.
If your target is "95% of replies under 3 s", this laptop takes about 2 requests a second, with static or continuous batching; at 4 a second, continuous batching's p95 was already 8.2 s.
Reuse the start: prefix caching
A different kind of request: sorting support emails, with a long prompt. Every request starts with the same rules and ten examples, 1,777 tokens, and only the email at the end is new, about 87 tokens (about 1,864 in all). This test ran on the bigger Qwen3-8B (8-bit); the trick works the same way on any model.
Before writing, the model goes through the whole prompt. As it reads, it keeps notes on every token (its keys and values for each layer), so it never has to go back over the earlier words. Those notes are the KV cache. Prefix caching makes the notes for the shared start once (252 MiB here) and gives each request a copy, so only the request's own part is read. Like a teacher who reads the rulebook once in the morning, not before every question.
| 20 test emails, one at a time | Prompt tokens read | First word, typical | Whole answer, typical | All 4 fields right |
|---|---|---|---|---|
| No prefix cache | 1,864 | 2.96 s | 3.80 s | 11 / 20 |
| Prefix cached | 87 | 0.21 s | 1.14 s | 12 / 20 |
The first word came 14× sooner. Nineteen of twenty answers were the same either way. The one that changed (and happened to go from wrong to right) most likely comes from tiny rounding differences in the reused notes: reuse is exact in exact arithmetic, but not always in 16-bit floating point. Repeating plain decoding gave identical answers every time, so it isn't randomness.
The rule: put the part that never changes first. A timestamp or user name at the top of the prompt makes every request's start different, and nothing can be reused. Paid APIs do the same thing as a discount (prompt caching); self-hosted, you get the time back directly.
What Lab 12 also covers
The video sticks to batching and prefix caching. Script 11 has two more KV-cache exercises, with numbers from the lab's own run:
- How big the KV cache is. Per token: 2 × layers × KV heads × head size × 2 bytes. Qwen3-8B: 144 KiB per token, 4.8 GB at 32,000 tokens, and a server holds one cache per active request: 32 users at 32K tokens would need about 154 GB of cache for 16 GB of weights. Qwen3-0.6B's cache passes its own weight size at about 10,000 tokens. Grouped-query attention (8 KV heads shared by 16 or 32 query heads) already makes it 2–4× smaller.
- A quantized KV cache. At 8 and 4 bits the cache shrank 47% and 72%. Finding one customer's name in a long pile held up at about 4,000 tokens (as far as 10 questions can show); at about 16,500 tokens the model failed most questions whatever the cache (3 of 10 found at 16 bits, 1 at 4), and reading the prompt got slower (30.7 → 36.6 s). It's a memory trade, not a speed one, and
mlx_lm.serverturns batching off when you use it.
Caveats
- Simulated arrivals. The 40 customers are random arrival times with a fixed seed; the model, prompts and timings are real.
- One laptop, one run per setting. Speeds are this Apple M4 Max with MLX. A GPU server with vLLM will give other numbers; the shape (a growing line one at a time, the first word saved by continuous batching, slower writing when crowded) is the lesson.
- Small samples. 40 requests per run, 20 emails for prefix caching. Whole-reply times between static and continuous batching were within noise, and flipped between runs.
- Two copies of the load test. The tables in the middle of the chapter and the widget come from the video's rerun, which kept every request's timing; the lab's recorded run is the table after them. They agree within 10% on typical first-word times.
- Different models. Batching and the load test use Qwen3-4B (4-bit); prefix caching uses Qwen3-8B (8-bit).
- A from-scratch scheduler. The three server designs are the lab's own scheduler on top of MLX's batch generator, so they differ only in when a waiting request is let in. Production servers add memory management (vLLM's PagedAttention) and limits this one doesn't have.
The checklist
- Serve with continuous batching (vLLM, TGI, or
mlx_lm.serveron a Mac) rather than one request at a time. - Load-test past your normal traffic, with random arrivals, and report the first word and the whole reply, at p50 and p95.
- Pick how many replies run at once from a latency target: more at once means more throughput and slower replies.
- Watch the first word, not only the total: in a chat, that's when the customer stops waiting.
- Put the part of the prompt that never changes first, so its KV cache can be reused, and keep timestamps and names out of it.
- Check that answers stay the same when you turn on prefix caching or another speed-up; floating-point rounding can change a few.
Practice: the rush, request by request
Compute a batch's tokens a second in total and per reply, the lab's nearest-rank median, and a load test's typical first word, writing time and whole reply. Then build the server designs' rules: a one-at-a-time queue, and when the static-batching bus leaves and how many board. The tests use the recorded rerun at about 4 requests a second: 13.82 s, 4.46 s and 0.43 s typical first words, and the six static batches exactly as they left (2, 7, 8, 8, 8 and 7 requests).
Quiz
1.Why can a batch make many replies for nearly the cost of one, at first?
2.On Qwen3-4B (4-bit), one reply alone got 138 tokens a second, and 32 at once got 482 in total. What did each of the 32 replies get?
3.40 customers arrive at about 4 a second, and the server answers one at a time at about 1 a second. What happens?
4.Why did static batching still make the typical customer wait 4.5 s for the first word?
5.Continuous batching made the first word about 30× sooner than one at a time. Why was the whole reply only about 2× sooner (14.6 s → 6.4 s)?
6.What does prefix caching save, and when does it work?
Check your understanding
Try each question before opening its answer.
Why does batching raise throughput, and what does it do to each reply?
Writing one token reads every number in the model, and the chip's arithmetic mostly sits idle. A batch uses each number it reads for every reply in it: 32 replies at once made 482 tokens a second in total, 3.5× one reply's 138. But each reply shares every step, so each got 15 tokens a second, and a 128-token reply took 8.4 s instead of 0.9 s.
Why did one-at-a-time serving fall apart at about 4 requests a second?
It finishes about 1.08 replies a second, so with four arriving every second the line only grows. The typical customer waited 13.8 s for the first word (the longest 27.6 s), though each reply was written in 0.81 s. At 1 request a second the typical wait was 2.2 s, which is why a load test has to go past normal traffic.
How do static and continuous batching differ?
Static batching leaves with 8 requests or after 0.5 s and runs every reply to the end, so anyone arriving meanwhile waits for the whole batch: typical first word 4.5 s. Continuous batching lets new requests join at the next step and finished ones leave: typical first word 0.43 s.
The first word came about 30× sooner. Why was the whole reply only about 2× sooner?
The wait to start almost vanished, but writing took 5.97 s instead of 0.81 s, because up to 33 replies at once shared the chip. Whole replies went 14.6 s → 6.4 s, and static batching was nearly as fast (6.7 s). Continuous batching's win is the first word, which ends the blank screen.
Which numbers in a load test show the difference between server designs?
The wait before starting and the time to the first word, at p50 and p95, as the request rate rises; plus the whole reply and tokens a second served. In the lab's run at 4 requests a second: first word p50 13.2, 4.6 and 0.46 s; tokens a second 141, 281 and 390.
What does prefix caching save, and how do you design a prompt for it?
When prompts share a start, the model's notes on it (the KV cache) are made once and copied, so only each request's own part is read: 87 tokens instead of about 1,864, and the first word in 0.21 s instead of 2.96 s. Put the fixed rules and examples first and anything that changes, such as the email or a timestamp, after them. Check the answers too: 19 of 20 were the same.
Next up: Chapter 31 puts Lea's fine-tuned laptop model against a paid API: right answers, speed, and what an email really costs.