Key takeaways
- ✓Writing each token reads every weight of the model from memory, so decode speed is capped by memory bandwidth ÷ bytes of weights. This laptop reads 464 GB/s; Qwen3-8B's 16.4 GB gives a rough ceiling of about 28 tokens/s, and it measured 25. With 14× the bytes of Qwen3-0.6B, it was 11× slower.
- ✓Quantization stores each number in fewer bits, so it is rounded to fewer possible values: 256 at 8 bits, 16 at 4, 8 at 3. Over one real weight matrix, numbers ended up 0.6%, 9.9% and 20.6% off on average. Each group of 64 numbers keeps its own scale, so 4 bits is just under a third of the size, not a quarter.
- ✓Qwen3-4B at 8 bits: 4.3 GB instead of 8.0, faster, the same 66% on 300 emails, and 99% of answers unchanged. At 4 bits: 2.3 GB, nearly 3× as fast, 60%. At 3 bits: 39%. Shrinking buys speed; how much it costs depends on the bits.
- ✓The score hides how much changed. At 4 bits, 50 of 300 answers changed (17%), but only 22 right → wrong and 3 wrong → right move the score, so it fell 6 points. Compare answers with the full-size model's, not only the score.
- ✓The short test missed the long input. Finding one customer's email in a pile of about 50 (about 4,000 tokens), Qwen3-4B got 8 of 10 at full size, 7 at 8 bits, and 1 at 4 bits. The lab measured that this happens, not why.
- ✓Same rounding, different damage: the fine-tuned 0.6B from the fine-tuning series kept 99% at 4 bits (0.34 GB, 98% of answers unchanged) and fell to 61% at 3 bits. 8 bits is a safe default; below that, test the shrunk model itself, on its task and on its longest inputs.
Running a model yourself, instead of paying a company per answer, is called serving. On your own machine the first surprise is speed: a free open model can write slowly, and the bigger it is, the slower. This chapter covers why, and what the usual fix, quantization (storing each number in fewer bits), buys and costs. The short version: it buys a lot of speed, it cost a few points on the usual test, and it broke a test nobody had run.
The examples come from Lab 12 (Inference & Serving), Part 2, scripts 07_local_speed.py (speed on your own hardware) and 08_quantization.py (quantization), recorded on one laptop (an Apple M4 Max) with free Qwen3 models through MLX. The 300 support emails are made up, with known answers: the outdoor-gear shop's emails from the fine-tuning chapters. Lab 12's Part 1 (streaming, prompt caching, the Batch API, a semantic cache and a cascade over paid APIs) and its scripts on speculative decoding and the KV cache aren't covered here. Batching is chapter 30; self-hosting against an API is chapter 31.
One token at a time
A model is a huge list of numbers learned in training (its weights): 0.6 billion for the smallest Qwen3 here, 8 billion for the biggest. It writes one token (about three quarters of a word) at a time, and to pick each token, the chip does math with every weight. So writing a token means reading the whole model from memory, and the next token reads it all again.
That makes decoding (writing the answer) limited by memory bandwidth, how many bytes per second the chip can read, not by how fast it does math:
decode tokens/s ≤ memory bandwidth ÷ bytes of weights
Script 07 measured this laptop at 464 GB/s. Qwen3-8B at 16 bits is 16.4 GB, so 464 ÷ 16.4 gives about 28 tokens/s. It measured 25. That's a rough ceiling, not an exact one: the bandwidth test runs below the chip's quoted peak (546 GB/s), and other runs of the same model reached 29–31.
Bigger is slower
If reading is the limit, a smaller model should write faster, roughly in step with its size:
| Model | Weights (16-bit) | Ceiling (464 ÷ GB) | Measured tokens/s | Share of the ceiling |
|---|---|---|---|---|
| Qwen3-0.6B | 1.2 GB | 389 | 284 | 73% |
| Qwen3-1.7B | 3.4 GB | 135 | 118 | 88% |
| Qwen3-4B | 8.0 GB | 58 | 45 | 78% |
| Qwen3-8B | 16.4 GB | 28 | 25 | 89% |
The biggest has 14× the bytes of the smallest and is 11× slower. The smallest gets the lowest share because fixed per-token costs (Python, launching work on the chip) matter more when each token is quick. So the way to write faster on the same chip is to give it fewer bytes to read.
464 GB/s ÷ 16.4 GB ≈ 28 tokens/s at most (a rough ceiling)
One request, 200 tokens each. The biggest has 14× the bytes of the smallest and is 11× slower. The bandwidth test runs below the chip's quoted peak, so other runs of the 8B model reached 29–31 tokens/s.
Shrinking: fewer bits
Numbers are stored in bits, each a 0 or a 1. A model is trained with 16 bits per number, which allows 65,536 possible values. Quantization stores each number with fewer bits, so with fewer possible values, and every number is rounded to the nearest one: like measuring with a ruler that has fewer marks.
| Bits | Possible values | Average rounding error (one real matrix) |
|---|---|---|
| 16 | 65,536 | as trained |
| 8 | 256 | 0.6% |
| 4 | 16 | 9.9% |
| 3 | 8 | 20.6% |
The errors are the average size of the change over one whole weight matrix of Qwen3-4B (layer 10, 4096 × 2560 numbers), rounded with MLX's own quantize, as script 08 does. Each group of 64 numbers gets its own scale and offset, its own ruler stretched to fit it. Storing those costs about half a bit per number, so 4 bits ends up just under a third of the 16-bit size, not a quarter (Qwen3-4B: 2.3 GB, not 2.0).
Here are five real numbers from one group, at each size. At 4 bits, the smallest became 0:
Store each number in
| As trained | 4 bits | Off by |
|---|---|---|
| 0.0050 | 0.0000 | −0.0050 |
| -0.0544 | -0.0528 | +0.0016 |
| -0.0432 | -0.0422 | +0.0010 |
| 0.0337 | 0.0317 | −0.0020 |
| 0.0227 | 0.0211 | −0.0016 |
Rounded with MLX's own quantize, as in script 08: each group of 64 numbers gets its own scale and offset (its own ruler), which is why 4 bits ends up just under a third of the size, not a quarter. The average error is over the whole 4096 × 2560 matrix, not these five.
What it buys, and what it costs
Script 08 ran Qwen3-4B at each size on the usual test: 300 made-up support emails, each prompted with the full rules and 10 worked examples (about 1,800 tokens). For each email the model fills in four things (what the customer wants, the order number, the product, and how urgent it is), and an answer counts as right only if all four are right. "Same answer" counts the emails where the shrunk model gave exactly the 16-bit model's answer, right or wrong.
| Qwen3-4B | Weights | Tokens/s | All 4 right (n = 300) | Same answer as 16-bit |
|---|---|---|---|---|
| 16-bit | 8.0 GB | 55 | 66% | — |
| 8-bit | 4.3 GB | 94 | 66% | 99% |
| 4-bit | 2.3 GB | 151 | 60% | 83% |
| 3-bit | 1.8 GB | 170 | 39% | 49% |
These speeds come from script 08's second run, timed on a support-email prompt, so the 16-bit 55 isn't the same measurement as script 07's 45. Speeds vary a little from run to run.
- 8 bits was free here: half the size, faster, the same score, 99% of answers unchanged.
- 4 bits is under a third of the size and nearly 3× as fast, for 6 points (a significant drop by McNemar's test in the lab).
- 3 bits breaks it, and doesn't buy much more speed.
The score hides how much changed. At 4 bits, 50 of 300 answers changed: 22 went from right to wrong, 3 from wrong to right, and 25 from one wrong answer to a different wrong one. Only the right ↔ wrong flips move the score, a net 19 answers, about 6 points. So the score fell 6 points, yet one answer in six changed. A regression test that compares only scores would understate how different the shrunk model is.
Qwen3-8B followed the same pattern: 76%, 76%, 71% and 61% at 16, 8, 4 and 3 bits, with 99%, 88% and 70% of answers unchanged. At 4 bits (4.6 GB) it beat the 16-bit Qwen3-4B (8.0 GB) on this test, 71% against 66%, in less memory.
A long pile: find one name
The short test asks about one email at a time. Script 08 also ran a long test: paste many emails together, about 50 for 4,000 tokens (around six pages), and ask one question, such as "What is the first name of the customer whose email mentions order A-3235? Reply with the name only." The answer is one name, so it's easy to check. Ten questions per length:
| Qwen3-4B | ~1,500 tokens | ~4,000 tokens | ~8,000 tokens |
|---|---|---|---|
| 16-bit | 10/10 | 8/10 | 7/10 |
| 8-bit | 10/10 | 7/10 | 6/10 |
| 4-bit | 8/10 | 1/10 | 1/10 |
| 3-bit | 4/10 | 1/10 | 0/10 |
The 4-bit model lost 6 points on the short test, and here fell from 8 of 10 to 1 of 10. With 10 questions the intervals are wide (8/10 is 49–94%, 1/10 is 2–40%), but they don't overlap. In the A-3235 question (52 emails, about 4,300 tokens), the full-size model said Fatima, which is right; the 4-bit model said Ravi, another customer. At 16, 8 and 4 bits, every miss named another customer from the pile; it never said it couldn't find the email. At 3 bits, 7 of its 24 misses weren't a customer in the pile at all: made-up names or broken sentences.
Qwen3-8B held up better at 4 bits (7 of 10 at 4,000 tokens, against 9 at 16 bits) but still lost ground at 8,000 (4 against 7).
Why does a long input break it? The lab measured that it happens, not why. A likely guess: small rounding errors add up over thousands of tokens, while the model has to pick out one small signal among many.
So that's what the short test missed. It asked about one email at a time, with a few examples; it never asked the model to find one email in a pile of fifty.
50 of 300 answers changed: 22 right → wrong, 3 wrong → right, 25 wrong → a different wrong. Only the first two move the score: net 19 answers, about 6 points.
The long-pile test · find the one email about an order, name its customer
Email 13 of 52 · about 4,335 tokens in all
“Hi Lantern Outfitters, I’m writing about my order A-3235—I received the wrong color of the water filter, and I think it was a mix-up; could you please help me sort this out? Thanks so much, Fatima”
Asked: what is the first name of the customer whose email mentions order A-3235? The other emails are from other customers, about other orders.
Graded as the lab does: the reply, trimmed of spaces and full stops, must equal the name. At 16, 8 and 4 bits, every miss named another customer from the pile; at 3 bits, 7 of 24 misses for the 4B model didn't. Ten questions per length, so the intervals are wide.
Speeds are from script 08's second run, timed on a support email, so the 16-bit 4B (55) differs from script 07's 45. Speeds vary a little from run to run.
The fine-tuned model, shrunk
Does shrinking always cost points? Script 08 also shrank the fine-tuned Qwen3-0.6B from chapter 23 (its LoRA merged into the weights first). Its job is one email at a time, never a pile, so its own test (the same 300 emails, no examples in the prompt) is the one that counts:
| Bits | Weights | All 4 right | Same answer as 16-bit |
|---|---|---|---|
| 16 | 1.19 GB | 99% | — |
| 8 | 0.63 GB | 99% | 100% |
| 4 | 0.34 GB | 99% | 98% |
| 3 | 0.26 GB | 61% | 59% |
At 4 bits it lost nothing, in a third of a gigabyte. At 3 bits it fell the furthest of the three models, 38 points. The same rounding changed 17% of the prompted 4B's answers and 2% of the fine-tune's. A plausible reason, not tested: a model trained for one narrow job is sure of its answers, so there are few close calls for rounding to tip. You can't assume either way; test the shrunk model itself. This 4-bit copy is the one chapter 31 puts up against a paid API.
Caveats
- One laptop. Speeds are an Apple M4 Max's, through MLX, and vary from run to run (Qwen3-8B at 16 bits: 25 in one run, 29–31 in others). Compare speeds from the same run, as each table here does.
- MLX's quantization. Affine, in groups of 64, at 8, 4 and 3 bits. It's the same idea as GPTQ- or AWQ-style 4-bit, not the same algorithm; GPU formats like FP8 weren't tested.
- Made-up emails. Both tests use synthetic support emails with known answers. The long test has only 10 questions per length.
- The why isn't measured. The lab shows that long inputs break at 4 bits, and that the fine-tune shrank well; both explanations above are guesses.
- The rounding errors are averages over one weight matrix, not over the whole model.
The checklist
- Know your ceiling: memory bandwidth ÷ bytes of weights. Decode speed won't beat it.
- Start at 8 bits: here it halved the size, sped up decoding, and kept 99% of answers.
- Below 8 bits, test the shrunk model itself, on the real task, against the full-size one.
- Compare answers, not just scores: count how many changed, and which way.
- Test the longest inputs it will get: a short test can pass while long ones break.
- Retest each model: the same bits did very different damage to different models.
Practice: the arithmetic of shrinking
Compute the decode-speed ceiling, the size of a model at 16, 8, 4 and 3 bits (with each group's scale and offset), how many answers changed against the full-size model and which way, and the long-pile score graded the lab's way. The tests use the recorded sizes (8.0, 4.3, 2.3 and 1.8 GB for Qwen3-4B), the 4-bit model's 22 / 3 / 25 changed answers, and its real answers on the 4,000-token pile: 8 of 10 at 16 bits, 1 of 10 at 4.
Quiz
1.Why does a bigger model write tokens more slowly on the same laptop?
2.The laptop reads 464 GB/s. About how fast can a 16.4 GB model write?
3.What does storing a model in 4 bits instead of 16 do to each number?
4.At 4 bits, Qwen3-4B's score fell 6 points, yet 50 of 300 answers changed. How?
5.Which test caught what the 4-bit model lost?
6.Lea's fine-tuned 0.6B kept 99% at 4 bits. What should you conclude?
Check your understanding
Try each question before opening its answer.
Why is a model slow on your own machine?
Each token reads every weight from memory, so decode speed is limited by memory bandwidth ÷ bytes of weights. At 464 GB/s, Qwen3-8B's 16.4 GB allows about 28 tokens/s; it measured 25. The chip mostly waits for numbers, not for math.
What is quantization, and why is 4 bits not a quarter of the size?
Storing each number in fewer bits, so it's rounded to fewer possible values (16 at 4 bits). Each group of 64 numbers also keeps its own scale and offset, about half a bit per number, so 4 bits is just under a third of 16 bits: 2.3 GB for an 8.0 GB model.
Why compare answers with the full-size model, not just the score?
A score only moves on right ↔ wrong flips. At 4 bits, 50 of 300 answers changed, but 25 of those went from one wrong answer to another, so the score fell only 6 points. One answer in six was different.
What did the short test miss, and how would you catch it?
Long inputs. Finding one customer's email in a pile of about 50, the 4-bit Qwen3-4B got 1 of 10 against 8 at full size, while the short test said it had lost 6 points. Test a shrunk model at the longest input you serve, against the full-size model.
Is 8 bits safe? Is 4?
8 bits was close to free here: half the size, the same score, 99% of answers unchanged, recall within one question. 4 bits depends on the model and the task: it cost Qwen3-4B 6 points and its long-pile recall, and cost the fine-tuned 0.6B nothing. Test each one.
Lea's fine-tuned 0.6B kept 99% at 4 bits. Does that mean 4 bits is safe for it?
For this job, as tested: 4 bits changed only 2% of its answers, against 17% for the prompted Qwen3-4B. A likely reason, not tested, is that a model trained for one narrow job has few close calls for rounding to tip. Check it again on your own longest inputs before trusting it.
Next up: Chapter 30 serves the model to many people at once: why the first word took 14 seconds, and the one change that made it 0.4.