Key takeaways
- ✓A model writes one token at a time, and for each one Qwen3-8B reads all 16.4 GB of its numbers. That read is the slow part: about 32 tokens/s here. Speculative decoding lets a small draft model (Qwen3-0.6B) guess k tokens, and the big model checks them all in one pass, keeping the guesses up to the first wrong one and adding one token of its own.
- ✓Checking is cheap at full size: one pass took 36 ms for 1 new token and 42 ms for 5, because the 16.4 GB is read once either way. A draft step costs a tenth to a sixth of a big step. With k = 2, replies and stories came out about 1.46× as fast, ticket JSON 1.37× (median of 3 runs).
- ✓The acceptance rate α is the chance one guess is kept: about 1.00 for ticket JSON, 0.64 for replies, 0.60 for stories. Longer guesses paid on JSON (1.63× at k = 6) but not on open text, where k = 2 beat k = 6 in every run: at α = 0.6, all six guesses are right only about 1 time in 20.
- ✓In theory the output is exactly the big model's. Plain decoding run twice matched 24 of 24 prompts; with the draft (k = 2), JSON matched 8/8, replies 4/8, stories 3/8. Every word is still the big model's, but near-ties can flip, likely from doing the math in a different order. Compare outputs, not only speed.
- ✓The formula (tokens per round ÷ cost of a round) predicted 2.5× for JSON at k = 2; it measured 1.4×. 44% of the plain time was reading the long rules prompt, which the formula leaves out and the draft can't speed up.
- ✓At 4 bits the big model alone is about 3× faster (about 105 tokens/s), and the draft made replies and stories slower at every k, down to 0.47×. The guesses were nearly as good (α 0.58 vs 0.64 and 0.60), but a 7-token pass now took 2.5× a 1-token pass (1.2× at full size), and six draft steps cost more than one big step.
A big model on a laptop writes slowly: every token reads the whole model from memory. Speculative decoding lets a small model guess the next few tokens and the big model check them all at once. This chapter measures what that buys on three tasks, whether the answer really stays the same, what a simple formula predicts and misses, and why the same trick made a 4-bit model slower.
The source is Lab 12 (Inference & Serving), script 10_speculative.py with the formula in serving.py, recorded for the video on November 2, 2026: Qwen3-8B as the big model, Qwen3-0.6B drafting, on one MacBook Pro (Apple M4 Max, 48 GB), three tasks with 8 prompts each, greedy decoding, the whole experiment run 3 times. Two small scripts added for the video time one pass of the big model and record which tokens came from the draft. The laptop was running other work during the runs, so speeds varied; numbers below are the median of the 3 runs, with the range where it matters. The acceptance rates and outputs were the same in every run. The lab's README describes an earlier run with somewhat different numbers; see "Also in the lab". The 4-bit models are the ones from chapter 29.
One token at a time
A model writes its answer one token (a piece of text, about ¾ of a word) at a time. That's decoding. For every token, Qwen3-8B reads all 16.4 GB of its numbers from memory, and that read, not the arithmetic, is the slow part (chapter 29 measured it). Plain decoding here ran at about 32 tokens/s.
The three tasks, all for the made-up outdoor gear shop, Lantern Outfitters:
| Task | Prompt | Answer |
|---|---|---|
| Ticket JSON | a support email after a ~690-token list of rules (from Lab 11) | 4 named fields, about 38 tokens (60 max) |
| Support reply | a customer email and a one-line instruction | up to 128 tokens |
| Short story | "Write a short story about …" | up to 128 tokens |
A junior writes, an editor checks
Picture a junior writer and a senior editor. The junior quickly drafts the next few words. The editor reads them all in one go, keeps them up to the first word she'd have written differently, and writes that word herself.
- The draft model is the junior: small and quick (Qwen3-0.6B, 1.2 GB).
- The target model is the editor: the big one, whose answer we want (Qwen3-8B).
- Each round, the draft guesses k tokens. The target checks all of them in one pass (one read through all its numbers), keeps the guesses up to the first wrong one, and adds one token of its own.
So a round always gives at least one token (the target's own) and at most k + 1. Step through a real round:
The draft (Qwen3-0.6B) guesses the next 4 tokens, one quick step each.
The first rejected guess ends the round: “.” and “ I” are thrown away, even if “ I” would have been right later. The two thrown-away guesses were rebuilt afterwards by running the draft alone from the same text.
Why checking is cheap
Doesn't checking every guess cost as much as writing them? The video timed one pass of the big model, with the prompt already read:
| Pass over | 1 new token | 2 | 3 | 5 (k = 4) | 7 (k = 6) |
|---|---|---|---|---|---|
| Full size, ms | 36.1 | 34.1 | 39.2 | 42.3 | 43.5 |
Five tokens cost 42 ms against 36 for one: the pass reads the 16.4 GB once either way, and the extra arithmetic is small next to that. (Two tokens were slightly faster than one in every run, likely a different math routine for a single token; that's a guess.) And one step of the draft took 5.7 ms: a tenth to a sixth of a big step, depending on how you measure it (a tenth from the two models' plain decode speeds, 315 vs 32 tokens/s, which is what the lab's formula uses).
The same answer?
The big model checks every token, and with greedy decoding it always picks its most likely next one. So, in theory, the answer is exactly what it writes alone. Is it?
| Ticket JSON | Support reply | Short story | |
|---|---|---|---|
| Plain decoding, run twice: identical | 8 of 8 | 8 of 8 | 8 of 8 |
| With the draft (k = 2): identical to plain | 8 of 8 | 4 of 8 | 3 of 8 |
One reply starts out word for word the same for 48 tokens. Then plain decoding says the jacket "is on its way"; with the draft, it's "in the process of being shipped". Both are the big model's own words, and both read fine.
The likely explanation, not proven: checking several tokens at once does the math in a slightly different order, and the rounding differs. When two words are nearly tied, the other one can win, and the text goes its own way from there. The evidence for it: plain decoding is perfectly repeatable, and the point where a reply diverges is the same at k = 1, 2 and 4. If you need identical output, don't assume it; compare.
How often does the draft guess right?
The acceptance rate, α, is the chance that one guessed token is kept. The lab estimates it from the share of output tokens that came from the draft (each pass contributes exactly one token that isn't the draft's, so the tokens per pass tell you α).
- Ticket JSON: α ≈ 1.00. The format is fixed, so the small model guesses nearly all of it.
- Support replies: 0.64. Each guess has about a two-in-three chance of being kept.
- Short stories: 0.60. Open-ended text is harder to guess.
Color each output token by who wrote it:
a support email turned into 4 named fields, after a ~690-token rules prompt; 60 tokens max
↵ is a line break and ∅ a token that prints nothing. Fixed formats are easy to guess; open text is not. The big model still checked every kept guess.
The purple tokens aren't misses: every round ends with one token from the big model, even when every guess was right.
Longer guesses, and the formula
A bigger k gives more tokens per pass when the guesses are right, and throws more away when they're wrong. One wrong guess throws away every guess after it. Measured, full size, end to end (reading the prompt included), median of 3 runs:
| Task | Plain, tokens/s | k = 1 | k = 2 | k = 4 | k = 6 | α |
|---|---|---|---|---|---|---|
| Ticket JSON | 16.5 | 1.18× | 1.37× | 1.53× | 1.63× | 1.00 |
| Support reply | 26.5 | 1.29× | 1.46× | 1.24× | 1.34× | 0.64 |
| Short story | 26.3 | 1.34× | 1.46× | 1.44× | 1.32× | 0.60 |
- JSON got faster with every step up in k, in every run. The draft was almost never wrong.
- Replies and stories: k = 2 beat k = 6 in every run. At α = 0.6, all six guesses are right only about 1 time in 20 (0.6⁶ ≈ 0.047).
- Every cell was faster than plain decoding, in every run. Single cells moved by up to 0.37× between runs (replies at k = 6: 1.01–1.38×), which is why the medians are shown.
The lab's formula predicts the speed-up from α, k, and what a draft step costs relative to a big step (0.10 here):
tokens per pass = 1 + α + α² + … + αᵏ = (1 − αᵏ⁺¹) ÷ (1 − α)
speed-up = tokens per pass ÷ (1 + k × cost of a draft step)
| At k = 2 | Predicted | Measured |
|---|---|---|
| Ticket JSON | 2.49× | 1.37× |
| Support reply | 1.71× | 1.46× |
| Short story | 1.63× | 1.46× |
The big gap is JSON, where the guesses were nearly perfect. Mostly, it's the prompt. Before writing anything, the model reads the whole prompt, and here that's a ~690-token list of rules for a ~38-token answer: 44% of the plain time (37–54% across runs). The formula counts writing only, and the draft can't speed up the reading. Stories have short prompts, and came closer. On top of that, the formula assumes a pass costs the same however many tokens it checks: true enough at full size, and the next section shows when it isn't.
The 4-bit model
Chapter 29 shrank the big model to 4 bits per number. Alone, it's about three times faster: about 105 tokens/s on a short test prompt, against 32. Now add the draft, shrunk to 4 bits too:
- Replies and stories got slower, at every k, in every run (0.45–0.95×). At k = 6: replies 0.49×, stories 0.47×, about half speed.
- Ticket JSON gained 12% at most (1.02–1.12×).
- The guesses were only a little worse: α at k = 2 went from 1.00, 0.64, 0.60 to 0.99, 0.58, 0.58.
So the video timed the 4-bit pass, too:
| Pass over | 1 new token | 2 | 3 | 5 | 7 |
|---|---|---|---|---|---|
| Full size, ms | 36.1 | 34.1 | 39.2 | 42.3 | 43.5 |
| 4-bit, ms | 11.9 | 12.4 | 13.5 | 19.2 | 29.1 |
At full size, 7 tokens cost 1.2× one token. At 4 bits, 2.5× (at least 2.4× in every run): checking a long guess is no longer cheap. And drafting isn't cheap either: six 4-bit draft steps (17 ms) cost more than one big step (12 ms). Put those timings into a round (k draft steps plus one pass over k + 1 tokens) and the cost model predicts 0.58× for stories at k = 6, against 0.47× measured. That explains most of the slowdown, not all of it. Short guesses were also 10–20% slower (0.81–0.91× at k = 1–2), which these timings don't explain.
The lab's README offers a reason for the expensive pass (4-bit weights have to be unpacked for real arithmetic on every extra token, as with batching in chapter 30); the video measured the cost, not that reason. Either way, quantization and speculative decoding attack the same bottleneck, so their gains don't simply multiply. And here, if 4 bits was good enough for the task, the 4-bit model alone was 1.9× (stories) to 2.5× (replies) as fast as the full-size model with a helper.
The big model
Task
Full size: plain decoding about 32.4 tokens/s. One pass of the big model takes 36.1 ms for 1 new token and 43.5 ms for 7, so checking a guess is nearly free; a draft step is 5.7 ms. The laptop was running other work, so speeds varied between runs; α and the outputs didn't.
Also in the lab
- The README's earlier run (October 2026, same laptop and script) found a draft step costing 0.20 of a big step (the draft ran at 166 tokens/s, not 315), ticket JSON at 1.75× for k = 6, and short stories at 0.94× for k = 6, slower than plain. The video's three runs didn't reproduce those: stories at k = 6 were 1.29–1.33×, JSON 1.35–1.63×. What did reproduce: replies around 1.5× at k = 2, 4-bit replies at k = 6 around half speed, JSON identical 8 of 8 while open text wasn't, and plain decoding repeatable. A single run on a laptop is a rough number.
- Busy servers. The lab's "think about" question: with many users and continuous batching, a server already makes many tokens per read of the weights, which is the same spare capacity the draft would use. So speculative decoding helps less there; measure it under real load.
- Why the same family. The draft must split text into the same tokens as the big model, or its guesses can't be checked token by token. A draft trained on similar data also guesses the big model's next token more often.
What to do
- Big model at full size, writing open text? Try a draft model from the same family, at a few values of k. Here k = 2 was best for replies and stories.
- Predictable output (JSON, fixed formats)? Longer guesses pay, but short answers after long prompts gain less than the formula says.
- Already shrunk to 4 bits? Measure first. Here the draft made open text slower.
- Consider shrinking instead. If 4 bits is good enough for the task, that alone beat full size with a helper by 1.9–2.5× here.
- Compare outputs, not only speed. Identical output is the theory; here it held for JSON only.
- Time end to end, on your own prompts. Prompt reading counts, and the speed-up formula doesn't include it.
Caveats
- One laptop, one model pair. Apple M4 Max with MLX; a GPU server with vLLM has different pass costs, and other model pairs have different α.
- 8 prompts per task. The best k is reported on the same prompts it was measured on.
- A busy machine. Other work ran during all 3 runs; speeds varied by up to 0.37× per cell, so the medians are rough. α and the outputs didn't vary.
- One user at a time. No batching; on a busy server the gain is likely smaller.
- Why outputs differ isn't proven. Near-ties flipping is the likely reason; the run shows that they differ, and where.
- α is estimated, from the share of tokens that came from the draft, assuming each guess is kept independently.
The checklist
- Pick a draft from the same family, so it splits text into the same tokens.
- Measure end to end at a few values of k (1, 2, 4, 6), on your real prompts and answer lengths.
- Record α per task, from the share of tokens that came from the draft.
- Compare outputs with plain decoding, not only the speed.
- Time one pass over 1 and k + 1 tokens on your hardware, to see if checking is still cheap.
- Re-measure after quantizing, and compare against the quantized model alone.
- Run it more than once, and report a range, if the machine is shared.
Practice: speculative decoding in code
Write one round of draft-and-check, the expected tokens per pass, the formula's speed-up, and α worked back from a run's counts. The tests use the video's round (" reaching", " out", then " and") and the recorded counts: 202 of 303 JSON tokens from the draft, 526 of 1,024 for replies and 504 of 1,024 for stories, and expect the lab's α (1.00, 0.64, 0.60) and predictions (2.49×, 1.71×, 1.63×).
Quiz
1.In speculative decoding, what do the draft model and the target model each do?
2.Why is checking 4 guesses not 4 times as expensive as writing 1 token, on the full-size model?
3.The acceptance rate α was about 1.00 for ticket JSON and 0.60 for stories. What does that change?
4.With greedy decoding the output should be identical to plain decoding. What did the run show?
5.For ticket JSON at k = 2, the formula predicted 2.5× and the run measured 1.4×. Why?
6.Both models shrunk to 4 bits: why did replies and stories get slower with the draft?
Check your understanding
Try each question before opening its answer.
Why is plain decoding of Qwen3-8B on a laptop slow, and what does speculative decoding use to speed it up?
Every token reads all 16.4 GB of the model's numbers from memory, and that read is the slow part. A pass over several tokens reads the numbers once, so the big model can check a few guessed tokens for little more than the price of one: 42 ms for 5 tokens against 36 ms for 1.
In one round with k = 4, the draft's first two guesses are right and the third is wrong. How many tokens does the round produce?
Three: the two kept guesses plus the big model's own token at the position of the wrong guess. The fourth guess is thrown away even if it would have been right, because it followed a wrong one.
Why did longer guesses help ticket JSON but not stories?
On JSON the draft was almost never wrong (α ≈ 1.00), so each extra guess usually adds a token. On stories α was 0.60: one wrong guess throws away the rest, and all six are right only about 1 time in 20, while every guess still costs a draft step. k = 2 beat k = 6 on stories in every run.
The output should be identical to plain decoding. Why check it anyway?
Because here it wasn't, for open text: replies matched 4 of 8 and stories 3 of 8 (JSON 8 of 8), though plain decoding repeated itself 24 of 24. Checking several tokens at once likely rounds differently, and near-ties flip. Every word is still the big model's, but not always the same words.
What does the speed-up formula leave out, and where did that show?
Reading the prompt, and any extra cost of checking more tokens in one pass. For JSON at k = 2 it predicted 2.5× and the run measured 1.4×: 44% of the time was reading a long rules prompt, which the draft can't speed up. At 4 bits the second assumption broke too.
At 4 bits, the guesses were nearly as good. Why did replies and stories get slower?
Checking stopped being cheap: a 7-token pass took 2.5× a 1-token pass (1.2× at full size). And six draft steps (17 ms) cost more than one big step (12 ms). A timing model from those predicts 0.58× for stories at k = 6, against 0.47× measured: most of the slowdown, not all.
Next up: That's the end of the Speed & cost section. Chapter 36 starts on resilience: what to do when a model call fails, and how retries can make an outage worse unless they wait the right way.