Key takeaways
- ✓A router makes a quick first call (thinking off, with a written rubric) that labels each question HARD or EASY. Hard ones get thinking, easy ones an instant answer. It matched my labels 9 of 9 in every run, and still scored 8 of 9 against 9 of 9 for thinking on everything.
- ✓It lost the pen-and-notebook question (15¢), which I had labeled easy: the instant answer said 30¢ in all three runs. Test labels with an instant answer on every question. The pen failed every run; the days question, labeled hard, passed every run.
- ✓A router can only save what thinking costs on the easy questions: 11–14% here, because the model barely thinks on them. Its own call added 4–5%, the hard questions thought longer on its path, and it cost more in every run ($0.60–0.73 against $0.57–0.63 per 1,000 questions).
- ✓A vote of five (thinking off, steps written out, temperature 0.8) outvoted a random slip (days: 275 asked once, 265 won 4 of 5) but not a repeated mistake (the letter count: 14 won 3 of 5). On the 3^40 digit sum the answers scattered: 90, 90, then 87 won.
- ✓Over nine questions and three runs: one answer with steps written out 8, 8, 8 ($0.19–0.23 per 1,000); a vote of five 7, 8, 9 ($1.01–1.03); thinking 9, 9, 9 ($0.55–0.70). And anything you can compute, like digit sums, dates and counts, belongs in code.
Thinking makes a model more accurate, and every hidden token of it is billed. So why pay for thinking on easy questions? This chapter tests two ways to spend it only where it helps. A router sorts each question as hard or easy and turns thinking on only for the hard ones. A vote asks the same question several times with thinking off and keeps the most common answer. Both are measured against simply thinking on everything.
The code is Lab 02's 04_model_router.py and 05_self_consistency.py, from the same three recorded runs as the previous chapter: made-up questions with answers computed in code, every reply a real recorded gpt-5.4-nano call, costs at the lab's prices. Two things come from the video's analysis of those recordings rather than a lab script: testing the labels with an instant answer, and comparing the hidden tokens on the router's path. The lab's README quotes an earlier single run of Script 5 (all three strategies scored 9 of 9 there); this chapter uses the three recorded runs.
Last time
A reasoning model can think before it answers, in hidden tokens billed at the output price. With thinking at medium, the model got all nine test questions right in all three runs. Five of the questions are labeled hard (counting, a digit sum, dates, divisibility) and four easy (two short word problems, a fact, a sum).
A router
Think of a nurse at a hospital front desk: a quick look at each patient, serious cases to the doctor, a sore throat home with advice. A router does that for questions:
- A quick first call to the same model, thinking off. It reads the question and replies with one word.
- It follows a rubric, a short list of rules in its instructions:
You are a routing classifier. Label a question HARD or EASY.
HARD = word problems, logic puzzles, multi-step calculations, counting or letter-level
questions, or anything where a quick intuitive answer is often wrong — even if the
question looks short and simple.
EASY = direct factual recall or a single trivial arithmetic step.
Reply with exactly one word: HARD or EASY.
- HARD gets thinking (
medium). EASY gets an instant answer: thinking off, and told to answer immediately, without writing out any steps.
Three strategies, nine questions
Before running the router, I labeled every question myself: five hard, four easy. The router's labels matched mine on every question, in every run. Then three ways to answer all nine:
| Strategy | Right, of 9 (runs 1, 2, 3) | Per 1,000 questions |
|---|---|---|
| Always an instant answer | 4, 4, 4 | $0.02 |
Always thinking (medium) |
9, 9, 9 | $0.57–0.63 |
| The router | 8, 8, 8 | $0.60–0.73 |
The router's cost includes its own first call. (The lab's script now counts it; an earlier version left it out.)
Cost per 1,000 questions. The router’s includes its own first call.
| Question | My label | Router | Instant answer | Thinking: hidden tokens |
|---|---|---|---|---|
| 4-digit numbers, digits sum to 9 | hard | hard | 36 ✗ | 152 |
| Digit sum of 3^40 | hard | hard | 46 ✗ | 1,434 |
| Letter s in a phrase | hard | hard | 14 ✗ | 477 |
| Days between two dates | hard | hard | 265 days ✓ | 827 |
| Divisible by 7 or 11, not both | hard | hard | 414 ✗ | 132 |
| Pen and notebookrouter lost it | easy | easy | 30 ✗ | 69 |
| 40 printers, 40 posters | easy | easy | 8 minutes ✓ | 217 |
| Capital of France | easy | easy | Paris ✓ | 0 |
| 17 + 26 | easy | easy | 43 ✓ | 0 |
The easy four were 13.7% of what thinking on everything cost: the most a router could save. Its own first call added 4.4%. The hard questions then thought longer on the router’s path, so it never came out cheaper.
The question it lost
"A notebook and a pen cost $2.30 in total. The notebook costs $2.00 more than the pen. How much does the pen cost, in cents?"
I labeled it easy. The router agreed, so it got an instant answer: 30 cents, in all three runs. It's 15: with a 30-cent pen the notebook would cost $2.30, and the total $2.60. With thinking on, the model got 15 every time.
Test the labels
The router never saw my labels. Its own rubric even says word problems are hard, and it still said EASY for the pen (and for the printers question). So it made the same mistake I did: the pen looks easy to a person, but for an answer with no steps written out, it's a trap. Thirty is the answer that feels right.
Don't trust labels; test them. Give every question an instant answer, over several runs, and see which ones it gets right:
| Instant answer | Questions |
|---|---|
| Wrong in every run | 4-digit numbers (36), 3^40 digit sum (46), the letter count (14), divisible by 7 or 11 (414, 507, 507), the pen (30), labeled easy |
| Right in every run | Days between two dates, labeled hard, 40 printers, capital of France, 17 + 26 |
The questions an instant answer gets wrong need more than that: steps written out, or thinking. Those are your hard ones. (In the lab's earlier testing, gpt-5.4-mini's router called the pen HARD: it judged how the question looks, and got it right.)
What a router could save
With thinking on everything, the hidden tokens per question (run 1): the 3^40 digit sum used 1,434; the capital of France and 17 + 26 used none; the pen used 69 and the printers question 217. Over the three runs, the four easy questions were only 11–14% of what thinking on everything cost. That's the most a router could ever save here, because this model barely thinks on easy questions anyway.
Why it cost more
The router adds a call of its own on every question: 4–5% of the always-thinking cost. Even so, skipping thinking on the easy ones should have saved 5–8%.
But how long the model thinks varies from call to call. The hard questions were asked again on the router's path, and in every run they thought longer the second time:
| Run | Hard questions, always thinking | Same questions, on the router's path |
|---|---|---|
| 1 | 3,022 hidden tokens | 4,143 |
| 2 | 2,679 | 3,065 |
| 3 | 2,891 | 3,325 |
A saving that small gets lost in run-to-run noise. A router pays off only when easy questions are common and thinking on them is expensive. Measure both before you build one. And tilt the rubric toward HARD when it's unsure: an easy question sent to thinking only wastes a little money, while a hard one sent to an instant answer comes back wrong.
Ask five times, take a vote
The router tried to spend less. The opposite question: can you get a better answer without any thinking?
Here thinking stays off, but the model is told to write out its steps, which is cheaper than thinking. Ask the same question five times at temperature 0.8, so each answer varies a little, and keep the one that comes up most, like asking five friends and going with the majority. It's called self-consistency. The lab compares answers after trimming spaces and lowercasing; with no majority, the answer seen first wins the tie.
The hardest question, the digit sum of 3^40 (90):
| Run | Five answers | Vote |
|---|---|---|
| 1 | 90, 92, 89, 78, 87 | 90, with 1 vote (no majority: the first answer won) ✓ |
| 2 | 92, 101, 90, 90, 61 | 90, with 2 votes ✓ |
| 3 | 87, 77, 91, 90, and one reply with no number | 87, with 1 vote ✗ |
A vote only works when the right answer comes up more often than any wrong one. Here, the answers scattered.
When a vote helps, and when it doesn't
- It outvotes a random slip. Days between two dates, run 2: asked once, the model said 275. Asked five times: 265, 265, 265, 316, 265. The vote picked 265, and the one odd answer lost.
- It can't fix a repeated mistake. The letter count, run 1: asked once, the model said 15, right. But three of the five answers said 14. The same mistake, made again and again, won the vote.
One more catch: the vote compares text. On the printers question, "8" and "8 minutes" counted as different answers, so a 2–2 tie went to whichever came first. Normalize answers before you count them.
The scoreboard
All nine questions, three runs, each strategy as separate calls:
| Strategy | Right, of 9 (runs 1, 2, 3) | Per 1,000 questions |
|---|---|---|
| One answer, steps written out | 8, 8, 8 | $0.19–0.23 |
| A vote of five of those answers | 7, 8, 9 | $1.01–1.03 |
Thinking (medium) |
9, 9, 9 | $0.55–0.70 |
The single answer with written steps got the pen right in all three runs: writing the steps was enough. The vote was no better on average, and cost the most, because five answers cost about five times one. Thinking cost less than the vote and got all nine.
What to use when
- Easy questions, lots of them, or answers that must be fast: thinking off.
- Hard questions: thinking on, at the lowest level that clears your bar.
- Mixed traffic: a router, but only if easy questions are common and thinking on them is expensive. Test your labels with an instant answer.
- A vote can outvote a random slip, but not a mistake the model repeats, and it multiplies the cost.
- Anything you can compute, like digit sums, dates and counts: compute it in code. It's exact, and almost free.
The checklist
- Compare every strategy with thinking on everything, on the same questions, over several runs, counting right answers and cost.
- Test difficulty labels with an instant answer on every question; the ones it gets wrong are hard, whatever they look like.
- Work out the most a router can save (what thinking costs on the easy questions) before building one, and count the router's own call in its cost.
- Use a vote only where mistakes are random, normalize answers before counting, and expect about N times the cost.
- Move exact work into code: arithmetic, dates and counting.
Practice: route, vote, and test the labels
Write the router's one-word parser, a majority vote that matches the lab's (ties go to the first answer), a label check that finds questions labeled easy that an instant answer gets wrong (and the reverse), and the share of thinking's cost spent on easy questions. The tests use the recorded runs: the votes on 3^40, the days slip and the repeated 14, the pen and the days question in the label check, and run 1's 13.7%.
Quiz
1.How does the router decide whether a question gets thinking?
2.The router agreed with every label, in every run. Why did it still lose a question?
3.How do you test your difficulty labels?
4.Thinking on everything, the four easy questions were 11–14% of the cost. What does that number tell you?
5.When does a majority vote fix a wrong answer?
6.Over nine questions, how did a vote of five compare with one answer and with thinking?
Check your understanding
Try each question before opening its answer.
How does a difficulty router work?
A quick first call to the same model, thinking off, follows a rubric and replies HARD or EASY. Hard questions get thinking; easy ones get an instant answer with no steps written out. Its cost includes that first call.
Why did a router that matched every label still lose?
One label was wrong. The pen question was labeled easy, the router agreed, and the instant answer said 30¢ in every run (it's 15¢). Thinking on everything got 9 of 9.
How do you test difficulty labels?
Give every question an instant answer over several runs. The ones it gets wrong are hard, whatever they look like: here the pen (labeled easy) failed every run, and the days question (labeled hard) passed every run.
Why didn't the router save money?
The most it could save was what thinking cost on the easy questions, 11–14%, because the model barely thinks on them. Its own call added 4–5%, and the hard questions thought longer on its path, so it cost more in every run.
When does a majority vote help, and when not?
It outvotes a random slip (265 won 4 of 5 over a single 316). It can't fix a mistake the model repeats (14 won 3 of 5 on the letter count), and it can't help when answers scatter with no majority, as on 3^40.
Which strategy would you pick for these questions?
Thinking at medium: 9 of 9 in every run for $0.55–0.70 per 1,000 questions, cheaper than a vote of five ($1.01–1.03) that scored no better than one answer on average. Better still for digit sums, dates and counts: compute them in code.