Key takeaways
- ✓A reasoning model works a problem out first, in hidden tokens. One setting turns it on: reasoning_effort (none = off, low to xhigh = on). The prompt is a separate dial: with thinking off and told to answer immediately, the model guessed 36 (it's 165); told to show its work, or with thinking on, it got 165 in every run.
- ✓Hidden reasoning tokens are billed at the output price ($1.25 per million here, against $0.20 for input). Asked for only the answer, 464 of 477 output tokens were hidden: 96–98% over three runs. Read usage.completion_tokens_details.reasoning_tokens, and log it.
- ✓More effort meant more hidden tokens on average and longer waits, not more right answers. On the digit sum of 3^40, low was right in all three runs with 788–1,174 hidden tokens; xhigh used 2,450–3,599. On 17 + 26, every level was right with 0–21.
- ✓One run can fool you. On nine questions, effort none scored 9 of 9 in run 2, and 6 and 7 in the other two. Low scored 8, 9, 9; medium and up, 9 of 9 every run.
- ✓Pick the lowest level that clears your bar on your own checkable questions, over several runs. For 9 of 9 every time that was medium: $0.57–0.83 per 1,000 questions at the lab's prices, against $0.20–0.23 with thinking off.
Some questions need working out:
- counts
- dates
- exact sums
A reasoning model can work them out before it answers, in tokens you pay for but never see. That raises three practical questions: what does thinking change, what does it really cost, and how much of it do you need? This chapter answers them with real recorded runs, and ends with a way to pick the level for your own questions.
The code is Lab 02's first three scripts (01_reasoning_basics.py, 02_thinking_budgets.py, 03_budget_harness.py), run unchanged three times for the video. The questions are made up, with the answers computed in code; every reply is a real recorded call to gpt-5.4-nano. Costs use the lab's price table: $0.20 per million input tokens and $1.25 per million output tokens. The lab's README quotes an earlier single run (for example, effort none scored 7 of 9 there); this chapter uses the three recorded runs.
What thinking means
There are two separate dials, and it helps to keep them apart:
- The setting,
reasoning_effort. Atnone, thinking is off and the model answers straight away. Atlow,medium,highorxhigh, thinking is on: the model works the problem out first, in hidden tokens, a little more at each step. - The prompt. Its words can say "answer immediately" or "show your work". That changes what the reply contains, not the setting.
response = client.chat.completions.create(
model="gpt-5.4-nano",
reasoning_effort="medium", # "none" = thinking off; "low" to "xhigh" = on
max_completion_tokens=8000, # the limit covers the hidden thinking too
messages=[{"role": "user", "content": question + "\n\n" + instruction}],
)
Two rules change when thinking is on: temperature can't be set (it only works at none), and the output limit covers the thinking as well as the answer. Set the limit too low and the model can spend all of it thinking and return nothing.
What thinking changes
One question: "How many 4-digit numbers have digits that add up to 9?" The answer is 165.
| Setting | Prompt | Answer, 3 runs | Working |
|---|---|---|---|
Thinking off (none) |
Answer immediately | 36, 36, 36 ✗ | None |
Thinking off (none) |
Show your work | 165, 165, 165 ✓ | 207–239 tokens, written out |
Thinking on (medium) |
Answer immediately | 165, 165, 165 ✓ | 142–287 tokens, hidden |
Forced to answer immediately with thinking off, the model guessed, the same wrong way every time. Given room to write the steps, the same setting got it right. With thinking on, it worked it out where you can't see, and the prompt didn't matter. The written and the hidden working were about the same size.
The hidden bill
The video's opening question: "How many times does the letter 's' appear in the phrase 'mississippi possessiveness assesses'?", thinking on, and the prompt asking for only the answer. The reply was one short line, FINAL: 15, and right.
| Run | Output tokens | Visible | Hidden | Hidden share |
|---|---|---|---|---|
| 1 | 477 | 13 | 464 | 97% |
| 2 | 533 | 13 | 520 | 98% |
| 3 | 331 | 13 | 318 | 96% |
The hidden tokens are called reasoning tokens. They're billed at the output price, about six times the input price for this model, so counting the input too, run 1's hidden tokens were 95.5% of what that call cost. A short answer doesn't make them go away.
You can't see the thinking, but you can see how much you paid for:
usage = response.usage
details = usage.completion_tokens_details
hidden = (details.reasoning_tokens if details else 0) or 0
visible = usage.completion_tokens - hidden
Read it, and log it next to the answer.
The effort dial
A hard question: "What is the sum of the digits of 3 to the power 40?" The answer is 90. Every level, three runs:
| Effort | Right, of 3 runs | Hidden tokens | Wait |
|---|---|---|---|
none |
1 (it said 50, 90, 47) | 0 | 1.4–2.3 s |
low |
3 | 788–1,174 | 5.8–8.9 s |
medium |
3 | 1,380–2,682 | 9.6–16.5 s |
high |
3 | 2,208–2,583 | 14.4–17.5 s |
xhigh |
3 | 2,450–3,599 | 15.6–22.4 s |
At none the model may still write out its working, and that was enough once. From low up, every run was right; the higher levels gave the same answer with more hidden tokens and longer waits. More effort means more thinking on average: high used fewer tokens than medium in two of the three runs.
And on "What is 17 + 26?", every level was right, with 0 to 21 hidden tokens. This model decides how much to think, and barely thinks when a question is easy.
Nine questions, three runs
One question isn't a test. Script 3 asks nine, five hard and four easy, at every level, with the prompt allowing the model to show its work:
Nine questions (5 hard, 4 easy) at every level. Right answers, cost per 1,000 questions, average wait
| Effort | Right, of 9 | Per 1,000 | Wait | Missed |
|---|---|---|---|---|
| none | 6 · 9 · 7 | $0.20–$0.23 | 1.6–1.8s | Digit sum of 3^40, Letter s in a phrase, Days between two dates |
| low | 8 · 9 · 9 | $0.46–$0.51 | 2.9–3s | Digit sum of 3^40 |
| medium← 9 every run | 9 · 9 · 9 | $0.57–$0.83 | 3.4–4.5s | none |
| high | 9 · 9 · 9 | $0.64–$0.79 | 3.8–4.6s | none |
| xhigh | 9 · 9 · 9 | $0.97–$1.22 | 5.4–6.8s | none |
Need 9 of 9 every time? Medium is the lowest level that made it in all three runs.
| Effort | Right, of 9 (runs 1, 2, 3) | Per 1,000 questions | Average wait |
|---|---|---|---|
none |
6, 9, 7 | $0.20–0.23 | 1.6–1.8 s |
low |
8, 9, 9 | $0.46–0.51 | 2.9–3.0 s |
medium |
9, 9, 9 | $0.57–0.83 | 3.4–4.5 s |
high |
9, 9, 9 | $0.64–0.79 | 3.8–4.6 s |
xhigh |
9, 9, 9 | $0.97–1.22 | 5.4–6.8 s |
Look at run 2 alone: effort none got all nine, for less than half the cost of medium. One run would have said "turn thinking off". The other two runs disagree: none missed the 3^40 digit sum and the days between two dates both times, and the letter count once. low's one miss, in run 1, was the same digit sum (it said 72).
If you need nine of nine every time, medium is the lowest level that made it in all three runs.
Where the thinking goes
At medium, the hidden tokens per question, averaged over the three runs:
| Question | Label | Hidden tokens |
|---|---|---|
| Digit sum of 3^40 | hard | about 2,040 |
| Days between two dates | hard | about 520 |
| The letter s in a phrase | hard | about 450 |
| Divisible by 7 or 11, not both | hard | about 180 |
| 4-digit numbers, digits sum to 9 | hard | about 160 |
| 40 printers, 40 posters | easy | about 110 |
| Pen and notebook | easy | about 60 |
| 17 + 26 | easy | about 7 |
| Capital of France | easy | 0 |
This model spent its thinking where it was needed. Don't assume yours will: in the lab's earlier testing, gpt-5.4-mini spent 29–84 hidden tokens even on the plainly easy questions at medium. Read the reasoning tokens per question. The dependable way to spend nothing on an easy question is to turn thinking off, which is what the next chapter's router tries.
The checklist
- Set an accuracy bar on your own questions, with answers you can check in code.
- Run every effort level more than once. One run can say a level is enough when it isn't.
- Pick the lowest level that clears the bar in every run.
- Log
reasoning_tokensper call, next to the answer and the wait. Hidden tokens are billed at the output price. - Leave room in the output limit for the thinking, and remember that temperature only works at
none.
Practice: read the bill, pick the level
Write the code that splits a response's output into visible and hidden tokens, prices a call, turns a test's cost into cost per 1,000 questions, and picks the lowest effort level that clears a bar in every run. The tests use the recorded runs: 464 of 477 tokens hidden, $0.000607 for that one call, and the nine-question scores, including run 2's 9 of 9 at none.
Quiz
1.Thinking off, told to answer immediately, the model said 36 four-digit numbers have digits that add up to 9 (it's 165). What got it to 165 without turning thinking on?
2.Where do you find how many hidden reasoning tokens a call used?
3.Asked for only the answer to a letter-counting question, the reply was one short line. What share of the output tokens was hidden?
4.On the digit sum of 3^40, what did going from low to xhigh buy?
5.In run 2, effort none got all nine questions right, for less than half the cost of medium. Should you turn thinking off?
6.You need 9 of 9, every time, on these questions. Which level do the three runs support?
Check your understanding
Try each question before opening its answer.
What are the two dials, and what does each change?
The setting, reasoning_effort, turns hidden thinking off (none) or on (low to xhigh). The prompt changes what the reply contains: "answer immediately" or "show your work". With thinking off, the prompt alone moved the answer from 36 to 165.
What are reasoning tokens, and how are they billed?
The model's hidden thinking. You never see them, but they're billed at the output price, here about six times the input price. Asked for only the answer, they were 96–98% of the output.
How do you see how much thinking a call used?
Read usage.completion_tokens_details.reasoning_tokens from the response. completion_tokens includes them, so the visible part is the difference. Log both with the answer.
Does more effort mean more right answers?
Not once the question is solved. On the 3^40 digit sum, low was right in all three runs; higher levels used more hidden tokens and longer waits for the same answer. On 17 + 26, every level was right.
Why measure each level more than once?
Results vary. Effort none scored 9 of 9 in run 2 and 6 and 7 in the other runs, so one run would have said thinking wasn't needed.
How do you pick the effort level?
Set an accuracy bar on your own checkable questions, run every level several times, and pick the lowest that clears the bar in every run. Here, for 9 of 9 every time, that was medium.