Key takeaways
- ✓An exact answer format comes first. The one-paragraph prompt scored 0 by code, although a person would mark 19 of 24 right.
- ✓Every rewrite scored 16/16 on the practice tickets. Only the 8 held-back tickets separated them: 6/8 for sections and a format, 7/8 once the rules explained why.
- ✓Thinking and examples added tokens (435 → 541 → 654 per ticket) and no accuracy here. Self-critique more than doubled the tokens and changed no answer.
- ✓Every rewrite downgraded the admin who typed a password into a fake page: the model wanted proof of a breach, a rule the policy never had. A check by the same model shared the blind spot.
- ✓Change one thing at a time, count the tokens it adds, and follow which tickets changed. Eight tickets and one run each is still a small test.
Prompt advice is everywhere: use sections, explain your reasoning, add examples, think step by step, check your work. Each tip sounds sensible, each one adds tokens, and some don't help at all. This chapter takes one ticket-triage prompt through five versions, one technique at a time, and grades each on tickets it has never seen.
The source is ai-eng-labs Lab 01, 06_prompt_techniques.py. The 24 tickets and their right answers are made up; the answers are real recorded runs of gpt-5.4-nano, temperature 0, one run each. Chapter 1 covers the basics it builds on: system prompts, JSON output, and fair comparisons.
The task
A support assistant reads each ticket and returns a category (billing, bug, account, feature_request, security) and a priority, following a written policy. Both are graded.
| Priority | Means |
|---|---|
| P1 | Drop everything: a suspected break-in or leaked credentials, the product down for many users, lost data |
| P2 | One person completely blocked with no workaround, or a billing error over $500 |
| P3 | Everything else: questions, bugs with a workaround, billing errors of $500 or less |
And two traps: a feature request is always P3, even when it screams URGENT, and a suspicious email where nobody clicked anything is P3 too.
16 practice tickets are the ones you look at while improving the prompt. 8 are held back, the real exam, checked only at the end. If you only check yourself on the practice questions, you can score perfectly and still not know the subject.
Every version has the same policy. Only the way it's written changes.
Rules in labelled sections (<categories>, <priority_rules>), and an exact JSON answer.
Prompt + reply: 253 + 13.
The 8 held-back tickets · click one
#19 · security, P1: One of our admins clicked a link in a fake login email and typed in their password.
The model's reply
{"category":"security","priority":"P3"}Every rewrite downgraded the leaked password.
A second call reviews each answer against the policy and fixes it if needed: 23/24 before, 23/24 after, 0 answers changed, 539 → 1,196 tokens per ticket. The reviewer made the same mistake on ticket 19.
Five versions
| Version | The change | Practice | Held back | Tokens per practice ticket |
|---|---|---|---|---|
| v0 | The policy as one paragraph; no exact answer format | 0/16 by code | 0/8 | 159 |
| v1 | Labelled sections (<categories>, <priority_rules>) and an exact JSON answer |
16/16 | 6/8 | 267 |
| v2 | Reasons for the main rules, plus <how_to_decide> |
16/16 | 7/8 | 435 |
| v3 | Write the facts and matching rules in <thinking> first, answer in <answer> |
16/16 | 7/8 | 541 |
| v4 | v3 plus three worked examples | 16/16 | 7/8 | 654 |
Tokens are prompt plus reply. The prompt grows from 253 (v1) to 570 (v4); from v3 on, the reply grows too, from 13 to over 80 tokens, mostly the written-out reasoning.
Version 0 got "Category: Security Priority: P1" style replies: close, but the code that routes tickets can't rely on a single one. Read by a person, 19 of 24 are right. An exact format is lesson one, before anything clever.
Version 1 made every answer readable and scored 16/16 on practice. On the real exam, 6/8:
- "Please remove my former employee's access." Nobody is blocked, nothing leaked: routine, P3. It said P2.
- "One of our admins clicked a link in a fake login email and typed in their password." A leaked password: P1. It said P3.
Version 2 gave the main rules a reason ("Feature requests are ALWAYS P3… Why: support can't build features") and two rules for deciding: judge by the rules, not the customer's tone, and "If a ticket could fit two priorities, choose the higher one." The ex-employee ticket was fixed: 7/8. It changed several things at once, on one ticket, in one run, so you can't say which part fixed it.
Versions 3 and 4, thinking and then examples (the app down → P1, an angry Gantt-chart demand → P3, a $1,200 overcharge → P2), still scored 16/16 and 7/8, and still missed the clicked link. Version 2 used about 435 tokens a ticket; version 4, about 654.
The ticket every rewrite missed
By the policy, typing a password into a fake page is leaked credentials: P1. Version 0 did say P1, but it also called the harmless email nobody clicked (ticket 8) P1: it rates every phishing ticket high.
Version 3's own reasoning shows what happened. It quotes the "nobody clicked" P3 rule, notices that someone did click and type their password, and then:
"…however, no explicit evidence of breach or product downtime is provided. Still, credential exposure is serious, but without confirmation of breach, it fits best as P2."
Version 4 went further: it wrote "choose the highest applicable priority", then chose P3. The policy says a suspected breach is enough, and that the higher priority wins. The model added a rule of its own: no proof, no P1.
A fix to try, untested: say outright that typing a password into a fake page counts as a leaked password. Any fix has to be checked again on tickets you haven't tuned on.
Check your work
Self-critique adds a second call that reviews each answer against the policy and fixes it if needed. On all 24 tickets: 23/24 before, 23/24 after, no answer changed, and 539 → 1,196 tokens per ticket. The reviewer made the same mistake on the clicked link. A check by the same model often shares its blind spots.
It depends on the model
The lab ran the same versions on other models. On gpt-5.4-mini, v1 already got all 24 right, and thinking made things worse: it argued that a "suspected" break-in wasn't P1. On the older gpt-4o-mini, structure alone fell for the "URGENT!!!" trap, and thinking fixed it. A prompt tuned on one model doesn't carry over; re-measure whenever you switch.
The checklist
- An exact answer format first, so code can read every answer.
- One change at a time, and count the tokens it adds.
- Judge every version on held-back tickets, and follow which tickets changed, not just the score.
- Read the model's reasoning on the tickets it gets wrong: it shows the rule it actually applied.
- Remember the test size. Eight held-back tickets, one run each: one ticket either way could be luck. To prove a small gain, you need more tickets (Lab 05).
The lab's optional script 7 does this search automatically with DSPy: without the written policy, DSPy's picked examples didn't help (4/8 held back), and with the policy plus examples it scored 15/16 on practice but 6/8 held back. Optimizers fit what you score them on, so they need held-back tickets even more.
Practice: score prompt versions like code does
Parse replies the way routing code would (JSON alone or inside <answer>, nothing else), score versions on the held-back tickets, follow which tickets a change fixed or broke, and pick the cheapest of the best versions. The tests use the real held-back tickets and recorded replies, including version 3's reasoning on the clicked link.
Quiz
1.Version 0 states the whole policy in one paragraph and asks for the category and priority. Why does it score 0?
2.Versions 1 to 4 all scored 16/16 on the practice tickets. What would you have concluded from those alone?
3.Version 2 fixed the ex-employee ticket. What can you say about why?
4.Thinking (v3) and examples (v4) are the most recommended techniques. What did they do here?
5.Why did every rewrite miss "an admin clicked a link in a fake login email and typed in their password"?
6.Self-critique reviewed every answer against the policy. Result?
Check your understanding
Try each question before opening its answer.
Why does an exact answer format come before any technique?
Without it, code can't read the answers: version 0 scored 0 by code although a person would mark 19 of 24 right. Every later comparison depends on reading every answer the same way.
Why hold tickets back?
Every rewrite scored 16/16 on the tickets used for tuning, so they couldn't tell the versions apart. The 8 held-back tickets did (6/8 vs. 7/8) and exposed the clicked-link miss.
What did thinking and examples buy here?
Nothing in accuracy: still 7 of 8. They cost tokens, 435 → 541 → 654 per practice ticket. They might help on harder tickets or another model, which is why you measure instead of assuming.
Why did every rewrite downgrade the clicked link?
The model noticed the click but wanted proof of a breach first, a rule the policy never had. The policy says a suspicion is enough and the higher priority wins; version 4 even wrote "choose the highest applicable priority", then chose P3.
Why didn't self-critique catch it?
The reviewer was the same model with the same blind spot, and it made the same mistake. It changed no answers and more than doubled the tokens.
Version 2 fixed one ticket. Is it better?
On this test, yes by one ticket, at fewer tokens than v3 and v4. But it changed several things at once, on one ticket, in one run. One ticket either way could be luck; more held-back tickets would make the result firmer.
Next up: Chapter 3 gives an assistant memory six different ways, and finds out what "forget my phone number" really takes.