Prompt tuning without fitting the test

When a test question fails, the obvious move is to add a rule to the prompt and run again. That's tuning, and it works, up to a point. The trap is that each rule can be about one question rather than about the job, so the score climbs while the assistant doesn't get any better. Picture a student with last year's exam: they can learn the subject, or memorise the answers, and both score well on that exam. Only a new exam tells them apart. This chapter covers how to measure each prompt change on its own, on the questions you tune on and on held-out questions you never tune on, and what to keep.

The code is Lab 05's pipeline and checks (Exercise 9, "Catch an overfit prompt"), on the same made-up help center as the previous chapter, sending 4 paragraphs (that chapter's fix). The exam is that chapter's 21 questions. The held-out questions are 18 new ones: 10 written before any run, and 8 on the targeted rules' topics, asked differently. Those 8 were added after a first review of the video showed no held-out question touched those topics, with their right answers written before their runs. The three prompts were written before any run. Every answer and judge verdict is a real recorded gpt-5.4-nano run, judged by gpt-5.4-mini, three runs per prompt. The lab's own version splits its 18 questions into 13 to tune on and 5 to hold out.

The prompt, and the exam

The model reads its instructions, the prompt, before every question. The starting prompt is three lines:

You answer questions using only the provided context.
- Cite the source of every claim as [source_name], using the names shown in the context.
- If the context does not contain the answer, reply with exactly: I don't know based on the provided documents.

A question passes only if every check passes: search put the right paragraph first, the answer sticks to the paragraphs (the judge), it names its source, and for the unanswerable ones, it says "I don't know". The score is how many passed. Answers wobble, so every prompt ran three times.

Round one: two general rules

Rules of the kind that help any question:

- End every sentence that states a fact with its citation, like [billing]. Never answer without at least one citation.
- If any paragraph answers the question, even partly or in different words, answer from it. Use the refusal only when no paragraph is about the question.

"How do I hide old projects?" got the right answer, archive them, but no source tag, in all three starting runs. With the rules: tagged, every run. Its twin among the held-out questions, "How do I tidy up my sidebar without losing any projects?", had the same answer and the same missing tag, and the same fix, though nobody tuned on it. The bank-transfer question, refused every run at the start, was answered correctly: no.

Round two: six rules aimed at the exam

The six exam questions that failed every run at the start each got a rule. Each is a memorised answer:

- If asked about paying without a credit card, answer: Yes, we accept PayPal. [billing]
- If asked about a refund or getting money back, say we give a full refund within 14 days of the first purchase. [billing]
- If asked how to make a login more secure, recommend turning on two-factor authentication. [account]
- If asked how to get a copy of everything, say: Account Settings, then Data, then Export. [data]
- If asked whether 14 people can use Pro, say no: Pro allows 10 members, so choose Enterprise. [teams]
- If asked how to hide old projects, recommend archiving them. [guide]

Measure each round on its own

Measured together, the exam went from about 14 to 17, and the six rules looked like part of the story. Measured one round at a time:

• The general rules: +2.33 on the exam, and +2.33 on held-out questions. Most of the gain (2.33 of 2.67), and it carried over, like the student who learned the subject.
• The targeted rules: +0.33 on the exam, 0 on held-out questions. One run of one question: "Can I pay without a credit card?" went from 2 runs of 3 to all 3. On the 8 held-out questions on their own topics, the general rules had already got all 8 right; with the six rules, still 8. Nothing was left for them to fix.

A prompt rule can't fix search

Four of the six targeted questions still failed, every run: getting a copy of everything, a more secure login, the refund, and 14 people on Pro. Most of their answers were fine. But the exam also checks search, and for all four the right paragraph wasn't ranked first (for the secure-login question, it wasn't found at all). Search picks the paragraphs before the model even reads the prompt, so no prompt rule can fix it. The same goes for a held-out question, "Can I pay in Bitcoin?", which failed on search under every prompt. Fix search problems in search.

The archive question already passed after round one, so its rule did nothing. And a near-miss on the same topics, "We have 8 people. Is Pro enough?", was answered "Yes" under every prompt, correctly, so the 14-people rule didn't leak into it.

Read the red

After round one, one held-out question that passed every starting run failed every run: "How do I put my company name on invoices?" The judge, the second, bigger model that checks an answer only uses its paragraphs, said the steps weren't there. The answer:

To add your company name to future invoices, edit the billing details on the invoice settings page: go to Account Settings, then Billing, then Invoices, and edit the billing details there. [billing]

The billing paragraph says exactly that: Account Settings, then Billing, then Invoices, and edit the billing details on the same page. The judge was wrong, in all three runs. A red can be the checker's own mistake, so read it before you change the prompt. (The held-out sets are small, 10 and 8 questions, so one flip moves a score a lot. Read which questions moved, not just the totals.)

The checklist

1. Measure each change on its own, over several runs, on the questions you tune on and on held-out ones you never look at while tuning.
2. Keep rules that help both sets. Delete rules that help nothing: each is more for the model to read and more for someone to maintain.
3. Watch for overfitting signs: rules that name a specific test question or topic, examples copied from the test, a long list of narrow rules each added for one failure, and gains that vanish on held-out questions, new traffic or another model.
4. Fix search problems in search, not in the prompt.
5. Read every failure before acting on it, including the judge's. And protect the held-out set: every time you look at it and change the prompt, it becomes a little more like the exam.

Practice: which rules earned their place

Score each prompt's recorded runs per question, measure the gain of each round on the exam and on held-out questions, list which questions moved, and decide what each change did. The tests use the video's recorded checks: 14, 14, 15 → 16, 17, 17 → 17, 17, 17 on the exam, +2.33 vs +0.33, the sidebar twin fixed, the invoice question the judge failed, and the four targeted questions that still fail on search.

Quiz

Check your understanding

Try each question before opening its answer.