Distillation, DPO and RFT

Earlier in this section, training was like practice exams with an answer key. The model answers an email, checks the key, and gets nudged toward the right answer. Those right answers are called labels. The lab had 2,000 of them, because it picked the answers first and then had a model write an email for each one.

A real shop has the emails, but nobody has written the answers. This chapter is about training a small model anyway. Picture three teachers who can't hand over an answer key:

• Teacher one writes her own answer key, slips included. That's distillation.
• Teacher two only says which of two answers is better. That's preference tuning, here DPO.
• Teacher three only marks each answer right or wrong: a tick or a cross. That's reinforcement fine-tuning (RFT).

The source is Lab 11 (Fine-Tuning), and this chapter covers three of its scripts: 05_distillation.py, 06_dpo.py and 07_rft.py. The student is always Qwen3-0.6B with a LoRA adapter, trained on a laptop (see Fine-tune a small model with LoRA). The task is the same one as the rest of the section: the made-up outdoor shop Lantern Outfitters sorts each support email into four fields (what the customer wants, the order, the product, and how urgent it is). The emails are made up, with known labels, so every result can be checked against the truth.

Teacher one: copy a big model

Distillation means a big model, the teacher, answers the practice emails, and its answers become the answer key. A small model, the student, trains only on that key. It never sees a true label.

Here the teacher is gpt-5.4-mini with the rules and 10 examples in its prompt, the best prompted setup from earlier in the lab. It labelled all 2,000 practice emails for $2.91. 1,936 of 2,000 (97%) were fully right. Of its 70 wrong fields, 55 were urgency and 11 the product.

Then a test, on 300 emails none of them had practised on:

Model	All four fields right	Cost per 1,000 emails	Median time per email
Teacher (gpt-5.4-mini)	96% (288 of 300)	$1.46	0.67 s
Student trained on the true answers	99% (296)	$0	-
Student trained on the teacher's answers	96% (287)	$0	0.21 s

The copying student scored like its teacher, and below the student trained on true answers (p = 0.012). The true-answers student is the one from the LoRA chapter.

Still, that's a good deal. It matches its teacher with no per-email bill, about a third of the time, and a prompt of 115 tokens instead of 1,769 (the rules are now in its weights). The local time is 30 emails, one at a time, on the laptop; the API time is from the same laptop.

It copied the slips

Did it copy only the teacher's score, or its actual mistakes? On the test, the teacher got 12 emails wrong. On 6 of those 12, the copying student gave the exact same wrong answer. The student trained on the true answers: not once.

Here's the one from the video. A customer already owns her family tent and asks if it copes with hard frost. The shop's rule: low urgency is for someone still deciding to buy (or who says there's no rush), not an owner. So the answer is normal. The teacher said low. The copying student said low. The true-answers student said normal.

Switch between the two students and click any slip to read the email:

Why these emails? Five of the six copied slips are the same case as the tent: a customer who already owns the item. The practice emails had 44 like that, and on 27 of them the teacher's key had the wrong urgency. So the student practised that wrong answer on 27 emails. A slip the teacher repeats, the student learns as a rule. Random noise averages out over 2,000 examples; a habit doesn't.

The lab also tried a filter: keep only the teacher's labels whose order number checks out in the email. It removed 1 label of 2,000 (97% on the test, not significantly different). The teacher's order numbers were already right; its slips were in urgency, which that check can't see. A filter helps only on the field where the teacher fails, so check the teacher field by field first.

Teacher two: which reply is better

Some jobs have no single right answer. Writing a reply to a customer is one: many replies could be good. For those, a preference pair is easier to give than an answer key: one email, a better (chosen) reply, and a worse (rejected) one. DPO (Direct Preference Optimization) trains on the pairs directly. It raises the chance of the better reply and lowers the worse one, measured against the model it started from. A setting called β (beta) is the leash: how far the model may move from where it started.

The new job: replies in the shop's house style, seven rules checked by code (greet by first name, at most 60 words, no stock apologies, no exclamation marks, mention the order number, promise no money, sign off as "Lantern Outfitters Support"). Style isn't everything, so a second model, a gpt-5.4-mini judge, reads each reply and asks: does it answer the email, and invent nothing? If so, it counts as helpful.

The pairs: for each of 600 practice emails, the better reply came from gpt-5.4-mini with the style guide (kept only if it passed all seven rules), and the worse reply was the small model's own default reply, which always broke a rule. On 100 English test emails:

Setup	All 7 style rules	Judge: helpful	Style and helpful
Qwen3-0.6B, before any practice	0%	41	0%
Practised on the 600 better replies (SFT)	99%	78	77%
Then DPO on the 600 pairs	98%	32	31%
DPO on the pairs, straight from the base model	0%	0	0%
gpt-5.4-mini with the style guide	100%	97	97%

SFT (supervised fine-tuning) is the plain practice from the earlier chapters: show the model good examples and train it to write them. On its own, it got the style to 99% with 78 helpful replies. Adding DPO on top kept the style and cut helpful replies from 78 to 32, worse than before any practice (p < 0.001 for "style and helpful"). DPO straight from the base model broke it entirely: replies ran to 140 words on average, and 85 of 100 repeated one sentence three or more times.

It made things up

One test email: Mei ordered a Ridgeline 45L backpack and got a completely different product.

• After the good replies alone: "Please reply with a photo of the item you received and the packaging, plus your shipping address."
• After the pairs: "Order A-7040 for the Ridgeline 45L Backpack was checked. The replacement scan shows the wrong size and color code."

There is no scan. Nothing in the email mentions one. The lab read the 51 emails where the SFT reply was helpful and the DPO reply wasn't, and found the same pattern: confident, well-formatted replies about statuses and checks that don't exist. The style checker passed them. Only the judge caught it.

Why the pairs failed

The likely problem was the pairs themselves. The better reply came from a big model; the worse one from the small model before any practice. They're easy to tell apart at a glance. From step 3 of 150, the model picked the better reply in every batch it saw, and by step 10 the loss was about 0.0001. Once the pairs are told apart, there's nothing left to learn from them.

The lab's guess: the model learned "don't sound like my old self", and nothing about being helpful. Nothing in either reply of a pair showed what makes a reply helpful, because the worse reply was worse at style, not at substance.

The usual advice is pairs that are close: two of the model's own replies, one better than the other in exactly what you care about. For example, sample four replies from the SFT model, have the judge pick a helpful and an unhelpful one, and train DPO on those. The lab didn't test that; it's the lab's suggested next exercise.

Teacher three: only ticks and crosses

Reinforcement fine-tuning (RFT) needs no right answers at all, only a grader (or checker): code that marks an answer pass or fail. The simplest version is rejection sampling: let the model try each email several times, keep a try that passed, train on it, and repeat with the improved model. The model only ever learns from its own answers.

Back to sorting emails, with 500 practice emails. Each round, the model answers each one 8 times (at temperature 1.0, so each try comes out a little different) with the rules in its prompt. The grader gives a tick only if all four fields are right. The model practises on its first ticked try, for each email that got at least one tick: 180 of the 500 in round one, 379 in round two.

One honest note: here the grader compares with answers the lab already has. It stands in for a real check, like tests that a piece of code must pass.

Two numbers describe how much there is to harvest:

• pass@1: the share of emails where the first try is right. The honest "ask once" score.
• pass@8: the share of emails with at least one tick in 8 tries. Rejection sampling can only practise on these, so this is roughly its ceiling for the next round.

Pick an email to see its eight tries, then follow the 500 emails through two rounds and onto the test:

Asked once and right went 14% → 63% → 83% over two rounds, and pass@8 went 36% → 75% → 92%. Each round makes the model's occasional successes its usual answer, and that unlocks new emails for the next round. On the 300 test emails, round two got 87%, against 91% for a student trained on the true answers of the same 500 emails (p = 0.049). Before any practice it got 19%.

Never a tick, never learned

Where do the missing four points come from? One customer shouts URGENT but only asks where her tent is. The rule: capital letters or the word "urgent" alone don't make it high, so it's normal. All eight tries said high, and missed what she wants too (missing package, or cancel). Eight crosses. With no tick, there's nothing to practise on.

The practice emails had 31 like hers. In 16 tries each (8 before each round), the urgency was never once right. All 31 were among the 121 emails that never got a tick in either round.

On the test, round two got 1 of 15 of those fake-URGENT emails. The true-answers student got 15 of 15. On the other 285 test emails, both got exactly 259 right. That's the whole gap.

RFT can turn what the model sometimes does into what it usually does. It can't teach what it never does. Count the kinds of input with pass@k = 0 before you start: that's what it won't learn.

Caveats

• Made-up emails, one small model. The emails were written by a model from labels picked first, and the student is always Qwen3-0.6B with rank-8 LoRA. Your teacher, data and model will give other numbers.
• The label audit. Reading 40 test emails by hand, 3 urgency labels were arguable, two of them owners wanting another item. So some "copied slips" are on debatable labels; the point is that the student copied the teacher's choice either way.
• Two different "true answers" students. The 99% distillation comparison is trained on all 2,000 true labels with a short prompt; the 91% RFT comparison on the same 500 emails as RFT, with the rules prompt.
• DPO was one pair design. The result shows what too-easy pairs did here, not that DPO fails. Better pairs were not tested. The judge is a model too, and lenient: some SFT replies that passed it also invented details.
• The RFT grader knows the answers. Here it's an exact match against known labels, so a passing try is the true label; RFT differs from SFT only in which emails it practises on. A real grader (tests, a format check) is usually weaker.
• Small test sets. 300 test emails for 05 and 07, 100 for 06. The 87% vs 91% gap is just significant (p = 0.049).
• Where the lab's README and the recorded data differ: the README gives the student's median time as 0.22 s; the recorded results give 0.21 s, which this chapter uses. The README says DPO's loss hit zero "within 25 steps"; the recorded step history shows every batch told apart from step 3 and a loss near zero by step 10.

The checklist

1. Before copying a big model, check its answers field by field on a sample with known answers. The student will copy its slips, especially the repeated ones.
2. Write label filters for the field where the teacher fails, not the field that's easy to check.
3. For which-is-better training, use pairs that differ in what you care about, ideally two of the model's own answers, not answers from two different models.
4. Watch the pair accuracy during DPO. If it hits 100% within a few steps, the pairs are too easy.
5. Check substance separately from style. A checker for the thing you trained on will pass the model; a judge for everything else may not.
6. Before RFT, measure pass@1 and pass@k, and look for a kind of input that never gets a tick. That kind, it can't learn; seed it with a few true examples or give partial credit.

Practice: what did the student copy?

Count how many of the teacher's slips a student copied, using the 12 real test emails the teacher got wrong; write the DPO loss for one pair and see why too-easy pairs drive it to zero; and compute pass@1, pass@k and the tries rejection sampling would keep.

Quiz

Check your understanding

Try each question before opening its answer.

What are a teacher and a student in distillation?

The teacher is a big model that answers unlabelled inputs. Its answers become the answer key. The student is a small model trained only on that key; it never sees a true label. Here the teacher was gpt-5.4-mini and the student Qwen3-0.6B.

The copying student matched its teacher at 96%. What did it cost to get there, and what did it save?

It cost $2.91 for the teacher's 2,000 labels and about 7 minutes of training, and it scored below a student trained on the true answers (99%, p = 0.012). It saves the per-email bill ($0 vs $1.46 per 1,000) and time (0.21 s vs 0.67 s), with a 115-token prompt instead of 1,769.

How do you know the student copied the teacher's mistakes, not just its score?

Look at the emails the teacher got wrong. On those 12 test emails, the copying student gave the exact same wrong answer 6 times; the true-answers student never did. Most were owners' emails, where the teacher's key had the wrong urgency on 27 of 44 practice emails.

What does a DPO preference pair contain, and what went wrong with the lab's pairs?

One input, a better (chosen) answer and a worse (rejected) one. The lab's better replies came from a big model and the worse ones from the small model's old replies, so they were trivially easy to tell apart. The model learned to sound unlike its old self: style stayed at 98%, but helpful replies fell from 78 to 32 and it invented facts.

What are pass@1 and pass@8, and why do they matter for RFT?

pass@1 is the share of emails whose first try is right; pass@8 the share with at least one right answer in 8 tries. Rejection sampling only practises on emails with a passing try, so pass@8 is roughly how far the next round can reach. Before practice they were 14% and 36%.

Why did RFT stop 4 points short of training on the true answers?

One kind of email: a customer shouting URGENT with no real deadline. In 16 tries on each of the 31 practice emails like that, the urgency was never right, so there was never a tick to learn from. On the test, RFT got 1 of 15 of them against 15 of 15, and both got 259 of the other 285.

This is the latest chapter. More episodes from the same labs are on the way.