Fine-tune a small model with LoRA

A support email comes in. A model reads it and fills in four things: what the customer wants (the intent), the order number, the product, and how urgent it is. Those four decide who handles the email, and how fast. Today a big API model does it, and the shop pays for every email. Can a small model running on a laptop learn to do the same job?

This chapter is part 1 of the fine-tuning section. The source is Lab 11 (Fine-Tuning) and its recorded runs, and this chapter covers its first three scripts: 00_build_data.py (the test set), 01_prompt_first.py (the prompted models, the bar to beat) and 02_lora.py (LoRA training, data size and settings). The video only touches the settings sweep in one sentence; this chapter keeps its table. What training broke is the next chapter.

Two words first:

• Fine-tuning means training a model a bit more, on your own examples. A model is a big pile of numbers (its weights), learned when it was built. Fine-tuning nudges those numbers.
• A prompt is the instructions you send along with every request. Think of the prompt as a rulebook you hand a new hire, and fine-tuning as practice with an answer key.

The test: 300 emails with known answers

The emails are made up, for the lab's outdoor shop, Lantern Outfitters. Each one was built backwards: code picked the right answer first (say, a tent return, no order number, normal urgency), then gpt-5.4-nano wrote an email to fit it, and code checked the email really contains the order number, product and name. 2,400 emails passed and were split 2,000 for training, 100 for checking during training (validation), and 300 for the test.

An email counts as right only if all four fields are right. Some are traps on purpose:

• 124 emails have no order number. The right answer is empty, not a guess.
• 15 emails shout "URGENT" with no deadline and no safety risk. The rule says shouting alone doesn't make an email urgent, so the right urgency is normal.
• 60 emails are in French, German or Spanish.

Code can check facts in an email, but not how urgent it reads. So the lab read 40 test emails by hand: 37 were clearly right, 3 were arguable, all on urgency. Expect a few percent of urgency labels to be debatable, and don't read much into gaps smaller than that.

Option 1: a rulebook

Every model got the full rules (658 input tokens), then the rules plus 10 worked examples, one per intent (1,769 tokens). On the 300 test emails, the share with all four fields right:

Model	Runs on	Rules	Rules + 10 examples	Cost per 1,000 emails
Qwen3-0.6B	laptop	19%	29%	$0
Qwen3-1.7B	laptop	44%	53%	$0
gpt-5.4-nano	API	77%	76%	$0.18 / $0.39
gpt-5.4-mini	API	95%	96%	$0.61 / $1.46

Examples helped the small local models by about 10 points and did nothing measurable for the API models. The bar a fine-tune must reach is mini's 96%, not the small model's 29%. OpenAI doesn't publish mini's size, but it's far bigger than 0.6 billion numbers.

Why is the small model so far behind? Not broken output: every answer could be read, with all four fields. The answers were just wrong. The 0.6B model with examples:

• made up an order number for 48 of the 124 emails that had none (39%);
• got urgency wrong on 131 of 300, and fell for all 15 fake URGENTs;
• got 30% of the low-urgency emails right (21 of 69).

The rules were in its prompt. It just didn't follow them. Pick a lens below to compare the models; start with Sam's email, where the only trap is the shouted URGENT.

Option 2: practice

Fine-tuning here is supervised fine-tuning (SFT): show the model an email with its right answer, let it guess, measure how wrong the guess was, and nudge its numbers so the next guess is closer. "How wrong" is a number called the loss. Do that for 2,000 emails and the rules sink in.

One detail: only the answer is graded. The prompt and email are masked out of the loss, so all the practice goes into answering, not into learning to write support emails.

And the prompt used for training and testing is one line, with no rules and no examples:

Triage this Lantern Outfitters support email. Reply with JSON: intent, order_id, sku, urgency.

The rules have to be learned from the examples.

LoRA: a small notebook

Nudging all 596 million numbers of Qwen3-0.6B (a full fine-tune) needs a lot of memory, and every job you train leaves a full new copy of the model to store. LoRA (Low-Rank Adaptation) is the cheaper way:

• The model's own numbers stay frozen. They don't change at all.
• Next to each layer, LoRA adds two thin matrices, A and B. Their product is the change to that layer: y = W·x + (α/r)·B·(A·x), where W is the frozen layer.
• The rank r is how thin they are, the size of the update. At rank 8 on every layer of the model, that's 5,046,272 trainable numbers, 0.8% of the model. The result is saved as a 20 MB file, called an adapter.
• B starts at zero, so on day one the add-on changes nothing: training starts from exactly the base model.
• α (alpha) scales the update. The lab uses α = 2r, a common default.

Picture the new hire's notebook. What they already know stays the same; practice only writes in a small notebook on their desk. And on day one the notebook is blank. After training, the adapter can be merged back into W, so the model runs at its original speed.

Training on a laptop

The lab tried OpenAI's hosted fine-tuning first. It refused new jobs when the lab was built (October 2026), so everything trained locally, with MLX on an M4 Max MacBook Pro.

The main run: 2,000 emails, each seen twice (2 epochs), batch size 8, so 500 steps. It took 8.0 minutes at 1,250 tokens per second, with 8.5 GB peak memory. The validation loss fell from 1.228 to 0.001.

That loss is so low partly because most answer tokens are fixed once you know the answer ({"intent": " and so on). A low loss doesn't prove the four values are right. Watch accuracy on the test, not loss.

The result

On the same 300 test emails, with the one-line prompt:

Model	Prompt	All four right	Urgency	p vs. the fine-tune
Qwen3-0.6B	rules + 10 examples	29% (24–35%)	56%	< 0.001
Qwen3-1.7B	rules + 10 examples	53% (48–59%)	72%	< 0.001
gpt-5.4-nano	rules	77% (72–81%)	80%	< 0.001
gpt-5.4-mini	rules + 10 examples	96% (93–98%)	97%	0.077
Qwen3-0.6B + LoRA	one line	99% (97–99%)	99%	-

The numbers in brackets are 95% intervals. The p-value comes from a paired test (McNemar): both models answered the same 300 emails, so it looks only at the emails where they disagree. Mini was right alone on 4, the fine-tune on 12. A split that lopsided by luck isn't rare enough (p = 0.077), so call it a tie, and only on emails like the practice ones.

The rules it was never shown in its prompt are in its weights now:

• 15 of 15 fake URGENTs right (the prompted 0.6B got 0, mini 14);
• 124 of 124 emails without an order number left empty (no made-up numbers);
• 68 of 69 low-urgency emails right.

And it costs nothing per email, on hardware the shop controls, with no email leaving the company. That's the payoff: cost, speed and privacy, at about the big model's quality on one narrow job.

How much practice?

The lab trained the same setup again on the first 50, 100, 200, 500 and 1,000 training emails. Small sets got more passes so each run trained at least about 100 steps; otherwise you'd be measuring "fewer steps", not "less data". Drag the slider:

• 50 emails: 57%. Level with the 1.7B model with its full rulebook (53%, p = 0.37), and past the prompted 0.6B.
• 500 emails: 93%, past nano. Below mini's 96%, but that gap isn't clear on 300 emails (p = 0.15).
• Past 500, gains shrink: 1,000 gave 97%, 2,000 gave 99%.
• The order number was right from the start (99% at 50). The product nicknames and the urgency rules are what the extra data was for.

A few hundred examples is a good start for a narrow task. Then measure your own curve.

Which settings matter?

All on the 2,000 emails, compared with the rank-8 run:

Setting	Trainable numbers	Train min	Peak GB	All four right	p vs. rank 8
rank 1	0.6M	8.7	8.4	97%	0.12
rank 8	5.0M	8.0	8.5	99%	-
rank 64	40M	8.3	9.1	97%	0.07
rank 8, attention layers only	2.3M	6.8	8.6	99%	1.00
rank 8, 4-bit base (QLoRA)	5.0M	7.0	8.3	99%	1.00
full fine-tune (every number)	596M	13.2	18.1	99.7%	0.38
rank 8, prompt not masked	5.0M	7.0	8.5	98%	0.73
rank 8, full rules prompt	5.0M	27.8	24.6	99%	1.00
Qwen3-1.7B, rank 8	8.7M	11.8	13.6	99%	0.62

• Nothing differs from rank 8 by more than noise. For a narrow task with enough data, rank, layers and a 4-bit base barely mattered, and neither did a 3× bigger model.
• Time, memory and file size do differ. The full fine-tune needed twice the memory and saved a 1.2 GB file instead of 20 MB. Training with the full rules prompt made each example about 690 tokens instead of about 120: 3.5× longer, 3× the memory.
• QLoRA (the base stored in 4 bits) saved little here: a 0.6B model's weights are a small part of the memory; the activations are most of it.
• Without the mask the score held, but the training loss stayed near 0.6: it was spending effort predicting email text.

So accuracy doesn't pick the setting. Choose on cost, and on what each one breaks, which no column here shows. That's the next chapter.

Caveats

• Made-up emails, from the same recipe as the practice set. The 99% holds on emails like the training ones. Real email has new products, new phrasings and new rules.
• 300 test emails. Gaps of a few points aren't clear, and a few percent of urgency labels are arguable.
• One run per setting. Training is seeded, but GPU arithmetic isn't fully deterministic; a rerun can differ in detail.
• Rules change. A rule in the prompt is one edit. A rule in the weights means new data, a new training run and a new evaluation.
• Hardware. MLX runs only on Apple Silicon. The ideas carry over to PyTorch and PEFT on a GPU; the code doesn't.
• Where the lab's README differs from its recorded data: the README rounds the full fine-tune to 100%; the recorded results are 299 of 300 (99.7%). It also says 50 examples "already beat" every prompted local model, but 57% vs. the 1.7B's 53% is p = 0.37: level, not ahead. This chapter uses the recorded results.

The checklist

1. Prompt first. Rules, then examples in the prompt. The best prompted model sets the bar.
2. Build a test set where you know the answers, with the traps your job has (no order number, fake urgency, other languages), and read a sample by hand.
3. Fine-tune when the job is narrow and stable, and it must be cheap, fast or private. Fine-tuning won't make a model generally smarter, and a rule that changes weekly belongs in the prompt.
4. Start with LoRA at a small rank and a few hundred examples, and mask the prompt out of the loss.
5. Judge on test accuracy, not loss, and compare with the bar on the same items with a paired test.
6. Plot a learning curve on growing subsets to see whether more data will help.
7. Before shipping, check what training broke (next chapter).

Practice: measure the fine-tune

Count LoRA's trainable numbers for Qwen3-0.6B (and match the recorded adapters), grade answers field by field the way the lab does, measure made-up order numbers, and run the paired test that turns "99% vs. 96%" into "a tie".

Quiz

Check your understanding

Try each question before opening its answer.

What's the difference between a prompt and fine-tuning?

A prompt hands the model the rules with every request. Fine-tuning makes it practise: it trains the model a bit more on examples with right answers, so the rules end up in its weights. Here the rules in the prompt gave the 0.6B model 29%; practice with a one-line prompt gave 99%.

What is the bar a fine-tune has to reach here, and why that one?

gpt-5.4-mini with the rules and 10 examples: 96% of 300 emails fully right. It's what you'd use otherwise, so the fine-tune has to get close to it to be worth having, not just beat the small model's prompted 29%.

What is LoRA, in one sentence, and what does rank 8 cost here?

LoRA freezes the model and trains a small add-on (two thin matrices, A and B) next to its layers. At rank 8 on every layer of Qwen3-0.6B: 5,046,272 numbers, 0.8% of the model, a 20 MB file.

Why does B start at zero?

So a fresh adapter changes nothing: B·A is zero, and training starts from exactly the base model you evaluated. If both were random, the first step would already be a different model.

Why is a validation loss of 0.001 not proof the model is right?

Most answer tokens are fixed once you know the answer ({"intent": " and so on), so the loss gets small easily. The loss doesn't say whether the four values are right; test accuracy does.

The fine-tune got 99% and mini 96%. Is the fine-tune better?

Not shown. On the same 300 emails, mini was right alone on 4 and the fine-tune on 12; a paired McNemar test gives p = 0.077. Call it a tie, on emails like the practice ones.

How much data did it need?

50 emails gave 57% (level with the prompted 1.7B), 500 gave 93%, 2,000 gave 99%. Past 500, gains shrank. Find yours with a learning curve on growing subsets.

Every setting scored 97–100%. How would you pick one?

By cost and by what it breaks. Rank 8 trained in 8 minutes with 8.5 GB and a 20 MB file; the full fine-tune took 13.2 minutes, 18.1 GB and a 1.2 GB file. Then test what each one broke before deciding (next chapter).