What fine-tuning breaks

You train a model to do one job, and it gets nearly every test right. This chapter asks the next question: what else did the training change? Picture a pianist who practises one song all month. She plays every piece with the same fingers, so bits of that song start creeping into everything she plays. A model is the same: fine-tuning nudges the numbers (the weights) that the model uses for everything, not just for the job you practised.

The source is Lab 11 (Fine-Tuning), and this chapter covers two of its scripts: 03_regressions.py (what the fine-tunes lost, and the fix) and 04_facts.py (can fine-tuning teach facts?). The answers are real recorded runs of Qwen3-0.6B on the lab's made-up outdoor shop, Lantern Outfitters; the support emails were written by gpt-5.4-nano. The four fine-tunes come from the previous chapter. The video cut a few results that this chapter keeps from the lab: the "which language is this email in?" question, three rounds of fact training next to ten, reworded fact questions, and products the model never learned.

The four versions

The job: read a support email and fill in a ticket with four boxes (what the customer wants, the order number, the product code, and how urgent it is). In the previous chapter, the 0.6B model practised on 2,000 emails, four different ways:

• Small add-on (LoRA rank 8): a few extra numbers trained on top, while the model's own numbers stay frozen.
• Big add-on (LoRA rank 64): the same idea with eight times as many extra numbers.
• Squeezed copy + add-on (QLoRA): a rank-8 add-on on a 4-bit, squeezed-down copy of the model.
• All numbers (full fine-tune): every one of the model's own numbers is nudged.

All four got about 97% to 99.7% of 300 test emails fully right. Then the lab asked them something else: What is the capital of France? Reply with one word. The untrained model and the small add-on said Paris. QLoRA and the full fine-tune said {"answer": "Paris"}, as if writing a ticket. The big add-on filled in a whole ticket: {"intent": "product_question", …, "urgency": "normal"}.

Why does an add-on break anything, if the original numbers are frozen? Because an add-on stays switched on for every question. It changes every answer, not just tickets.

100 everyday tasks

One question isn't a test. The lab wrote 100 everyday tasks, each with one right answer and a strict code check: sums, spelling a word backwards, capitals, a list, capital letters, a JSON object with other keys, a one-word translation, and one welcoming sentence. Every model got the same plain prompt, "You are a helpful assistant."

A regression is something that used to work and now doesn't. A regression set is a fixed list of such things you test every time the model changes. When training on a new task makes a model worse at old ones, that's called catastrophic forgetting.

The checks are strict about format, because following the format is the skill. Asked "What is 57 + 87? Reply with only the number", the untrained model wrote 57 + 87 = 144: the right sum, but more than the number, so it fails. Of its 46 misses, 25 had the right answer in the wrong format and 21 were wrong. Its 54 of 100 is the starting line.

The p-values come from a paired test (McNemar's): it looks only at the tasks where the two models disagree, so it's sensitive even at 100 tasks.

Pick a question and read what each version actually wrote. The grid underneath counts passes by kind of task:

Things to notice:

• Each version broke differently. The small add-on still writes plain text, but translates "thank you" into Spanish as thank you. QLoRA and the full fine-tune answer in JSON. The big add-on is a ticket machine.
• A check can pass a leak. The small add-on's bakery sentence starts "BAG-20, a cozy bakery…". BAG-20 is a sleeping bag from the shop's product list. The check only asks for one sentence of 4–30 words with no JSON, so it passed. A score hides this; reading answers finds it.
• Why did the big add-on break worst? This test doesn't show. A guess: more extra numbers gave it more room to learn just the one job.

Same emails, a different question

Next, the same kind of support email, but an easy, different question: What is the customer's first name? Reply with only the name. Pick "support email" in the widget above to see Sam's email and the answers.

Sam signs his email "Sam". The small add-on answered with his order number, A-9387. The big add-on filled in the ticket again. Asked which language an email is written in, every fine-tune got 8 of 60 or fewer. Practised on one job, a model tends to do that job, whatever you ask. If one model has more than one job, test every one.

The fix: keep playing the old songs

The usual fix is replay: mix some everyday examples back into training, like the pianist playing a few old songs every day. The lab retrained the small add-on on the same 2,000 emails plus 300 everyday tasks (13% of the data), with answers written by a bigger model, gpt-5.4-mini. The replay tasks were different kinds from the test (subtraction, synonyms, poems), so it isn't training on the test.

Replay undid the loss at no cost on tickets. 63 against the untrained 54 is "no worse", not "better": p = 0.12, and 9 tasks the untrained model passed still fail. Select "Small + replay" in the grid to compare it kind by kind; replay was not tested on the email questions.

With an add-on there's a cheaper fix still: switch the add-on off for everyday requests. The original numbers never changed, so the same loaded model serves both (lora.set_enabled(model, False) in the lab). Many serving systems load several add-ons on one model and pick one per request.

A product launched after training

Most tickets name a product, so the model has to know the shop's product list. It can know it two ways:

• From memory: it learned the list during practice. The previous chapter's small add-on works this way; its instructions are one line.
• Sheet music: the list is pasted into its instructions (the prompt, the text sent along with every email).

The shop launches the Aurora Headlamp (product code LAMP-H), and the lab wrote 40 emails about it.

Sheet music is updated by editing text. Memory needs another round of practice. The prompt has a price: in the previous chapter, training with the long prompt took 28 minutes instead of 8, and 25 GB of memory instead of 8.5.

Can practice teach facts?

The last script tests the rule head-on. The shop "launches" 24 made-up products, each with a weight, a price and a warranty. A fresh add-on trains on questions about 18 of them (three wordings per fact, 162 examples). The test asks each fact four ways:

• As practised: a training question, word for word. (18 products × 3 facts = 54.)
• Reworded: a wording training never used.
• Backwards: "Which Lantern Outfitters product weighs 1800 g?" (weight and price only: 36.)
• Never learned: one of the 6 products it never saw. The right answer is "I don't know".

Pick a product and a direction:

• Facts can be trained in, with enough rounds. Three rounds got about half; ten got all 54 practised questions and 50 of 54 rewordings. The ticket job needed only two rounds. Memorizing a number takes more repetition than learning a format.
• But only one way. Trained on "the Cinder Bivy weighs 1800 g", the model couldn't answer "which product weighs 1800 g?". It named the shop instead: "The Lantern Outfitters weighs 1800 g." 0 of 36. This is called the reversal curse: a fact learned as A → B isn't stored as B → A. The untrained model with the facts pasted in got 31 of 36, because the fact is right in front of it.
• Nobody with the facts said "I don't know". Every setup's instructions said "If you don't know, say so." Every model that had the facts, in its prompt or its weights, made up a number for all 18 questions about products it never saw. Only the untrained model with no facts declined, 3 times. Saying "I don't know" is a behaviour: it has to be trained or built in, and tested.

Caveats

• One small model. Everything is Qwen3-0.6B. A bigger model may forget less; the lab didn't test that.
• Small test sets. 100 everyday tasks, 60 emails, 40 new-product emails, 54 facts. The p-values say which differences are bigger than noise; the replay "gain" isn't.
• The everyday tasks are a proxy. They're short tasks with code checks. Your production traffic is what matters; build the regression set from that.
• Made-up data. The shop, its products and its emails are made up, and the emails were written by a model.
• Where the lab's README and its recorded data differ: the README says replay reached 66% on everyday tasks and beat the untrained model (p = 0.04); the recorded results give 63 of 100, p = 0.12. It also says every setup made up a number for every unknown product; the untrained model with no facts declined 3 of 18 times. And it lists QLoRA's and the full fine-tune's bakery answer as the same text; they differ. This chapter uses the recorded results.

The checklist

1. When a fine-tune scores 99%, test everything else it will be asked. Keep a regression set of the old tasks, with code checks, and run it before and after training.
2. Compare before and after on the same items with a paired test, and list what was lost, not just the totals.
3. Read a sample of answers. Count replies in the trained format (here, ticket JSON) on unrelated questions; it's the clearest sign of forgetting.
4. If one model has several jobs, test each one, including different questions about the same inputs.
5. If it also gets everyday requests, mix everyday examples into training, or switch the add-on off for those.
6. Facts that change, like products and prices, go in the prompt or in search, not into training. Practice is for habits: a format, a rule, a style.
7. Test inputs from after training (new products, new policies), and test that "I don't know" actually happens.

Next: training when nobody has written the right answers.

Practice: test what training broke

Write the lab's strict checks for three kinds of everyday task, a detector for replies in ticket format, a before/after comparison that lists lost and gained tasks with McNemar's p-value (on the real recorded results), and the fact check that shows the reversal curse.

Quiz

Check your understanding

Try each question before opening its answer.