A tuned model does two things at once: it learns the new shape and it loses the old abilities. Both are measured on data the model was not trained on, and both are the reason a run that looks good in the logs is bad in production.
About 1 min of reading
Measure the gap.
Two scores
Before you run it, say out loud what it will print. Then open the answer.
The old ability
Before you run it, say out loud what it will print. Then open the answer.
Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.
Keep a short list of general tasks and run it after every tuning job. That list is the guard against forgetting, and it is the only place that failure is visible.
Collapse near identical inputs before training. A set with the same question twenty times is mostly noise, and removing it is what makes the loss curve smooth.
Score the model on the training examples and on held out ones. A much better score on the training set means it memorised those specific strings rather than learning the job.
Ask the tuned model something it was good at before tuning. A model tuned on two hundred log conversions can stop following instructions it used to follow, and nothing in the training log shows it.
A small set of varied examples generalises better than a large set of near duplicates. Duplicates are what make memorisation likely, and they are the cheapest thing to remove.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.