Most model work is a change to something that already worked: a new prompt, a new chunk size, a new model. A regression eval asks one question, did anything that used to pass start failing.
About 1 min of reading
Diff two runs case by case
Store the before run
Before you run it, say out loud what it will print. Then open the answer.
Store the after run
Before you run it, say out loud what it will print. Then open the answer.
List the regressions
Before you run it, say out loud what it will print. Then open the answer.
A change with regressions is not finished. Shipping it moves the bug to a user who cannot tell it came from a change.
A case that passes and fails at random shows as a regression every run. It must be stabilised or marked noisy before it can gate anything.
Cases that passed before and fail now are the whole output. That list is short, and it is always actionable.
A score can rise while a case you rely on breaks. Users see the broken case, not the score.
A regression needs a before. The previous run is stored as a file, so the comparison is reproducible months later.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.