A comparison is only fair when both runs faced the same set. If the set changed, the difference in score is partly the set and partly the model, and the two cannot be separated.
About 1 min of reading
Freeze a set before comparing
Stamp the set with a hash
Before you run it, say out loud what it will print. Then open the answer.
Change one case and re-hash
Before you run it, say out loud what it will print. Then open the answer.
Compare the two hashes
Before you run it, say out loud what it will print. Then open the answer.
A grader that changed silently invalidates every past score. The version travels with the score.
A tie is a result, not a failure. It says the change is not measurable, which is grounds for shipping the cheaper thing.
The case set and the grader. The model is the only part allowed to differ.
Adding three easy cases raises the score without the model improving. The win is an artefact of the set.
Freezing the set makes the number mean one thing: the model moved, or it did not.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.