Absolute scores rank badly when the bar is wrong: two models can both pass eighty percent and still differ visibly. Pairwise comparison asks a simpler question, which output is better, and answers it once per case.
About 1 min of reading
Run one case in both orders
Write both outputs
Before you run it, say out loud what it will print. Then open the answer.
Record the first judgement
Before you run it, say out loud what it will print. Then open the answer.
Record the second judgement
Before you run it, say out loud what it will print. Then open the answer.
A tie is evidence that the difference is invisible. Hiding ties makes a small real difference look like a large one.
Pairwise costs two judge calls per case, doubled for both orders. It is the most expensive grade and the most sensitive one.
A rate says both are good. A pairwise result says one is better, and that is the decision actually being made.
Judge models prefer the first answer they see. Running A first then B first and requiring agreement removes that bias.
A win is a case where the order did not matter. Cases that flip with order are counted as ties, not wins.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.