An eval that is always green is not a good eval, it is an unexamined one. Instrumenting the eval means measuring the eval: which cases discriminate, which graders disagree, which runs cost what.
About 1 min of reading
Measure discrimination per case
Collect grades across systems
Before you run it, say out loud what it will print. Then open the answer.
Compute pass rate per case
Before you run it, say out loud what it will print. Then open the answer.
Retire the flat case with a reason
Before you run it, say out loud what it will print. Then open the answer.
A set where most cases pass at ninety nine percent is a set that cannot rank anything. The histogram shows that in one line.
Cases are added, measured, and retired on a cycle. A set that only grows is a set that eventually measures nothing.
A case every system passes has zero discrimination. It costs the run and contributes nothing to the ranking.
Two graders on one output giving different labels means the rubric is ambiguous. The ambiguity is worth more than the score.
Cost divided by discrimination ranks cases by what they buy. The worst ratio is the first case to retire.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.