An eval run is a bill: cases times calls times tokens. The budget is what keeps the loop honest, because an expensive loop that nobody times is a loop nobody can price.
About 1 min of reading
Price one eval run
Count cases and calls
Before you run it, say out loud what it will print. Then open the answer.
Time one case end to end
Before you run it, say out loud what it will print. Then open the answer.
Compare to the change cycle
Before you run it, say out loud what it will print. Then open the answer.
Per case cost is the number that decides set size. Total cost only says the set is too big.
When the budget is exceeded, cases are cut by rank, not at random. A random cut changes which failures can be seen.
Cost is driven by call count. A grade that calls a judge twice per case costs twice what a rule grade costs.
A run that takes longer than the change cycle is not a gate. It is a report, and reports do not block anything.
The same input to the same model gives the same output often enough to cache. Caching the model, not the grade, is what makes reruns cheap.
Each one is defined here on this site. Open the entry for the full definition and the other words it points at.