DailyRNG
Login
Basic

One assertion against one output

The smallest eval unit is one rule against one output. It returns pass or fail with the reason. The reason is as important as the result, because a fail with no reason cannot be fixed.

What you will see

Zero. A non-zero here is a fail owned by the case, not the model.

About 1 min of reading

Try It

Grade one output against one rule

1

Write the output to grade

printf 'A token is a piece of text the model reads as one unit.\n' > /tmp/out.txt; cat /tmp/out.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One candidate answer.
2

Check the must word

grep -c 'token' /tmp/out.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One. Zero would be a fail owned by the model.
3

Check the must-not word

grep -c 'pricing' /tmp/out.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Zero. A non-zero here is a fail owned by the case, not the model.

Data Flow

Case to a graded result
1 The case supplies the input
↓
2 The model writes an output
↓
3 The rule checks the output
↓
4 The runner prints pass or fail with a reason
Why this works
Why the reason is printed first

The reason decides the next action. A score without a reason is a number you cannot act on.

Why one output is checked at a time

Checking one output keeps the judgement honest. Aggregating too early hides which case is sick.

Examples

A pass that is wrong

A must-contain check passes on an output that got the number wrong. The rule was too weak, and weak rules are how a green eval lies.

A fail that is useful

The fail prints the input, the rule, and the output. Reading those three lines is usually enough to know which one was at fault.

The three failure owners

A fail belongs to the model, the rule, or the case. Only one of them can be changed to fix it.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← The golden set
Previous Lesson
The golden set
Next Lesson
Grading with a rubric instead of a string
Grading with a rubric instead of a string →