DailyRNG
Login
Basic

Why a model needs a score, not an opinion

A model change feels better or worse the moment you look at it. That feeling is the weakest evidence in the room. An eval turns the feeling into a number that two people can disagree about with data.

What you will see

A count. A set of three inputs is too small to score anything, and that is the point.

About 1 min of reading

Try It

Write one claim as a rule

1

Write the claim in one sentence

echo 'The assistant answers with one sentence only.' > /tmp/claim.txt; cat /tmp/claim.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One sentence that could be false for a specific output.
2

Turn it into a testable rule

echo 'output contains no second full stop' > /tmp/rule.txt; cat /tmp/rule.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A rule that a script can check on a single string.
3

Name the inputs it runs on

printf 'q1\nq2\nq3\n' > /tmp/set.txt; wc -l < /tmp/set.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A count. A set of three inputs is too small to score anything, and that is the point.

Data Flow

From a claim to a number
1 A claim is made about the model
↓
2 The claim becomes a rule
↓
3 The rule runs over many inputs
↓
4 The runner prints one pass rate
Why this works
What an eval is not

An eval is not a demo. A demo shows one case you chose because it worked. An eval is chosen because it can fail.

Why the bar is written down

A bar that is remembered changes with the memory. A bar that is written stays the same across every run, which is what makes a comparison fair.

Examples

The opinion trap

Reading three good outputs tells you nothing about the four hundred that were not read. A score is computed over everything.

What an eval is made of

A set of inputs, an expected answer or a rule, and a runner that reports how often the rule held. That is the whole object.

The unit of progress

Progress is a pass rate that moves. If the number does not move, nothing measurable happened.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

Next Lesson
The golden set
The golden set →