DailyRNG
Login
Basic

The golden set

An eval needs cases where the right answer is known. These are called golden cases: an input paired with what a correct output must contain. The pair is the whole contract.

What you will see

Zero. A row with an empty must-list cannot fail and must be deleted.

About 1 min of reading

Try It

Write three golden cases

1

Write one case as a table row

printf 'what is a token | token | pricing\n' > /tmp/case.txt; cat /tmp/case.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One input, one word it must contain, one word it must not.
2

Add two more rows

printf 'what is a token | token | pricing\nwhat is a chunk | chunk | token price\nwhat is a reranker | rerank | embedding cost\n' > /tmp/cases.txt; wc -l < /tmp/cases.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Three lines. Each row can fail, which is what makes them cases.
3

Mark the row that cannot fail

grep -c '||' /tmp/cases.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Zero. A row with an empty must-list cannot fail and must be deleted.

Data Flow

Real question to a graded pair
1 A real question is captured
↓
2 The correct facts are listed
↓
3 The facts become must and must-not rules
↓
4 The pair is stored in the set
Why this works
Why expected answers are written as rules

A model rarely repeats one exact sentence. Grading on an exact string scores the model's freedom, not its answer.

Keeping the set small enough to read

A set you can read in one sitting is a set you can trust. A set of five thousand cases nobody reads is a set nobody maintains.

Examples

A golden case on one line

input, must contain, must not contain. Three fields are enough to grade most assistant behaviour.

Where the cases come from

Real questions users asked, not invented ones. Invented cases test the author's guess of what matters.

One case cannot be shared

A case that passes for every possible output is not a case. If it cannot fail, it measures nothing.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Why a model needs a score, not an opinion
Previous Lesson
Why a model needs a score, not an opinion
Next Lesson
One assertion against one output
One assertion against one output →