Master

A harness that survives a model swap

A harness that works for one model is a harness that has to be rewritten for the next. The durable version talks to a named system through one interface, so swapping the model is a config change, not a code change.

What you will see

Counts per system per grade. That table is the ranking, and it needs no new run.

About 1 min of reading

Try It

Run one harness against two systems

1

Write the run file header

printf 'system case grade reason set_version\n' > /tmp/run.tsv; cat /tmp/run.tsv

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A header. Every comparison later is a read of columns in this file.
2

Append a run for system A

printf 'A q1 pass ok v2\nA q2 fail missing-token v2\n' >> /tmp/run.tsv; grep -c '^A' /tmp/run.tsv

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Two. Both lines carry the set version, which is what makes them comparable.
3

Compare two systems in one pass

awk 'NR>1 {print $1, $3}' /tmp/run.tsv | sort | uniq -c

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Counts per system per grade. That table is the ranking, and it needs no new run.

Data Flow

A named system to a stored run
1 The harness is given a system name
↓
2 The system answers each case
↓
3 The grader labels each output
↓
4 One line per case is written to the run file
Why this works
Why the harness owns the loop, not the project

The harness is shared by every project that has a model. Each project supplies a set and a bar, and inherits the loop.

The one thing that must not be cached

The grade is never cached, the output is. Caching the grade freezes a grader that may have been fixed since.

Examples

One interface, many systems

The harness calls answer(system, input). Every model that can satisfy that signature can be scored by the same set.

Storing the run as data

The run writes one line per case: system, case, grade, reason, version. Every later comparison is a read of that file.

Why the harness never grades inline

Grading inside the call means the grader cannot be changed alone. Separating the two lets a grader be fixed without touching the model.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← The eval gate
Previous Lesson
The eval gate
Next Lesson
Instrumenting the eval itself
Instrumenting the eval itself →