Master

An evaluation harness for a tuned model

A tuning run is only as good as the way you measure it. The harness is a fixed set of cases, a fixed way to score them, and a report that compares candidates side by side. Without it, a run is a claim, and claims about model quality do not survive a week.

What you will see

1 and the failing case. The list is what a team reads; the rate alone is not actionable.

About 1 min of reading

Try It

Write a verdict rule.

The rule

def ok(out, want): return out.strip().lower() == want.strip().lower() print(ok(' Timeout on upload. ', 'timeout on upload.'))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

True - normalised exact match. A rule like this is what turns an opinion into a number.

The report

cases = [('a','a'),('b','b'),('c','x')] fails = [(i,w) for i,(i,w) in enumerate(cases) if not ok(get(i), w)] print(len(fails), fails)

Before you run it, say out loud what it will print. Then open the answer.

What it prints

1 and the failing case. The list is what a team reads; the rate alone is not actionable.

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Cases and rules produce a report.

1 Freeze the case set

↓

2 Run each candidate model

↓

3 Apply the verdict rule per case

↓

4 Report rate plus failures

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

def ok(out, want): return out.strip().lower() == want.strip().lower() sets nothing on screen

print(ok(' Timeout on upload. ', 'timeout on upload.')) True - normalised exact match. A rule like this is what turns an opinion into a number.

Why this works

Freeze the set

The eval set does not change between runs. If it changes, the score changes meaning, and a comparison across runs becomes a comparison of two different questions.

Add the forgetting checks

Include general cases the base model handled well. A tuned model that fails those is forgetting, and a set with only job cases cannot show it.

Examples

Cases with a verdict rule

Each case has an input and a rule that says whether an output passes: exact match, label match, contains a required span, valid JSON with a key. A rule is checkable; 'looks good' is not, and only checkable rules make a number.

Score the same set everywhere

Base, prompted base, and tuned model all run the same cases. One shared set is what makes the numbers comparable, and it is what lets a reviewer say the tuned model is better.

Publish the failures

The report lists the cases that failed, not only the rate. A rate of 0.8 with no list hides whether the ten failures are the important ones, and the list is what a team acts on.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Serving a tuned model

Previous Lesson

Serving a tuned model

Next Lesson

The tuning loop as a program

The tuning loop as a program →