Advanced

Serving a tuned model

A tuned model is an artifact with a version, a base, and a set of examples behind it. Serving it means those three are known, because a model served with the wrong template or the wrong base produces output that is confidently wrong rather than obviously broken.

What you will see

Prompted beats tuned. Ship the prompted base and keep one model, which is a smaller system for the same quality.

About 1 min of reading

Try It

Run the three way eval.

1

Three scores

base = 0.58; prompted = 0.71; tuned = 0.83 print(base, prompted, tuned)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Three numbers. Tuned wins here, and the gap over prompted is what justifies the training run.
2

The other case

print(0.58, 0.80, 0.79)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Prompted beats tuned. Ship the prompted base and keep one model, which is a smaller system for the same quality.

Data Flow

An artifact ships with checks.
1 Name the run and date
↓
2 Pin the training template
↓
3 Run the eval on deploy
↓
4 Keep the previous model ready
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

base = 0.58; prompted = 0.71; tuned = 0.83 sets nothing on screen
print(base, prompted, tuned) Three numbers. Tuned wins here, and the gap over prompted is what justifies the training run.
Why this works
One eval, every candidate

Run the same test set against the base, the prompted base, and the tuned model. Publishing the three numbers is what makes the choice defensible to anyone reading the deploy.

Revert must be instant

The previous model stays loaded or one step away. A tuned model that turns out bad has to be swapped in minutes, and a rebuild is not minutes.

Examples

Version the artifact

Name the model with the run that produced it and the date. Two tuned models that differ by one epoch are indistinguishable in production unless one of them has a name.

Pin the template

The serving code carries the exact template used in training, in one place, tested. A template edited later is the bug where quality drops with no code change visible in the diff.

Keep the eval running

The held out test set runs against the served model on deploy. That is the check that catches a wrong base, a wrong template, and a bad run, and it is the only place all three are caught.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Generating examples at scale
Previous Lesson
Generating examples at scale
Next Lesson
An evaluation harness for a tuned model
An evaluation harness for a tuned model →