DailyRNG
Login
Master

Shipping a model change with its number

A model change ships with a sentence attached: the set version, the score before, the score after, and the regressions that were cleared. That sentence is what the next person reads when they wonder why the system changed.

What you will see

One line differs. A rollback that changes one line is a rollback that can be done under pressure.

About 1 min of reading

Try It

Write the change note and ship

1

Write the four field note

printf 'set v2 | before 0.75 | after 0.83 | regressions cleared q2\n' > /tmp/note.txt; cat /tmp/note.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One line, four fields. That line is the whole contract of the change.
2

Attach the note to the deploy

printf 'system=answerB set=v2\n' > /tmp/deploy.env; cat /tmp/deploy.env /tmp/note.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
The config names the set version, so the note and the config point at the same run.
3

Prove the rollback is one line

printf 'system=answerA set=v2\n' > /tmp/rollback.env; diff /tmp/deploy.env /tmp/rollback.env

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One line differs. A rollback that changes one line is a rollback that can be done under pressure.

Data Flow

A cleared run to a shipped change
1 The gate allows the change
↓
2 The note is written with four fields
↓
3 The change is deployed with the note attached
↓
4 The next change compares against this stored pair
Why this works
Why the note is written before the deploy

Writing the note first forces the number to exist. A note written after is filled in with whatever the run happened to show.

The end state of the eval system

Frozen set, named grader, stored runs, gate with a written override. Four objects, and every model change since is comparable.

Examples

The change note is four fields

Set version, before, after, regressions cleared. Anything longer is a document nobody skims before deploying.

Why the set version travels with the number

A score without a set version is a number that cannot be reproduced. Six months later it is a number that cannot be argued with.

Rolling back is a config change

Because the previous system and set version are both named, a rollback is pointing at them again. No reconstruction is needed.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Instrumenting the eval itself
Previous Lesson
Instrumenting the eval itself