Advanced

The knobs that matter

Learning rate, epochs, and batch size are the visible knobs, and each has a symptom it produces when wrong. The way to set them is to change one at a time and read the held out score, because two wrong knobs can cancel and leave a number that looks fine.

What you will see

A real gain. If you changed two knobs at once, this gain cannot be attributed to either.

About 1 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Compare two runs.

1

The scores

base = 0.61; one_change = 0.74 print(one_change - base)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A real gain. If you changed two knobs at once, this gain cannot be attributed to either.

Data Flow

One knob at a time, measured.
1 Record the baseline score
↓
2 Change one knob
↓
3 Rerun with the same seed
↓
4 Compare held out scores
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

base = 0.61; one_change = 0.74 sets nothing on screen
print(one_change - base) A real gain. If you changed two knobs at once, this gain cannot be attributed to either.
Why this works
One change per run

Change one knob, rerun, compare the held out number. Changing three at once produces a better number you cannot explain, which is a result you cannot repeat.

Seed the run

Two runs of the same setup can differ. Fixing the seed and recording it is what makes a difference between runs real rather than noise.

Examples

Learning rate

Too high and the model overshoots and the curve is jagged. Too low and the curve barely moves. Both symptoms are visible, and both are fixed by moving one notch rather than guessing a new value.

Epochs

Set by the held out curve, not by a habit. The right count is where held out loss stops falling, and that point moves with the size and variety of the set.

Batch size

It changes speed and memory. If quality changes with batch size alone, something else in the setup is also wrong, since batch size should not be doing the learning.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Data quality is the whole job
Previous Lesson
Data quality is the whole job
Next Lesson
LoRA and partial tuning
LoRA and partial tuning →