Basic

Epochs and steps

Training walks the set repeatedly. One pass is an epoch, and the number of passes is a knob you set. Too few leaves the model near its starting point, too many makes it repeat the training set, and the loss curve shows which is happening.

What you will see

Index 2 - the third pass was the best on held out data, so that is the checkpoint to keep, not the last one.

About 1 min of reading

Try It

Read a tiny curve.

1

The losses

loss = [2.1, 1.4, 0.9, 0.7, 0.66, 0.65] print(loss)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Falling then flattening. The flattening is the point past which more epochs change little.
2

The stop

held = [1.9, 1.2, 1.0, 1.05, 1.2] print(held.index(min(held)))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Index 2 - the third pass was the best on held out data, so that is the checkpoint to keep, not the last one.

Data Flow

Passes produce a loss curve.
1 Walk the set once
↓
2 Record the loss
↓
3 Repeat for each epoch
↓
4 Keep the best checkpoint
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

loss = [2.1, 1.4, 0.9, 0.7, 0.66, 0.65] sets nothing on screen
print(loss) Falling then flattening. The flattening is the point past which more epochs change little.
Why this works
Record the curve

Save the loss after each pass. Without the curve you have a number and no way to know whether the run stopped early or ran too long, which is the question the number alone cannot answer.

Stop at the held out best

Evaluate on the held out set each pass and keep the checkpoint that scored best. That is the rule that turns 'how long do I train' into a measured answer.

Examples

Small sets need more passes

A hundred examples seen once is a weak signal. The same hundred seen ten times is a strong one, which is why a small set usually needs more epochs than a large one.

More is not better

Past the point where the set is learned, extra passes push the model toward copying those exact examples. That is overfitting, and it is visible as training loss falling while held out loss stops falling.

Batch size is memory

A larger batch is fewer updates from the same data. It changes speed and memory, not what is learned, so set it by what fits rather than by a guess about quality.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Formatting the examples
Previous Lesson
Formatting the examples
Next Lesson
Reading the loss curve
Reading the loss curve →