Advanced

Data quality is the whole job

A tuning run has few knobs and a lot of data. The data is where the quality lives, and the way to see that is to hold the model fixed and change only the set. Two sets of the same size, one curated and one scraped, give a difference large enough to measure.

What you will see

The action. A balanced set is the one where the accuracy number means something.

About 1 min of reading

Try It

Count the labels.

The counts

from collections import Counter labels = ['a']*90 + ['b']*8 + ['c']*2 print(Counter(labels))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

a 90, b 8, c 2 - a set where the model can score 90 percent by always predicting a. That is the trap.

The fix

print('add more b and c examples, or collect the rare cases separately')

Before you run it, say out loud what it will print. Then open the answer.

What it prints

The action. A balanced set is the one where the accuracy number means something.

Data Flow

The set is inspected before training.

1 Count by label and length

↓

2 Remove arguable examples

↓

3 Balance the labels

↓

4 Train on what remains

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

from collections import Counter sets nothing on screen

labels = ['a']*90 + ['b']*8 + ['c']*2 sets nothing on screen

print(Counter(labels)) a 90, b 8, c 2 - a set where the model can score 90 percent by always predicting a. That is the trap.

Why this works

Print the distribution

Count outputs by length, by label, and by input type before training. A set that is ninety percent one shape is a set that teaches one shape, and the count shows it in seconds.

Curate harder, not bigger

Doubling a noisy set does about nothing; doubling a curated one helps. When a run is bad, the next change to try is removing bad examples, not adding more good ones.

Examples

Labelled by a person

Outputs written by someone who knows the job are the gold standard. A set where the outputs came from a weaker model inherits that model's mistakes, and the tuned model learns to repeat them.

Reject the in-betweens

Examples where the wanted output is arguable teach the model a coin flip. Removing them is better than keeping them, and the removal is what makes the loss curve smooth.

Balance the labels

A classification set with ninety percent one label produces a model that predicts that label always. The accuracy number looks good and the model is useless, which is why per label counts are printed.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← When not to fine-tune

Previous Lesson

When not to fine-tune

Next Lesson

The knobs that matter

The knobs that matter →