DailyRNG
Login
Advanced

The overfitting budget

A model has parameters, and each parameter is a chance to fit the past. The budget is the number of parameters a dataset can support, and past that number the result is a description of noise.

What you will see

refit. Five knobs on eight samples is a fit of noise, and the ratio is what says so.

About 1 min of reading

Try It

Count knobs and check the ratio

1

Write the knobs

printf 'window\nsmooth\nthreshold\nscale\ncarry\n' > /tmp/knobs.txt; wc -l < /tmp/knobs.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Five knobs. That number is the budget input.
2

Count independent samples

printf 'days 400\n' > /tmp/s.txt; awk -F' ' '{print $2/50}' /tmp/s.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
8. Four hundred days gives about eight independent windows, and eight is small.
3

Refuse when knobs approach samples

printf 'knobs 5 samples 8\n' | awk '{print ($1/$2>0.4) ? "refit" : "ok"}' -F' '

Before you run it, say out loud what it will print. Then open the answer.

What it prints
refit. Five knobs on eight samples is a fit of noise, and the ratio is what says so.

Data Flow

A dataset to a parameter budget
1 Independent samples are counted
↓
2 The parameter count is written
↓
3 The ratio is compared to the budget
↓
4 Extra knobs are removed until the ratio holds
Why this works
Why regularisation is a budget, not a trick

A penalty is a rule that says small is better. The rule is the budget, and it is the part that has to be stated rather than hidden in a library default.

The test that catches a fitted model

Shuffle the labels and fit again. A model that still scores well is fitting the arrangement, and the arrangement is noise.

Examples

Parameters cost samples

Each parameter needs enough samples to be estimated. A model with more parameters than independent samples is a model that memorised.

Why the count is written down

A model with eleven knobs is a model whose owner cannot say which knob carries the edge. Writing the count makes the question answerable.

The knob that does nothing

A parameter that can be changed without changing the result is a parameter that is not in the model. It is removed, not tuned.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Hedging a position across legs
Previous Lesson
Hedging a position across legs
Next Lesson
Portfolio level risk
Portfolio level risk →