DailyRNG
Login
Basic

Calibrating the error, not the value

A model that predicts 0.02 and is wrong by 0.05 is not a model with an edge. The error bar is the thing that decides whether the edge exists, and it is estimated separately from the value.

What you will see

refuse. The bar decided, and that is the calibration working.

About 1 min of reading

Try It

Print a value with its bar

1

Write value and bar

printf 'value 0.07\nbar 0.05\n' > /tmp/c.txt; cat /tmp/c.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Two numbers. The bar is most of the value, which is already the story.
2

Compute the worst case edge

awk -F' ' '{s[$1]=$2} END {printf "%.2f\n", s["value"]-s["bar"]}' /tmp/c.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
0.02. The worst case is smaller than the cost, so the honest answer is refuse.
3

Refuse on the bar

awk -F' ' '{s[$1]=$2} END {print (s["value"]-s["bar"]>0.05) ? "size" : "refuse"}' /tmp/c.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
refuse. The bar decided, and that is the calibration working.

Data Flow

A value to a decision
1 The model gives a value
↓
2 The error is estimated on held out data
↓
3 The bar is printed with the value
↓
4 The size is computed from value minus bar
Why this works
Why the bar is printed with the value always

A value alone invites a size. A value with a bar invites a decision, and the decision is the thing that is correct.

The failure of a shrinking bar

A bar that shrinks every run is a bar that is being estimated on the answer. The estimator has to be fixed before the model is.

Examples

The bar is measured on the same data

An error bar from a different window is a decoration. It has to come from the same sample the value came from.

Why a wide bar is not a warning

A wide bar is a decision. It says refuse, and refusing is cheaper than losing.

The calibration set is held out

Calibrating on the data the model was fitted on measures the fit, not the world. The held out set is the only honest bar.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Data feeds and what they hide
Previous Lesson
Data feeds and what they hide
Next Lesson
Walk forward instead of one split
Walk forward instead of one split →