Advanced

Outliers and what a mean hides

A mean is a summary, and summaries hide the shape of the data. One huge value can move a mean by ten times while the median does not move. Look at the distribution before you quote an average.

What you will see

[2.0, 3.0, 4.0] - the quartiles show the tight cluster and make the 1000 visibly separate.

About 1 min of reading

Try It

See a mean hide an outlier.

1

Compute both

import statistics v = [1,2,3,4,1000] print(statistics.mean(v), statistics.median(v))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
202.0 3 - the mean is dragged by one value; the median is not.
2

Percentiles

print(statistics.quantiles(v, n=4))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
[2.0, 3.0, 4.0] - the quartiles show the tight cluster and make the 1000 visibly separate.

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

The distribution is inspected before summarising.
1 Sort and inspect the extremes
↓
2 Compute median and percentiles
↓
3 Compare to the mean
↓
4 Report the shape, not just the average
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

import statistics sets nothing on screen
v = [1,2,3,4,1000] sets nothing on screen
print(statistics.mean(v), statistics.median(v)) 202.0 3 - the mean is dragged by one value; the median is not.
Why this works
Is it an outlier or an error

A real extreme is a finding; a unit error is a bug. Check the source of the biggest values before deciding. Dismissing a real outlier with an automatic rule throws away the interesting part of the data.

Log scale for ratios

When values span three orders of magnitude, a linear chart puts everything in one bar and one spike. A log scale makes ratios readable, and labelling it as log prevents a reader reading it as a normal bar.

Examples

Median versus mean

For [1,2,3,4,1000] the mean is 202 and the median is 3. Both are correct and only one is useful. Quote both when they differ, because the difference is the finding.

Percentiles instead of adjectives

'p50 is 4, p95 is 90, max is 1200' says more than 'mostly small with a few big ones'. Percentiles are checkable numbers, so two people can agree on the description instead of arguing about a word.

Check the biggest values first

Sort descending and look at the top ten before any statistics. A single input error, a unit mistake, or a duplicate row shows up there, and it changes every summary you were about to publish.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Dates and time zones
Previous Lesson
Dates and time zones
Next Lesson
A reproducible analysis pipeline
A reproducible analysis pipeline →