Basic

Missing values

Blank cells are not zeros and they are not errors: they are unknown. Every statistic you take over a column with unknowns needs a stated rule for what to do with them. The default behaviours differ between languages, which is why the same dataset gives different numbers on each side of the team.

What you will see

6.333... 3 2 - the average over the three real values, with the two gaps counted.

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Average a column that has blanks, honestly.

1

Build a column with gaps

raw = ['3', '', '7', 'nan', '9']

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Two problem values: an empty string and the text nan.
2

Clean and count

import math vals = [] skipped = 0 for s in raw: try: v = float(s) except ValueError: skipped += 1; continue if math.isnan(v): skipped += 1; continue vals.append(v) print(sum(vals)/len(vals), len(vals), skipped)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
6.333... 3 2 - the average over the three real values, with the two gaps counted.

Data Flow

Raw cells become values or an explicit unknown.
1 Read the raw cell
↓
2 Test for blank, None and NaN
↓
3 Return a value or an explicit unknown
↓
4 Count the unknowns you skipped
Why this works
Decide the rule once, apply it everywhere

Write one function that turns a raw cell into a value or None, and use it in every analysis. Two different blank handling rules in one report is the bug that makes two sections disagree.

A gap can be the finding

Before dropping rows with blanks, count them by group. Missing values that cluster in one region or one month are a signal about the collection process, not noise to be discarded.

Examples

Empty string is not zero

int('') crashes; float('') crashes; treating '' as 0 invents data. Decide whether the row should be dropped, whether the column should be skipped, or whether the gap should be reported as a count of unknowns.

NaN is not equal to itself

In Python, float('nan') != float('nan') and in JavaScript NaN !== NaN. So 'x == NaN' is always false; use math.isnan() or Number.isNaN() to test. This surprises anyone writing a filter for bad numbers.

Report the coverage

'average 4.2 over 118 of 140 rows' is honest. 'average 4.2' hides that 22 rows had no value, which is often the interesting finding. Print the denominator with every statistic.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← A chart that answers the question
Previous Lesson
A chart that answers the question
Next Lesson
Joining two datasets
Joining two datasets →