Master

A reproducible analysis pipeline

An analysis that cannot be re-run is not an analysis. The number in the report has to come back from the same script on the same input, months later, by a different person. That means ordering the steps, naming them, and refusing to hand edit the middle.

What you will see

3 2 - and if a cleaning step ever introduced a row or a None, the assert names the stage that did it.

About 2 min of reading

Try It

Add an invariant that catches a bad cleaning step.

Clean and assert

raw = [{'v': 1}, {'v': None}, {'v': 3}] clean = [r for r in raw if r['v'] is not None] assert len(clean) <= len(raw), 'cleaning must not add rows' assert all(r['v'] is not None for r in clean), 'no unknowns left' print(len(raw), len(clean))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

3 2 - and if a cleaning step ever introduced a row or a None, the assert names the stage that did it.

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Stages chain from raw file to report.

1 Load and record the input

↓

2 Clean with encoded rules

↓

3 Transform and assert invariants

↓

4 Aggregate and report

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

raw = [{'v': 1}, {'v': None}, {'v': 3}] sets nothing on screen

clean = [r for r in raw if r['v'] is not None] sets nothing on screen

assert len(clean) <= len(raw), 'cleaning must not add rows' sets nothing on screen

assert all(r['v'] is not None for r in clean), 'no unknowns left' sets nothing on screen

print(len(raw), len(clean)) 3 2 - and if a cleaning step ever introduced a row or a None, the assert names the stage that did it.

Why this works

Assert the invariants between stages

Row counts should not grow through cleaning; a share column should still sum to one; a key should stay unique. Assert these between stages so a bad transformation fails at the stage that caused it rather than in the report.

Make the whole thing one command

One command from raw file to finished report is the test that the pipeline is reproducible. If a human has to click something in the middle, that step is not reproducible and will not survive the next month.

Examples

Fixed stages, named

load, clean, transform, aggregate, report. Each stage takes the previous stage's output and writes the next. A stage is a function with a name, so a reader can run any stage alone and check it.

No hand edits in the middle

The moment you delete rows by hand in a spreadsheet, the script no longer produces the report. Encode the exclusion as a rule in the code so the next run gives the same rows and the rule is visible.

Pin the inputs

Record the file name, the date pulled, and the row count at the top of the report. When the numbers change, that header tells you whether the input changed or the analysis did.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Outliers and what a mean hides

Previous Lesson

Outliers and what a mean hides

Next Lesson

Feature engineering

Feature engineering →