Basic

The first look at a new dataset

Before any analysis, spend two minutes describing what you have. Row count, column types, and a few sample values catch a wrong file or a broken column before they corrupt a conclusion. Skipping this step is the most expensive shortcut in analysis, because the error it hides is invisible later.

What you will see

3 3 - equal means no duplicate observations. If they differed, every total would be inflated before you noticed.

About 2 min of reading

Try It

Describe the fruit rows before computing anything.

1

Count and sample

print(len(rows), rows[0], rows[-1])

Before you run it, say out loud what it will print. Then open the answer.

What it prints
3 plus the first and last dictionaries - you now know the file is complete and the columns are named name and count.
2

Check the key is unique

names = [r['name'] for r in rows] print(len(names), len(set(names)))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
3 3 - equal means no duplicate observations. If they differed, every total would be inflated before you noticed.

Data Flow

The file is described before it is analysed.
1 Print the row count
↓
2 Sample the first and last rows
↓
3 Check type and range per column
↓
4 Compare to what you expected
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

print(len(rows), rows[0], rows[-1]) 3 plus the first and last dictionaries - you now know the file is complete and the columns are named name and count.
Why this works
Write down what you expected

Before the first look, write the row count and the columns you expect. Comparing expectation to reality turns the description into a test. Without the expectation, everything looks normal and nothing is checked.

Uniqueness of the row key

If one row should mean one thing, the key column should have no duplicates. Count distinct values against the row count. A mismatch means duplicated observations, and duplicates double count every total you are about to report.

Examples

Count and sample

Print len(rows), the first three rows, and the last three. That is enough to see the shape, the ordering, and whether the file is truncated. It takes ten seconds and it catches most file level mistakes.

Check the types column by column

For each column, print the type of one value and whether every value parses as the expected type. A column of IDs that is sometimes blank or sometimes text is a bug you want to find now, not after the grouping.

Look for the range

Min and max per numeric column. A percentage that goes to 400, an age of 1970, a date in the year 1026 all show up here as absurd ranges, and each one is a data entry bug worth finding before the average is published.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

Next Lesson
Loading a dataset
Loading a dataset →