Basic

Selecting and filtering rows

Most questions are answered by a filter: rows where something is true. Write the condition the way you would say it in English, and check the count after each filter. A filter that keeps everything or keeps nothing is the most common silent bug in analysis.

What you will see

0 - which is how you learn a threshold is wrong before you publish a number.

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Filter the fruit rows to counts above 5 and count them.

1

Write the filter

big = [r for r in rows if int(r['count']) > 5] print(len(big), big)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
2 rows: apple 3 is out, pear 12 and fig 7 are in.
2

Break the filter on to prove it works

print(len([r for r in rows if int(r['count']) > 100]))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
0 - which is how you learn a threshold is wrong before you publish a number.

Data Flow

Rows pass through a chain of conditions.
1 Start with all rows
↓
2 Apply one condition at a time
↓
3 Name each intermediate result
↓
4 Print the count after each stage
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

big = [r for r in rows if int(r['count']) > 5] sets nothing on screen
print(len(big), big) 2 rows: apple 3 is out, pear 12 and fig 7 are in.
Why this works
Comparison on the right type

Filtering on a column that is still a string compares text: '9' > '10' is true. Convert once when you load, filter on the real type, and the bug disappears. Convert at the edges, not in the middle of every condition.

Missing values need a decision

Decide what a filter should do with a blank: keep it, drop it, or flag it. Python's None and JavaScript's null compare differently, and NaN is not equal to itself. An explicit rule beats an inherited one.

Examples

Filter as a comprehension

In Python, [r for r in rows if int(r['count']) > 5]. In JavaScript, rows.filter(r => r.count > 5). The condition reads left to right like the question you are asking.

Count after every filter

Print len() after each step while exploring. '412 rows, 89 after the date filter, 4 after the status filter' tells you immediately if a filter is wrong. Without counts you build an answer on an empty set and it looks plausible.

Chaining filters

Each filter feeds the next, so order matters for readability but not for the result. Keep each step in a named variable (recent, then valid, then expensive) so a reader can see what each stage means.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Loading a dataset
Previous Lesson
Loading a dataset
Next Lesson
Computing a new column
Computing a new column →