DailyRNG
Login
Basic

Grouping and aggregating

Grouping answers 'per what?'. Pick the thing one row describes, group by it, and reduce each group to one number. The classic error is grouping by something that looks unique per row, which gives you a group of one each time.

What you will see

[(4, 3), (5, 1), (3, 1)] - the biggest group first.

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Group a list of words by length and count.

1

Build the buckets

words = ['fig','apple','pear','kiwi','plum'] buckets = {} for w in words: buckets.setdefault(len(w), []).append(w) print(buckets)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
{3: ['fig'], 5: ['apple'], 4: ['pear', 'kiwi', 'plum']}.
2

Reduce to counts and sort

summary = sorted(((k, len(v)) for k, v in buckets.items()), key=lambda kv: -kv[1]) print(summary)

Before you run it, say out loud what it will print. Then open the answer.

What it prints
[(4, 3), (5, 1), (3, 1)] - the biggest group first.

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Rows collect into buckets, then reduce.
1 Choose the key that one row describes
↓
2 Append each row to its bucket
↓
3 Reduce each bucket to one number
↓
4 Sort the summary by that number
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

words = ['fig','apple','pear','kiwi','plum'] sets nothing on screen
buckets = {} sets nothing on screen
for w in words: sets nothing on screen
buckets.setdefault(len(w), []).append(w) sets nothing on screen
print(buckets) {3: ['fig'], 5: ['apple'], 4: ['pear', 'kiwi', 'plum']}.
Why this works
Check the group count

If the number of groups equals the number of rows, you grouped by a unique field and learned nothing. If it is one, you grouped by a constant. Both bugs produce a plausible looking table, so print the group count and compare it to what you expected.

Sort by the number, not the key

Sort the summary by the value you care about so the biggest cases are at the top. Alphabetical order hides the answer in the middle of a long list and makes a small dataset look like a large one.

Examples

Group with a dictionary

buckets = {} then for r in rows: buckets.setdefault(r['region'], []).append(r). Now each key holds its rows and you can reduce each list. This is the same shape as a SQL GROUP BY.

Reduce to one number per group

sum, count, mean, max, min are the reductions you will use most. Decide which one answers the question. A mean over a group of two and a mean over a group of two thousand look identical in a table and mean very different things.

Always keep the group size

Report the count next to the average. 'mean 4.2 (n=3)' tells the reader the number is shaky; 'mean 4.2' alone invites them to treat it as solid. This single habit removes a whole class of misleading summary.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Computing a new column
Previous Lesson
Computing a new column
Next Lesson
A chart that answers the question
A chart that answers the question →