DailyRNG
Login
Advanced

Loops versus vectorised work

A loop over a million rows is fine in a tutorial and slow in production. The fix is not cleverer code, it is moving the inner step into a library that does it in C. Measure before you optimise, and measure after.

What you will see

The same number, faster - the loop moved into C. Same answer is the point, not the timing.

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Compare a loop to a comprehension on the same work.

1

Time the loop version

import time t = time.time() acc = 0 for i in range(1_000_000): acc += i print(acc, round(time.time() - t, 3))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
499999500000 and a time around a tenth of a second or more.
2

Time the built-in version

t = time.time() print(sum(range(1_000_000)), round(time.time() - t, 3))

Before you run it, say out loud what it will print. Then open the answer.

What it prints
The same number, faster - the loop moved into C. Same answer is the point, not the timing.

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

Work moves from per element to per batch.
1 Time the whole script
↓
2 Find the step that dominates
↓
3 Move that step into a library call
↓
4 Assert the answer is unchanged
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

import time sets nothing on screen
t = time.time() sets nothing on screen
acc = 0 sets nothing on screen
for i in range(1_000_000): sets nothing on screen
acc += i sets nothing on screen
print(acc, round(time.time() - t, 3)) 499999500000 and a time around a tenth of a second or more.
Why this works
Correctness first, then speed

A vectorised version that gives a different answer is worse than a slow version that is right. Keep the readable loop as a test case and assert the fast version matches it on a small input.

Memory is a limit too

Loading a file that is larger than RAM fails, and the failure is a crash rather than a slow answer. Stream in chunks or aggregate in one pass when the dataset will not fit, and know the size before you try.

Examples

Time it before you change it

Wrap the step in a timer and print seconds. If the whole script takes 0.4 seconds, an optimisation is noise. If one step takes 40 of the 45 seconds, that is the only thing worth changing.

Vectorise the inner step

In Python, a list comprehension or a numpy operation replaces a per element loop; in JavaScript, one typed array operation replaces a per element loop. The win is that the per element work leaves the interpreter.

Read once, reuse many times

The biggest speedups come from not repeating work: parse the file once, cache the index, and reuse the parsed rows across all the questions. Re-reading a file inside a loop is the bug that turns seconds into minutes.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Long and wide
Previous Lesson
Long and wide
Next Lesson
Dates and time zones
Dates and time zones →