Master

Attributing a regression to one owner

A regression is only actionable once one owner is named. The move is to bisect: change one variable at a time and keep the rest fixed, so the difference you see belongs to that variable and nothing else.

What you will see

One line naming the file. If you cannot name the file, the bisect is not finished.

About 1 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Bisect to one owner

1

Fix the settings

printf 'connection=throttled\nsamples=5\nviewport=375x812\ncache=cold\n' | tee perf-settings.md

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Four fixed settings. Every later run reads this file so the runs are comparable.
2

Measure the baseline

for i in 1 2 3 4 5; do curl -s -o /dev/null -w '%{time_total}\n' http://localhost:3000/tutorials/lessons; done | sort -n | tail -2 | head -1

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One number, the p75 of five samples at fixed settings.
3

Remove half and re-measure

comment out half the head subresources, reload, run the same five samples

Before you run it, say out loud what it will print. Then open the answer.

What it prints
If the number is unchanged, the owner is in the half you removed. If it drops, the owner is in the half you kept.
4

Write the report line

printf 'p75=%s threshold=1500ms owner=public/css/main.css action=split first-patch inline\n' "$p" | tee -a perf-report.md

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One line naming the file. If you cannot name the file, the bisect is not finished.
Do not change the connection and remove an asset in the same run. The two changes make the difference unattributable.

Data Flow

Gap to one owner
1 Fix connection, samples, viewport, cache
↓
2 Remove half the assets, measure
↓
3 Keep the half that still misses
↓
4 Repeat until one file remains
Why this works
Why one variable at a time

Two moving variables produce a difference that belongs to neither. The cost of fixing the rest is a few more runs, and the gain is a claim you can act on.

When the owner is a feature

If no single asset explains the gap, the owner is a feature. Bisect features the same way, by disabling one and re-measuring.

Examples

Fix everything else

Same connection, same sample count, same viewport, same cache state. If two variables move together, the measurement tells you nothing about either.

Bisect the asset set

Remove half the assets, measure. Still slow: the owner is in the half you kept. Fast: it is in the half you removed. Three rounds names one file out of eight.

Name the owner in the report

The report line names the file, the number, and the threshold. A report that names only the number cannot be acted on by anyone else.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← A budget that a person can act on
Previous Lesson
A budget that a person can act on
Next Lesson
Guarding against a regression after the fix
Guarding against a regression after the fix →