Master

Guarding against a regression after the fix

A fix that is not guarded is a fix that can silently come back. The guard is the same gate that failed, run on every release, plus a record of the number that passed. The record is what turns a later pass into a known pass rather than an unknown one.

What you will see

A new threshold ten percent under what was achieved. Only set this after the gate passes.

About 1 min of reading

Try It

Add the gate and the record

1

Run the gate

./perf-gate.sh; echo "exit=$?"

Before you run it, say out loud what it will print. Then open the answer.

What it prints
exit=0 after the fix. Non-zero means the fix did not move the owner.
2

Record the number

printf '%s p75=%s\n' "$(date -u +%F)" "$p" | tee -a perf-history.md

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One dated line per release. The file grows and never overwrites.
3

Compare to the previous release

tail -2 perf-history.md

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Two lines. If the newer number is worse by more than the noise, that is a regression to chase.
4

Tighten the threshold

printf 'threshold_ms: %s ' "$(( p_int * 9 / 10 ))" | tee -a docs/budget.md

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A new threshold ten percent under what was achieved. Only set this after the gate passes.
Do not tighten below the noise band of the measurement. If five samples spread more than the tightening, the gate will fail on noise.

Data Flow

Fix to a guarded pass
1 Fix the owner
↓
2 Run the gate on the release
↓
3 Record the measured number
↓
4 Compare the next release to the record
Why this works
What the record is for

Comparing releases. Without it, a passing gate tells you the page is under a threshold and nothing else.

Where the guard is wrong

If the gate measures a page nobody visits, it guards the wrong thing. Name the page in the budget and keep it the page users load.

Examples

Run the gate on every release

The gate is part of the release, not a task someone remembers. A gate that runs only when a person chooses is not a guard.

Record the passing number

Store the measured value with the release. A later release that passes at a much worse number is a regression the gate did not catch.

Tighten the budget over time

Once a fix lands, set the next threshold near the number you achieved. Budgets move down as the site gets faster, and the record is what tells you the new number.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Attributing a regression to one owner
Previous Lesson
Attributing a regression to one owner
Next Lesson
Shipping the performance change safely
Shipping the performance change safely →