DailyRNG
Login
Advanced

Taming a flaky frontend test

A test that passes sometimes is worse than a test that fails always. The cause is almost always a wait the test did not make: it read a state before the work that produced it finished. The fix is to wait for the condition, never for a duration.

What you will see

Zero. If it is still above zero, the cause is not the wait and the guess was wrong.

About 1 min of reading

Try It

Make one flaky assertion deterministic

1

Run it twenty times

for i in $(seq 20); do ./fe-one.sh > /dev/null 2>&1 || echo "fail $i"; done | tee /tmp/flakes.txt; wc -l < /tmp/flakes.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
The failure count. Zero is a pass, anything else names the flake rate.
2

Replace the sleep

swap any fixed wait for a loop that checks the condition and breaks when true

Before you run it, say out loud what it will print. Then open the answer.

What it prints
The assertion now waits for the state, not for a duration.
3

Add the ceiling

after the wait loop, if the condition is still false, print FAIL with the condition name and exit 1

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A timeout becomes a named failure instead of a hang or a silent pass.
4

Re-run twenty times

for i in $(seq 20); do ./fe-one.sh > /dev/null 2>&1 || echo "fail $i"; done | wc -l

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Zero. If it is still above zero, the cause is not the wait and the guess was wrong.
Do not add a retry to make a test green. A retry that hides a real failure turns a red suite into a green one that lies.

Data Flow

Flake to a waited assertion
1 Run the test twenty times
↓
2 Find the assertion that fails sometimes
↓
3 Replace the sleep with a condition wait
↓
4 Add a ceiling that fails with a message
Why this works
How to find the cause

Run the test twenty times and read the failures. The same line failing each time is the assertion that does not wait.

Where flake is environmental

A slow server under test load answers late. Fix the wait in the test, and separately make the server warm before the suite starts.

Examples

Wait for the condition

Poll until the element exists, with a ceiling. A fixed sleep is a guess, and a guess that is too short is the flake.

Name the ceiling

A wait with no ceiling turns a hang into a stuck test run. The ceiling turns it into a failure with a message.

Retry only what is external

A retry hides the flake where the cause is unknown. Retry only a known external dependency, and log that you retried.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Testing reachability and focus order
Previous Lesson
Testing reachability and focus order
Next Lesson
Putting frontend tests in the release
Putting frontend tests in the release →