Basic

Reporting a frontend test so a person can act

A frontend test that prints FAIL is a test that produces a bug report nobody can file. The report needs the URL, the assertion, and the observed value. Those three are enough to open an issue and start a fix.

What you will see

exit=0 on a clean run and non-zero with failures. A suite that always exits 0 gates nothing.

About 1 min of reading

Try It

Add the report format to a suite

1

Print the three fields

printf 'FAIL url=%s expected=%s observed=%s ' "$u" "$want" "$got" | tee -a /tmp/report.txt

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One line naming the URL, the wanted value, and what was seen.
2

Collect instead of stopping

bad=0; for u in /a /b /c; do check "$u" || { printf 'FAIL url=%s\n' "$u"; bad=$((bad+1)); }; done; echo "failures=$bad"

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Every failure printed and counted, then a non-zero exit at the end.
3

Print the totals

echo "passed=$((total-bad))/$total"

Before you run it, say out loud what it will print. Then open the answer.

What it prints
A fraction a person can compare to the expected total for the suite.
4

Confirm the exit code

./fe-suite.sh; echo "exit=$?"

Before you run it, say out loud what it will print. Then open the answer.

What it prints
exit=0 on a clean run and non-zero with failures. A suite that always exits 0 gates nothing.
Do not exit on the first failure. One run should name every breakage, or the next run hides a second bug.

Data Flow

Failure to an actionable line
1 Run the assertion
↓
2 On failure, print URL expected observed
↓
3 Continue to the next assertion
↓
4 Exit non-zero if any failed
Why this works
Where the report goes

The same place as every other test run, so a frontend failure is not a different kind of thing to notice.

When to stop the run

Do not stop on the first failure in a suite. Collect all failures and exit non-zero at the end, so one run names every breakage.

Examples

Three fields per failure

URL, expected, observed. A failure without the observed value cannot be compared to what the browser shows.

One line per failure

One line each, so the output can be pasted into an issue. A stack trace from a crawl names nothing actionable.

Count at the end

Print the pass count and the total. A suite that prints nothing on success is indistinguishable from a suite that did not run.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Testing state that a page depends on
Previous Lesson
Testing state that a page depends on
Next Lesson
Driving a real browser from a test
Driving a real browser from a test →