Master

Running the suite on every change

The suite passed on my machine and broke on the server. A test suite is only as strong as the habit of running it, and the habit is to run it automatically on every change, everywhere.

What you will see

>>>

About 2 min of reading

Quick check

Before this new lesson, one small question about the lesson you just finished. Recall it, do not look it up.

Try It

Run the whole suite, read the exit code, and let it gate a step.

Run every test in the project. The summary counts pass and fail across all files.

pytest

Before you run it, say out loud what it will print. Then open the answer.

What it prints

collected 12 items test_add.py ... test_props.py .. test_fix.py .. test_reg.py . test_prop.py . test_mock.py .. ============================== 12 passed in 0.20s ==============================

A red suite exits non-zero. This is the signal a build or hook acts on.

pytest; echo $?

Before you run it, say out loud what it will print. Then open the answer.

What it prints

FAILED test_add.py::test_add_zero ============================== 1 failed, 11 passed ============================= 1

Gate a deploy on the suite. If it is red, the deploy step never runs.

pytest && deploy()

Before you run it, say out loud what it will print. Then open the answer.

What it prints

>>>

The habit is the point, not the command. A suite nobody runs on every change is a false sense of safety. Wire it into CI and the commit hook so the guard runs whether or not you chose to run it.

Data Flow

The whole suite runs on every change, a red run exits non-zero, and that exit code gates the build or deploy so a failure cannot ship.

1 Every change triggers the suite

↓

2 Red run exits non-zero

↓

3 Exit code gates the build

↓

4 CI runs it on the real environment

Why this works

The suite is a gate

A red suite should block the merge or the deploy, not be a number you glance at and dismiss. Wire the exit code into the build so a failure cannot be ignored. The discipline is that shipping with a red suite is not an option, which is what keeps the wall honest over time.

Run it where it counts

Running locally proves nothing about the server. The same suite must run in CI, on the real environment, on every change. That is the habit that turns a suite from a local aid into a permanent guard, and it is where the works-on-my-machine bugs get caught.

Examples

Run the whole suite

pytest

This runs every test in the project, not one file. The suite is the wall of guards, and running it is the single action that checks the wall after any edit.

Fail the build on red

pytest || exit 1

A red suite exits non-zero, so a build or a commit hook can refuse to ship. The test result becomes a gate, not a suggestion you can ignore.

Run it in CI

- run: pytest

The same command runs on the CI server on every push. The machine that checks the wall is not the machine you trust, which is what catches the works-on-my-machine class of bug.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Testing the seam, not the world

Previous Lesson

Testing the seam, not the world