DailyRNG
Login
Master

Rolling back a release that is already live

Rollback is the one switch that makes a release system survivable, and it is the switch people get wrong most often. A rollback is only safe if the data shape is still readable by the older box. If the new release migrated data one way, rolling back the code without a data path is a outage.

What you will see

At least 1. The rollback check runs as part of the release gate, not as a separate manual step.

About 1 min of reading

Try It

Make a rollback real by testing it before shipping

1

Keep the previous box

docker images --format '{{.Tag}}' | grep -E '^v1\.4\.' | head -2

Before you run it, say out loud what it will print. Then open the answer.

What it prints
At least two tags. If only the current one exists, there is nothing to roll back to.
2

Write the migration data

docker run --rm app:v1.4.2 python -c 'import app; app.write_notes('''app.db''')'

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Data in the new shape, written by the new release.
3

Run the old box against it

docker run --rm app:v1.4.1 python -c "import app; app.read_notes('app.db')"

Before you run it, say out loud what it will print. Then open the answer.

What it prints
Exit 0. The older release can read the newer data, which is the precondition for a safe rollback.
4

Put it in the gate

grep -c 'v1.4.1' release-gate.sh

Before you run it, say out loud what it will print. Then open the answer.

What it prints
At least 1. The rollback check runs as part of the release gate, not as a separate manual step.
If the old box cannot read the new data, do not ship the release until you add a path that lets it. Shipping first means the next rollback is an outage.

Data Flow

Live release to previous release
1 Keep v(N-1) artifact on disk
↓
2 Assert v(N-1) reads data written by vN
↓
3 On failure, repoint to v(N-1)
↓
4 Log the pointer change and the reason
Why this works
The rollback gate

Before publishing vN, assert that v(N-1) starts and passes its contract tests against data written by vN. That single assertion is what makes the rollback real rather than intended.

When rollback is not enough

If the release deleted data or changed a wire format a third party reads, rollback cannot restore it. Those releases need a written forward-only note, and the plan page should say so.

Examples

Two artifacts, one decision

Keep the previous release's box and its lock file on the machine. The rollback is a pointer change, not a rebuild, so it works when the network or the registry is down.

The data is the trap

If v1.5 wrote a new column layout, v1.4 must still read it for the rollback to work. That is the two-release window: the older box supports the newer shape for exactly one release.

Test the rollback before the release

Run the older box against the migrated data as part of the gate. If that check is not in the gate, the rollback is untested and you will find out during an outage.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Debugging a container that will not start
Previous Lesson
Debugging a container that will not start