Master

Rolling back a bad release

A release that turned out worse than the one before it has to be undone, and undone fast. The rollback is not a rebuild from source, it is a switch back to an artifact that already exists. The whole skill is having that artifact ready before the need for it appears.

What you will see

One line with symptom, action, and time. That is the record that makes the next release avoid the same mistake.

About 1 min of reading

Try It

Make the rollback one step.

1

The switch

serving = {'model': 'v2.5.0', 'previous': 'v2.4.9'} print(serving['previous'])

Before you run it, say out loud what it will print. Then open the answer.

What it prints
The previous artifact named in the config - the field that makes a rollback one value change instead of a rebuild.
2

The record

print('v2.5.0 rolled back: upload timeout on Windows; v2.4.9 restored 14:05')

Before you run it, say out loud what it will print. Then open the answer.

What it prints
One line with symptom, action, and time. That is the record that makes the next release avoid the same mistake.

Data Flow

A rollback restores a known artifact.
1 Detect the symptom
↓
2 Switch to the previous artifact
↓
3 Confirm the data still reads
↓
4 Correct the published claims
Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

serving = {'model': 'v2.5.0', 'previous': 'v2.4.9'} sets nothing on screen
print(serving['previous']) The previous artifact named in the config - the field that makes a rollback one value change instead of a rebuild.
Why this works
Time the rollback

How long a rollback takes is a number you want before the emergency. Minutes is a prepared system; an hour of figuring is a system that was never rolled back and never prepared.

Write the postmortem

What shipped, what the symptom was, what the rollback did, what the fix will be. That record is what stops the same bad release being shipped again next quarter, and it is the only record that survives a week.

Examples

Keep the previous build

The artifact before this one stays available and addressable. A rollback that requires rebuilding from a branch is a rollback that can fail, and a failing rollback is a worse situation than the one it was fixing.

Roll back the data too

If the new version migrated files, a rollback has to read them. A loader that only reads the new format makes a rollback that breaks on the first open, which is why migration rules are written per step.

Roll back the announcement

The changelog entry and any published claim have to be corrected, not left. A rollback that leaves a changelog claiming a feature shipped is a system whose documentation is a lie about itself.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← A release that exists only to fix
Previous Lesson
A release that exists only to fix