Master

Monitoring and rollback

A pipeline does not end at deploy. You must watch the running app and be able to undo a bad release quickly.

What you will see

Post-deploy smoke test: curl -f health endpoint: failed Rollback on error-rate spike: Previous version: v1 Rollback triggered: Yes Users noticed: No

About 2 min of reading

Try It

Deploying is not the end of the pipeline. After a release I watch the error rate, run a smoke test against the health endpoint, and if the metrics are bad the previous version is swapped back automatically before users ever notice.

I poll the error rate right after deploy and print whether it crossed the threshold, showing the first step of automatic rollback.

def watch_error_rate(current, baseline, threshold): return (current - baseline) > threshold print("Monitoring and rollback") print("After deploy v2: error rate 0.4%, baseline 0.3%, threshold 1%") print("Error spike: %s" % watch_error_rate(0.4, 0.3, 1.0))

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Monitoring and rollback After deploy v2: error rate 0.4%, baseline 0.3%, threshold 1% Error spike: False

A spike in error rate usually means a bad release. Catch it before users complain by watching the metric continuously, not once after the fact.

I hit the health endpoint with a fail-on-error curl, then roll back only if the watch step failed, printing the version being restored and that users stayed unaware.

def watch_error_rate(current, baseline, threshold): return (current - baseline) > threshold print("Post-deploy smoke test:") health_ok = False print("curl -f health endpoint: %s" % ("ok" if health_ok else "failed")) print() print("Rollback on error-rate spike:") print("Previous version: v1") print("Rollback triggered: %s" % ("Yes" if watch_error_rate(5.0, 0.3, 1.0) else "No")) print("Users noticed: No")

Before you run it, say out loud what it will print. Then open the answer.

What it prints

Post-deploy smoke test: curl -f health endpoint: failed Rollback on error-rate spike: Previous version: v1 Rollback triggered: Yes Users noticed: No

Keep the previous version ready so the swap back is instant. A versioned deploy makes rollback trivial; an unversioned one makes it a manual scramble.

Put it in order

Before you write anything, put these lines back in the right order. Click a line to move it up one step. This support step is easier than writing code from scratch, and it is here to make the last example stick.

Data Flow

After deploy, watch metrics; if they are bad, roll back to the last good version.

1 Deploy

↓

2 Watch metrics

↓

3 Metrics bad?

↓

4 Roll back or keep

Line by line

Read the first example one line at a time. A line that prints shows what appears on screen; the rest set things up quietly.

def watch_error_rate(current, baseline, threshold): sets nothing on screen

return (current - baseline) > threshold sets nothing on screen

print("Monitoring and rollback") Monitoring and rollback

print("After deploy v2: error rate 0.4%, baseline 0.3%, threshold 1%") After deploy v2: error rate 0.4%, baseline 0.3%, threshold 1%

print("Error spike: %s" % watch_error_rate(0.4, 0.3, 1.0)) Error spike: False

Why this works

Key metrics

Watch error rate, response time, and server load. A spike usually means a bad release. Catch it before users complain.

Rollback strategy

Keep the previous version ready. If a new release fails, switch back instantly. Versioned deploys make this easy.

Examples

Watching after deploy

After deploy: check error rate, latency, and CPU.
If errors jump, trigger an automatic rollback.

Deploy v2 -> errors rise -> auto-rollback to v1
Users never noticed the bad release.

Monitoring turns a deploy from a scary event into a safe, reversible one.

Post-deploy health check

- name: Smoke test
  run: |
    curl -f https://staging.example.com/health
    echo "status ok" | mail -s "deploy check" oncall

Right after deploy, hit a health endpoint. If it returns non-2xx, the curl -f command fails and the pipeline marks the deploy as broken. Fail fast, alert the on-call engineer immediately.

Rollback on error-rate spike

- name: Watch metrics
  run: ./watch.sh --threshold 1%
- name: Rollback
  if: failure()
  run: ./rollback.sh --to ${{ env.PREVIOUS_VERSION }}

The watch step polls the error rate for a few minutes. If it crosses the threshold, the if: failure() step runs the rollback to the previous version. Users never see the bad release twice.

Words this lesson uses

Each one is defined here on this site. Open the entry for the full definition and the other words it points at.

← Staging and environments

Previous Lesson

Staging and environments