Skip to content
brainNotFound

Build Systems/Operations

Deployment rollback playbook

What to run, in what order, when a deploy breaks production and you have four minutes of patience left.

beginner8 min

// read first

Read this before you need it. The whole point of a playbook is that the decisions were made calmly, in advance, by someone who was not watching an error graph climb.

The sequence

  1. Step 01

    Confirm it is the deploy

    Compare the failure start time against the deploy timestamp. If they do not line up within a minute or two, rolling back will waste the only window you have.

    bash
    journalctl -u app --since "20 min ago" | grep -i error | head
ls -1dt /home/deploy/app/releases/* | head -3

  2. Step 02

    Revert to the previous release

    One command. It swaps the symlink back and restarts the unit.

    bash
    /home/deploy/app/bin/rollback

  3. Step 03

    Verify from outside the box

    Checking from the server tells you the process is up, not that users can reach it. Test the public URL, and test a page that hits the database.

    bash
    curl -sS -o /dev/null -w '%{http_code} %{time_total}s\n' https://example.com/
curl -sS -o /dev/null -w '%{http_code}\n' https://example.com/docs

  4. Step 04

    Write the timeline while it is fresh

    Five bullet points, before you start debugging: when it broke, how you noticed, what you did, when it recovered, what you do not yet understand. This is the entire value of the incident.

If rollback does not fix it

Then the deploy was not the cause, and you have spent two minutes to learn something valuable. Work outward: database, upstream API, DNS, certificate expiry, disk full. The last one is more common than it should be.

bash
df -h /                       # disk
systemctl --failed            # anything else down
curl -sI https://example.com  # cert + edge
dig +short example.com        # DNS

// related

From the rest of the site.