Rollback Plan

Definition

A Rollback Plan is the pre-agreed, pre-tested sequence of steps that will revert a change if it fails after release to production. It exists because no amount of pre-release testing is a substitute for the production environment; every change carries residual risk, and the correct question during a bad release is not "can we fix this?" but "can we get back to a known good state within our error budget?" The rollback plan is the answer to that question, written down before the change goes out, not improvised at 03:00.

Why It Matters

When a production release goes wrong, three things are true simultaneously: the on-call engineer is under pressure, information is incomplete, and every minute of continued failure costs money or trust. In that condition, engineers make choices that seem sensible in the moment and disastrous in review — "let's just push a quick fix," "let's manually adjust the database," "let's disable this feature by editing the config." A written, tested rollback plan replaces improvisation with a rehearsed sequence, and it lets the on-call engineer act with confidence instead of caution.

What Belongs in a Rollback Plan

• Named trigger conditions — the observable signals that mean "roll back now."
• The rollback sequence itself, step by step, with commands or console paths.
• Estimated rollback duration and expected user impact during rollback.
• Data considerations — schema migrations, feature flags, cache states.
• Who has authority to trigger the rollback and who needs to be notified.
• Post-rollback verification — the checks that confirm the system is back to a known good state.

Real-World Example

A logistics platform released a change to its route-optimisation service on a Tuesday evening. Within eight minutes, delivery ETAs were being calculated 40 percent short across the whole fleet, and dispatchers began overriding manually. The on-call engineer's first instinct was to hot-patch the algorithm. The team's practice, however, required the rollback plan to be tried first. The plan was one command — a redeploy of the previous container image with a flag to fall back to the cached ETA model. The engineer ran the command, verified the ETAs corrected within 90 seconds, and the on-call log recorded the whole event as a 14-minute incident. The root cause investigation happened the next morning without any adrenaline in the room. The alternative — hot-patching a poorly-understood algorithm under pressure at 21:15 — was the incident that had cost the team a full evening two months earlier. The rollback plan turned an operational skill into a habit.

What Makes a Rollback Plan Actually Work

1. The rollback is a single command. Anything more than that, under pressure, becomes a source of new incidents. Wrap the sequence into one script or one pipeline.
2. The plan is tested before the change ships. If the rollback has never been executed, the release is riskier than the change itself.
3. Data changes are separated from code changes. Non-reversible schema migrations turn a rollback into a data recovery, and those two problems belong in different windows.
4. The trigger conditions are named. "We know it when we see it" is not a trigger condition; a specific SLO breach, error rate or business metric is.
5. The authority to roll back sits with the on-call engineer. Escalation before rolling back adds minutes the business cannot afford.

Rollback and Modern Deployment Patterns

Blue-green deployments make rollback a routing change — flip the load balancer back to the previous colour. Canary releases make rollback a canary stop and traffic re-shift. Feature flags make rollback a flag flip that takes milliseconds. All of these patterns are, at their core, ways of making the rollback plan cheap and fast. The pattern matters less than the discipline: choose the pattern that fits the change, and rehearse it.

Practical Lessons Learned

• The rollback plan is written by the engineer making the change, not delegated to the reviewer. If the author cannot describe how to reverse the change, the change is not ready.
• Rehearse in staging, not on paper. An untested rollback plan is a wish, not a plan.
• Time-box the "let's fix it forward" temptation. A published rule such as "if not recovered in 10 minutes, roll back" prevents pride-driven decisions during the incident.
• Include rollback duration in the change description. A change with a 30-second rollback and a change with a two-hour rollback deserve different scrutiny.
• Track rollback events as data, not shame. Every rollback is a saved incident.

Expert Tips

• Attach the rollback command to the pull request template. If the field is blank, the PR is not approved.
• Make rollback the default reflex, forward-fix the exception. Culture change; worth the effort.
• Distinguish rollback from restoration. Rolling back a code change is minutes; restoring lost customer data is hours or days. Design changes so the second scenario is rare.
• Publish a rollback runbook per critical service. The plan for a specific change references the service runbook, not the other way round.
• Practise rollback in game days. The muscle memory built during exercises is what carries the team through the real incident.

Common Mistakes

• Assuming the deployment pipeline's "redeploy previous version" button is sufficient without testing it.
• Coupling code and non-reversible schema changes in the same release.
• Requiring escalation to roll back — minutes lost while paging a director.
• No named trigger conditions; the team debates whether to roll back while the incident continues.
• Forward-fixing under pressure and creating a second incident on top of the first.
• Treating a rollback as a career-damaging event, which teaches the team to hide degradations.

Key Takeaways

• The rollback plan is written before the change ships and tested before it is needed.
• The rollback is a single command, executable by the on-call engineer without escalation.
• Trigger conditions are named and observable, not intuitive.
• Data and code changes belong in separate windows to keep rollback cheap.
• Rollback is a saved incident, not a failure — celebrate it accordingly.

Related Concepts

Interlocks with Blue-Green Deployment, Canary Release, Feature Flag, Incident Management and Blameless Postmortem. Rollback runbook templates at PMMilestone.org.