Runbook

Definition

A Runbook is a short, structured operational document that tells an engineer — often on-call at 03:00 — exactly what to do when a specific alert fires or a specific class of incident occurs. Good runbooks are written for a stressed human with limited context. They contain commands to copy-paste, dashboards to open, decision points to evaluate, and escalation paths to follow. They are one of the highest-leverage artefacts an operations team can produce.

What a Good Runbook Contains

• Alert / symptom that triggers the runbook (linked from the alert message itself).
• Impact statement — who is affected, how severely.
• Detection — which dashboards and queries confirm the incident.
• Diagnosis — the ordered checks that isolate the root cause.
• Remediation — the specific commands or actions to restore service, in copy-paste form.
• Verification — how to confirm the fix worked.
• Escalation — who to page and when, with contact channels.
• Post-incident actions — logs to capture, tickets to open, comms to send.

Why It Matters

Alerting without runbooks is punishment. The engineer wakes up, sees an alert they've never seen, spends 40 minutes reconstructing tribal knowledge, and either resolves the incident poorly or escalates in panic. Runbooks compress that 40 minutes to five. They also make on-call rotations sustainable — junior engineers can take primary on-call because the runbook contains what a senior engineer would have known.

Real-World Example

A SaaS platform ran a post-mortem on a two-hour outage caused by a Redis eviction storm. The alert had fired to the on-call engineer at 02:14; the incident was declared at 02:47 after 33 minutes of exploratory investigation; the actual remediation (increase eviction target, restart affected pods) took 90 seconds once identified. The team wrote a runbook the next day: three symptoms, one diagnosis command, one remediation script, one verification query. Six weeks later the same alert fired. Time to resolution: 4 minutes. The engineer paged did not need to wake anyone else. The runbook had already paid for itself several times over on its first genuine use.

Practical Lessons Learned

• Write runbooks from real incidents. Speculative runbooks are usually wrong; incident-driven ones capture what actually matters.
• Link the alert to the runbook. The runbook URL belongs in the alert body, not in a wiki three clicks away.
• Test them in game days. A runbook that has never been executed is a hope, not a control.
• Prefer copy-paste over prose. An engineer at 03:00 does not want to read; they want to type.
• Version them with the code. Runbooks that live in a wiki go stale; runbooks in the repo update with the deploy.

Common Mistakes

• Wiki runbooks that document last year's architecture and confidently mislead the on-call engineer.
• Runbooks written by architects who have never been on-call — long, comprehensive, and useless.
• No link from the alert to the runbook — the runbook exists but nobody finds it in the moment.
• Runbooks that describe symptoms without specifying the exact commands to run.
• No verification step, so engineers roll back an outage without confirming they've actually fixed it.
• Escalation paths that reference people who left the company.
• One giant runbook per service instead of one per alert — the on-call engineer scans past the relevant section.

Expert Tips

• One runbook per alert, one page maximum. Longer runbooks mean smaller runbook adoption.
• Include the "should I page the escalation?" decision explicitly. Engineers hesitate at 03:00; the runbook removes the hesitation.
• Automate the runbook. Any step that can be scripted, should be. The runbook eventually becomes the automation.
• Track runbook coverage. Every alert that fires without a linked runbook is a gap; treat coverage as an operational KPI.
• Review after every use. The engineer who just ran the runbook is the best person to correct it.

Key Takeaways

• Runbooks compress on-call cognitive load from "reconstruct from memory" to "follow the procedure."
• One runbook per alert, linked from the alert message, versioned with the code.
• Copy-paste commands beat prose descriptions at 03:00.
• The best runbooks are written by the engineer who just handled the incident.
• Runbook coverage is an operational KPI, not an afterthought.

Related Concepts

Interlocks with Incident Management, Blameless Post-Mortem, Observability, SLIs, and Chaos Engineering. Runbook templates and coverage-tracking guides at PMMilestone.org.