On-Call Rotation

Definition

An On-Call Rotation is the arrangement by which a team's engineers take turns to be reachable outside normal working hours to respond to production incidents. It is one of the defining practices of modern software operations, and one of the surest indicators of an engineering culture's maturity. A humane rotation attracts and keeps senior engineers; an inhumane one drives them away and quietly corrodes reliability.

What a Good Rotation Looks Like

• A pager duty schedule with primary and secondary responders, published weeks in advance.
• A defined severity model with clear paging criteria — no paging below Sev-2.
• Working runbooks for every alert that pages.
• Alert budgets — when a team blows the budget, alerts get tuned, not tolerated.
• Post-incident review for every page, with time-in-lieu for the responder.
• Rotation cadence sustainable by the team — weekly for larger teams, longer intervals for smaller.
• Compensation model — payment, time in lieu, or both — documented and honoured.

Why It Matters

On-call touches four things at once: reliability, sustainability, culture and hiring. A poorly designed rotation degrades reliability because tired responders make bad choices under pressure. It burns out engineers, who leave. It signals to the ones who remain that operational pain is not a real concern. And it eventually turns up in the recruitment funnel, where senior candidates ask about on-call in the second interview and disqualify companies whose answer is bad.

Real-World Example

A payments platform ran a six-person rotation with 24/7 primary on-call, one week at a time. Alert volume was around 30 pages per week per rotation, mostly false positives from a threshold-based monitor that had never been tuned. The team's most experienced engineer gave notice, citing on-call. In the exit conversation they estimated they had lost 3 hours of sleep per night during their on-call weeks for three years. The engineering director made three changes: cut the primary rotation to five days with a Sunday handover, invested two weeks of engineering time to move the noisy monitor to a symptom-based alert, and introduced a hard rule that any alert firing more than twice per rotation had to be tuned or deleted the following week. Alert volume dropped from 30 to 4 per rotation inside a quarter. Two other engineers who had been quietly considering leaving stayed.

How to Design a Rotation

1. Start with the alerts, not the schedule. A rotation with 30 pages a week is broken before you begin.
2. Publish paging criteria. Sev-1 pages; Sev-2 pages during business hours; anything below is a ticket.
3. Build the runbook for every alert that pages. No runbook, no alert.
4. Choose the cadence honestly. Weekly is common; on very small teams, split the day into 12-hour segments and rotate two people daily.
5. Provide a secondary responder so the primary is not on their own at 3am.
6. Compensate. Time in lieu, pay, or both, but not "it's part of the job."
7. Review each page the next working day — was it real, was the runbook adequate, does the alert need to be tuned.

Practical Lessons Learned

• Alert fatigue kills rotations. Every noisy alert erodes trust in every other alert; fix them relentlessly.
• Symptom-based alerting outperforms cause-based alerting. Page on user-visible impact, not internal state.
• Runbooks are load-bearing. A responder at 3am on their second call in an hour cannot reconstruct the mental model — the runbook must.
• Time in lieu must be enforced. Otherwise it drifts into never being taken.
• Junior engineers should join secondary before primary. A first shift alone at 3am is a bad first experience.
• Rotations don't survive across time zones without deliberate design. Follow-the-sun coverage is a project of its own, not a policy tweak.

Expert Tips

• Track pages per rotation as a first-class metric. If it is going up, reliability is going down whether or not incidents are.
• Publish the rotation eight weeks ahead. Engineers plan their lives; a two-week horizon is not enough.
• Allow swaps by mutual agreement with a lightweight sign-off.
• Never page for a business decision. Missed revenue is not a 3am problem; missing SLOs is.
• Have an escalation policy that reaches management after two failed responder pages, not after five.

Common Mistakes

• Building the rotation before fixing the alerts.
• Paging on cause instead of symptom.
• No runbooks, or stale ones.
• Time in lieu granted in principle, never in practice.
• Junior engineers assigned primary on their first month.
• Swap requests treated as management decisions rather than peer agreements.
• Rotation designed for time-zone convenience of the leadership, not sustainability of the team.

Key Takeaways

• A rotation is only as humane as its alert budget.
• Runbooks and severity models are prerequisites, not optional.
• Compensation must be real, not rhetorical.
• Every page gets reviewed the next working day.
• Sustainability is a leading indicator of reliability, not a trade-off against it.

Related Concepts

Interlocks with Incident Management, Runbooks, Blameless Postmortems, SLOs, Error Budgets and Observability. Templates at PMMilestone.org.