On-Call Routine

Lead Summary

On-call routines are the operational structures that allow engineering teams to detect, respond to, and recover from production incidents around the clock. At their core, they answer a simple question: who is responsible for the service right now, and what happens when something breaks? In practice, the answer involves rotation schedules, escalation policies, alert instrumentation, postmortem processes, and — increasingly — AI-assisted triage.

When designed well, on-call routines preserve service reliability without burning out the engineers who maintain it. When designed poorly, they become a leading cause of attrition: 65% of engineers report burnout in the past year, and on-call stress is consistently cited as a major contributing factor. The challenge is structural, not personal — sustainable on-call requires deliberate system design, not just individual resilience.

Rotation Design & Team Sizing

Minimum viable team size

Sustainable 24/7 on-call for a single site requires a minimum of eight engineers assuming week-long shifts with a primary and secondary responder on simultaneously, with each engineer serving roughly once per month. Adding one more provides a buffer against attrition, bringing the practical minimum to nine per site. Multi-site follow-the-sun configurations can reduce that to approximately five to six engineers per regional site by distributing coverage across time zones.

Each engineer should serve on-call at least once every six weeks to maintain procedural familiarity — rotations spaced further apart increase the likelihood of knowledge gaps during incidents.

Teams smaller than the eight-engineer ideal must adapt. Smaller teams use shorter shifts (8 hours rather than 12) to offset more frequent rotation cycles, and teams of three or fewer often fall back to daily alternation or informal ad-hoc coverage rather than a formal rotation structure.

Shift duration

Google SRE recommends 12-hour shifts as the practical middle ground for on-call rotations: long enough to minimize handoff overhead and context switching, short enough to manage fatigue and preserve decision-making quality. Overnight shifts carry particular burnout and error-rate risks. Operations-heavy or high-alert environments favor 12-hour shifts, while high-context services with lower alert volume often use weekly rotations where continuity is more valuable than frequent handoffs.

Incident volume limits

Google SRE establishes a sustainable cognitive baseline of 2–3 actionable incidents per 8-12 hour shift. This limit gives engineers time to handle incidents accurately, complete service restoration, conduct postmortems, and recover between pages. Teams consistently handling 8–10 incidents per shift do not primarily have an on-call staffing problem — they have an alerting and classification problem.

Primary and secondary structure

Industry best practice pairs a primary and secondary responder for every shift. The primary is first to respond with a 5-minute SLA, handling initial triage and incident execution. The secondary backs up with a 15-minute response window, providing support during major incidents and domain expertise. Escalation policies automatically engage the secondary if the primary is unavailable — due to illness, connectivity loss, or another incident — preventing a single missed page from creating a service gap.

Time allocation

Google's SRE model caps on-call time at 25% of total working time, with at least 50% reserved for engineering projects that improve stability and operability. The remaining 25% covers other operational work. This structure prevents SRE teams from becoming purely reactive operations groups: without a protected engineering budget, teams cannot address the systemic issues that drive incident volume in the first place.

Handoffs & Escalation Policies

Structured handoffs

Effective rotation transitions require a dedicated 30-minute weekly handoff meeting between outgoing and incoming engineers. The agenda covers active incidents, silenced alerts, upcoming risky changes, and on-call tooling status. For multi-region follow-the-sun rotations, written handoff documentation should include short incident summaries, current status, and next actions to guide the incoming region without requiring synchronous overlap.

Escalation policy design

Effective escalation policies account for incident severity, duration, scope, and time of day. A SEV-1 may escalate to director-level after 4 hours; a SEV-3 after 8 hours or on manual request. Policies implement two distinct mechanisms: hierarchical escalation moves incidents up the seniority chain (junior to senior engineer), while functional escalation routes them to domain experts regardless of rank. Most mature teams combine both within a single policy.

Time-based escalation is the standard automated mechanism that moves incidents to the next tier when a responder fails to acknowledge within a defined timeout — commonly 5, 10, or 15 minutes depending on severity. Automation ensures this happens reliably even under stress or fatigue, without depending on a tired engineer to manually judge when to escalate.

Round-robin scheduling distributes incidents equitably across team members in a predetermined sequence, preventing workload concentration on specific individuals during periods of high incident volume.

Escalation documentation and testing

Escalation criteria must be clearly documented and communicated: explicit ladders, timelines, decision points, and role responsibilities for each tier. Industry best practice recommends quarterly testing — injecting simulated failures to verify that triggers fire correctly, auto-escalation activates, and communication channels remain functional. Policies should also be reviewed after major organizational changes, significant incidents, or new system implementations.

Alert Hygiene

Alert fatigue is not a willpower issue — it is a cognitive overload mechanism. When the stream of alerts exceeds an engineer's ability to interpret them, attention declines, reaction times slow, and critical issues become harder to distinguish from noise. The solution is instrumentation design, not training.

"If your team is consistently seeing 8–10 incidents per shift, you don't have an on-call problem — you have an alerting problem."

The scale of the problem

62% of on-call engineers report having ignored a critical alert because it was buried in noise. Teams receiving more than 40 alerts per shift experience 3x higher mean time to resolution compared to teams receiving fewer than 10. Alert noise is a direct MTTR multiplier.

SLO-based alerting

SLO-based alerting is the primary instrumentation strategy for reducing noise while maintaining reliability. Instead of paging on resource exhaustion or latency spikes, alerts fire when the service is actually burning its error budget — delivering pages that correlate with user-visible degradation. Infrastructure metrics should inform dashboards and diagnostics; they should not be primary on-call triggers.

Multi-window, multi-burn-rate alerting

Multi-window, multi-burn-rate alerting evaluates conditions across different time windows simultaneously, catching both sudden and gradual budget burns without producing false alarms from transient spikes. The Google SRE Workbook identifies this as the most appropriate technique for defending SLOs while protecting on-call engineers from noise.

Occupational Health

Burnout as a structural outcome

On-call responsibilities are a documented driver of engineering burnout and attrition. The anxiety of carrying a pager — even during quiet periods — disrupts sleep and creates persistent psychological stress. Burned-out on-call engineers make more mistakes during critical incidents, take longer to resolve issues, and eventually leave organizations, concentrating the remaining on-call load on fewer people.

Circadian disruption

Irregular and night-shift on-call schedules disrupt the circadian system, creating misalignment between the body's internal clock and the external light-dark cycle. Peer-reviewed research documents both immediate and long-term negative health outcomes: metabolic disorders, gastrointestinal problems, elevated cancer risk, cardiovascular issues, and mental health deterioration. Crucially, shift workers do not physiologically acclimate to irregular schedules — the internal desynchronization compounds rather than resolving over time.

Postmortem Culture

Why postmortems exist

Postmortems are structured reviews conducted after incidents to extract learning and prevent recurrence. Their value is only realized when the culture around them enables honest, objective communication — which requires psychological safety as a precondition.

Blameless culture

Blameless postmortems focus on system failures and contributing factors rather than individual fault. The underlying philosophy assumes that everyone involved acted with good intentions on the best information available, shifting accountability from people to systems. When engineers fear being blamed for mistakes, they hesitate to speak up during incidents, increasing mean time to acknowledge and mean time to resolve.

When human error contributed to an incident, facilitators explicitly redirect the discussion from "Person X should have checked the configuration" to "What about our process made it easy to miss that configuration check?" Every human error is treated as a signal that the system allowed or enabled the mistake.

Senior leadership participation is essential for sustaining this culture. When engineering leaders actively model blameless behavior and redirect blame-focused conversations toward systemic improvements, they signal that the organization prioritizes learning over punishment.

Timing and participants

Postmortems should be held within 24–72 hours of incident resolution, ideally within 48. This window balances two competing needs: sufficient emotional distance from the incident to enable objective analysis, while keeping details fresh enough for accurate timelines. Reviews held too late lose the granular recollection of decision points and bottlenecks.

Effective postmortems include a cross-functional participant set: the incident commander, engineers who handled the response, service owners for affected systems, product managers who can assess business impact, and customer support or communications representatives. This composition ensures technical accuracy, business context, and customer perspective simultaneously.

Action items

Postmortems regularly produce clear alignment but fail to actualize the learning — action items silently disappear or are deferred indefinitely. This is one of the most common failure modes in postmortem processes. The mitigation is structural:

Action items must live in the team's actual task management system — not in postmortem documents or spreadsheets — where they are visible in the team's daily workflow. Each action item requires five elements: a named individual owner, a verifiable action verb, a specific measurable outcome, a clear deadline, and residence in the primary task tracker. Action items lacking these elements tend to be vague intentions that slip through without completion.

AI Augmentation

AI tooling is increasingly present in on-call workflows, with meaningful demonstrated benefits and documented failure modes that shape how it should be deployed.

What AI does well

Alert noise reduction. AI-powered filtering and suppression can reduce alert noise by up to 70% by identifying and suppressing known noise patterns — flapping services, routine informational alerts, and other low-value events. AI-driven alert correlation automatically groups related alerts across services into unified incident presentations, so engineers receive a single deduplicated incident rather than dozens of individual notifications. Automated correlation compresses raw alerts into practical incidents at approximately 95% efficiency.

Investigation speed. Autonomous SRE agents complete triage and root cause analysis in minutes rather than hours. AWS DevOps Agent autonomously detects and diagnoses incidents in approximately 4 minutes by systematically testing hypotheses across logs, metrics, and traces. These agents form parallel hypotheses using the same tools human engineers use — querying metrics, checking logs, analyzing deployment data — rather than producing a single linear diagnosis.

Documentation automation. AI agents can draft incident channels, create initial timelines, and suggest remediation steps from runbooks, shifting on-call work from manual triage and documentation toward reviewing AI-prepared context and making final decisions.

MTTR impact. AI-driven incident automation reduces MTTR by approximately 40% in documented production cases, with enterprise organizations reporting reductions of 40–60% through AI-driven observability. Teams cut MTTR by 25% within 90 days in some reported deployments.

Grounding AI in real data

Retrieval-Augmented Generation (RAG) over telemetry, runbooks, and documentation mitigates AI hallucination by grounding AI recommendations in actual log events and verified procedures. RAG enables AI systems to explain anomalies with references to specific data rather than generating unsupported diagnoses.

Risks and failure modes

Hallucination. AI systems produce plausible but false outputs that appear authoritative, leading responders to accept them without verification. Tool-calling failure rates run at 3–15% in production. In 2024, 47% of enterprise AI users made at least one major business decision based on hallucinated content.

Agent loops. Autonomous agents can enter problematic recursion patterns: multi-agent loops where agents call each other without a shared stop condition, hallucination cascades where invented information triggers corrupting downstream actions, and infinite refinement loops that consume compute resources without converging. Real-world incidents document agents repeating operations for 11 days and spending thousands of dollars on phantom operations.

Deployment approach

Operational best practice follows a graduated, human-in-the-loop trust model. Most teams begin with AI in an advisory capacity — presenting recommendations that require human approval before execution — and expand autonomy as the system demonstrates consistent accuracy. This staged approach builds organizational confidence while limiting exposure to hallucination and loop risks.