How to design an incident escalation policy in 4 steps
Build an on-call escalation policy that routes incidents by impact, defines timers and handoffs, and measures acknowledgment and recovery separately.
The checkout error rate spikes during your shift. You acknowledge the incident and start looking at the gateway, but the next thing you need is someone who knows the payment integration. That calls for a different route from a page that has gone unanswered.
An escalation policy gives the team both routes. It tells the system what to do when nobody takes ownership and tells responders how to get help once somebody has.
What an incident escalation policy does
An incident escalation widens the response. You might notify the backup because the primary hasn't responded, bring in a service owner, or ask someone to coordinate several teams. The policy sets the condition, the recipient, and the timing. For a handoff, it also helps to say who owns the incident while the new responder is joining.
Track acknowledgment and recovery separately. Acknowledgment tells the team someone has picked up the work. It can stop an acknowledgment-based escalation even when users are still affected. PagerDuty incident states
Automatic and manual escalation
An automatic escalation follows configured timers and states. The primary gets the first page, and the backup gets the next one if the incident still qualifies. Check whether your tool advances the policy, adds a recipient, or reassigns the incident. Check whether it repeats, and which state stops the repeat. PagerDuty, for instance, pauses escalation on acknowledgment and can resume it when a configured acknowledgment timeout expires. PagerDuty escalation policies
A manual escalation starts with a responder's judgment. If checkout is failing and you can see a dependency outside your team's control, ask the owning team for help while you continue coordinating. A notification sent to that team still needs a response or an explicit transfer of ownership.
Step 1. Map the alerts that need a response
Start with the monitoring rules that can create incidents. Write down what each signal tells you about user impact, then decide which team should own the first response. A certificate expiry reminder and a failed TLS check on a production entry point should take different routes. Clients that validate an invalid certificate may fail to connect, while an expiry reminder usually leaves time to renew.
| Signal | What it can mean | Example response class |
|---|---|---|
| Invalid TLS certificate at the production entry point | Clients that validate the certificate may fail to connect | High when a critical entry point is affected |
| Sharp rise in errors on the payment API | Transactions may be failing | High when impact warrants it |
| Disk usage crosses 80% | Remaining time depends on the rate of growth | Low when headroom and growth allow |
| One noncritical scheduled job fails | The job may be safe to rerun | Low when impact is limited |
Start with high and low if that is enough to make routing decisions. Split out more severity levels when they change who needs to act or how quickly. Incident severity levels (Simplified Chinese)
Step 2. Set the escalation chain and timers
For a high-impact service, one example is to notify the primary at incident creation, the backup at minute 5 if the incident still needs a responder, and the team lead at minute 15 if the same condition still holds. These are example times measured from creation. Enter the corresponding per-step delays if your tool measures each wait from the previous level. Decide whether that path is covered around the clock. Lower-impact signals can route to a working-hours channel or a review queue.
In Guance, a Monitor can use an alert policy to escalate notifications when it repeatedly detects an anomaly of the same level. An incident record follows a separate on-call workflow in Incident Center. Match the on-call rule to the incident's tags or dimensions. The initial level can notify the current on-call responder. Later levels use elapsed time from incident creation along with the configured severity and state. If a level includes Working, it can notify more people even after someone has taken ownership. With auto-acknowledgment enabled, an incident that matches only one on-call person is assigned automatically and moves to Working. Check the response record if you need to know whether someone has acted. Guance alert policies · Guance on-call rules
Check that both the primary and fallback schedules have coverage at handoff. A rule that points to an empty shift cannot supply a responder. On-call schedule templates (Simplified Chinese)
Step 3. Decide when responders should escalate manually
A hierarchical escalation gets someone with the authority to coordinate resources or make a decision. A functional escalation gets someone who knows the dependency. The first may help when two teams need to change their deployment plans. The second is the faster path when the gateway configuration needs someone who can read it.
Put the service-to-owner map in the runbook. Send the new responder the symptom, impact, actions taken, and what you need next. Say whether you are requesting help or handing off. Until the handoff is accepted, keep an owner for the incident.
Step 4. Review response time and on-call load
Set the clock boundaries before comparing results. For MTTA, you might use the mean time from incident creation to first acknowledgment. MTTR can mean time to recovery or time to resolution. If you use recovery, end the clock when the service returns and track administrative closure separately. Compare similar incidents and keep unacknowledged incidents visible. The average of the acknowledged ones cannot tell you how many were missed.
Guance incident details separate the trigger, takeover, recovery, and closure timeline from cumulative statistics. A reopened incident shows the latest cycle in the current-cycle view. Check that you are comparing the same scope before using those timings to assess a rule change. Ask the on-call team which pages led to action and which woke a backup without changing the response. Guance incident details
Frequently asked questions
How many levels do we need?
Start with a primary and a backup. Add a level when it has a distinct job, such as coordinating a cross-team response. A longer chain does not tell you whether earlier notifications arrived.
How long should we wait before escalating?
Work back from the time an incident can go without an owner. Allow for coverage and the notification channel. If several people need to act immediately, notify them in parallel instead of making them wait for the next level.
What if people hesitate to ask for help?
Give them examples of when to escalate in the runbook, such as missing access or a failure in an unfamiliar dependency. In the review, discuss what earlier help would have changed.
Can an automatic rule page a lead for a minor issue overnight?
Yes. Check the severity mapping, the routing conditions, working hours, the default schedule, and the notification channel. A rule that still qualifies overnight will follow its configuration.
Further reading
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
