Escalation policies

Deep dive

How it works

Parts:

• Schedule — layers, handoff time, overrides; "who is on call at t".
• Policy — ordered levels (targets, wait) and a repeat count.
• Incident record — key, state, level, pass, count, urgency, context packet, generation.
• Timer store — pending (due, incident, generation) events; durable.
• Dedup index — open incidents by key.
• Notifier — SMS, push, call or chat; at-least-once, so pages must be repeat-safe.

1. Trigger with a dedup key. An open incident with that key gets count+1 and fresher context; else create it triggered.
2. Page the level: resolve each target (a user, or a schedule's on-call now), notify, arm an escalate timer at now + wait. Low urgency arms none.
3. Escalate when the timer fires and the state is still triggered: next level; past the last, another pass, or exhausted once repeats are spent.
4. Acknowledge: invalidate the escalate timer, arm an ack timeout. Snooze = acknowledge with a longer timeout.
5. Ack timeout or snooze end: back to triggered, re-notify.
6. Resolve: cancel timers, free the key.

         trigger(key)            ack / snooze
 (none) ─────────────▶ TRIGGERED ───────────────▶ ACKED ──── resolve ───▶ RESOLVED
          same key:     │  ▲  ▲    ack timeout /    │                        ▲
          count++       │  │  └─── snooze end ──────┘                        │
                        │  └─ timer: level+1; after the last level pass+1    │
                        └─ pass > repeats ─▶ EXHAUSTED (park, alert a manager)

Schedules. PagerDuty puts one user on call per schedule at a time. A layer rotates from a handoff time; the last layer overrides earlier ones (up to 60 layers). Restrictions can leave gaps: the incident then escalates to the next level, and if nothing in the policy is covered, none is created. Generate shifts with recurrence-rule-expansion.

Dedup keys. PagerDuty's dedup_key is up to 255 characters: a trigger joins the open alert with that key; after resolve it starts a new alert, and acknowledge or resolve events for the old one are dropped. Opsgenie's alias: one open alert per alias, count stops at 100. Key on the problem (service plus check, case id), not a timestamp. See idempotency-keys, alert-grouping.

Loops. PagerDuty's example repeats a policy three times; Opsgenie's repeat block has wait_interval and count. Cap it, and make the end state reach someone outside the rota, like a dead-letter-queue.

Urgency and hours. In PagerDuty, low-urgency incidents do not escalate automatically, and support hours set one urgency inside and another outside. Choose urgency at trigger time from severity and clock.

Durable timers. The escalate timer is the promise that the next person is tried. Held only in memory, a crash between "paged level 1" and "paged level 2" drops it silently: nobody is paged and nothing looks wrong. So store the due time with the incident in the same transaction as the state change, and rebuild the heap from open incidents at start. A restart may fire a timer twice, so handlers check state and generation (as the code does), as in a durable-job-queue. Run one scheduler under a lease.

Support-bot handoff. Evaluate triggers each turn, in priority order: the user asks for a human (an intent from intent-classification-slot-filling), a policy topic such as a refund over a limit, repeated failure, low confidence. The conversation id is the dedup key. The context packet holds a transcript summary, the bot's state (slots filled, actions tried, trigger reason) and a link to the full log, refreshed on each merge. Tiers escalate on SLA timers: agent, senior agent, lead. The SRE book's command handoff is the same idea: brief the successor.

Code

Minute-based simulation: weekly layers plus an override, three levels, BinaryHeap timers cancelled lazily by generation, dedup, ack, snooze, resolve, one repeat, and the bot trigger. The 5000 refund limit, 3 failures and 0.5 confidence are this example's choices.

use std::cmp::Reverse;
use std::collections::{BinaryHeap, HashMap};
type Min = u64; // simulated minutes; week 0 starts at t = 0
struct Layer { start: Min, shift: Min, members: Vec<&'static str>, active: (Min, Min) } // active = minute-of-week window
struct Schedule { layers: Vec<Layer>, overrides: Vec<(Min, Min, &'static str)> }
impl Schedule {
    /// Override first, else the last layer covering t; None = gap.
    fn on_call(&self, t: Min) -> Option<&'static str> {
        if let Some(o) = self.overrides.iter().find(|o| o.0 <= t && t < o.1) { return Some(o.2); }
        let w = t % 10080;
        self.layers.iter().rev().find(|l| t >= l.start && l.active.0 <= w && w < l.active.1)
            .map(|l| l.members[((t - l.start) / l.shift) as usize % l.members.len()])
    }
}
enum Target { User(&'static str), Sched(usize) }
struct Policy { levels: Vec<(Vec<Target>, Min)>, repeats: u32 } // (targets, wait)
#[derive(PartialEq, Debug, Clone, Copy)]
enum St { Triggered, Acked, Resolved, Exhausted }
struct Inc { key: String, st: St, level: usize, pass: u32, count: u32, urgent: bool, gen: u64, ctx: String }
struct Sys {
    pol: Policy, scheds: Vec<Schedule>, incs: Vec<Inc>, open: HashMap<String, usize>,
    q: BinaryHeap<Reverse<(Min, u64, usize, u64, u8)>>, // (due, seq, incident, generation, 0=escalate 1=ack-timeout)
    seq: u64, log: Vec<(Min, usize, String)>,           // (time, incident, who)
}
impl Sys {
    fn at(&mut self, due: Min, i: usize, kind: u8) {
        self.seq += 1; self.q.push(Reverse((due, self.seq, i, self.incs[i].gen, kind)));
    }
    /// Dedup: an open incident with this key absorbs the alert.
    fn trigger(&mut self, t: Min, key: &str, ctx: &str, urgent: bool) -> usize {
        if let Some(&i) = self.open.get(key) {
            self.incs[i].count += 1;
            self.incs[i].ctx = ctx.into(); // newest context wins
            return i;
        }
        let i = self.incs.len();
        self.incs.push(Inc { key: key.into(), st: St::Triggered, level: 0, pass: 0, count: 1, urgent, gen: 0, ctx: ctx.into() });
        self.open.insert(key.into(), i);
        self.page(t, i);
        i
    }
    fn page(&mut self, t: Min, i: usize) {
        let (targets, wait) = &self.pol.levels[self.incs[i].level];
        let wait = *wait;
        for tg in targets {
            let who = match tg { Target::User(u) => Some(*u), Target::Sched(s) => self.scheds[*s].on_call(t) };
            if let Some(w) = who { self.log.push((t, i, w.to_string())); }
        }
        if self.incs[i].urgent { self.at(t + wait, i, 0); } // low urgency: no escalation
    }
    /// `hold` = ack timeout, or a longer snooze.
    fn ack(&mut self, t: Min, i: usize, hold: Min) {
        if self.incs[i].st != St::Triggered { return; }
        self.incs[i].st = St::Acked; self.incs[i].gen += 1; // invalidates the escalate timer
        self.at(t + hold, i, 1);
    }
    fn resolve(&mut self, i: usize) {
        self.incs[i].st = St::Resolved; self.incs[i].gen += 1;
        self.open.remove(&self.incs[i].key);
    }
    fn run_until(&mut self, t: Min) {
        while let Some(&Reverse((due, _, i, gen, kind))) = self.q.peek() {
            if due > t { break; }
            self.q.pop();
            if self.incs[i].gen != gen { continue; } // stale timer
            if kind == 1 && self.incs[i].st == St::Acked { // re-trigger
                self.incs[i].st = St::Triggered;
                self.page(due, i);
            } else if kind == 0 && self.incs[i].st == St::Triggered {
                self.incs[i].level += 1;
                if self.incs[i].level == self.pol.levels.len() { // end of a pass
                    self.incs[i].pass += 1; self.incs[i].level = 0;
                    if self.incs[i].pass > self.pol.repeats { self.incs[i].st = St::Exhausted; continue; }
                }
                self.page(due, i);
            }
        }
    }
    fn paged(&self, i: usize) -> Vec<(Min, &str)> { self.log.iter().filter(|e| e.1 == i).map(|e| (e.0, e.2.as_str())).collect() }
}
#[derive(Debug, PartialEq)]
enum Why { AskedHuman, Policy, RepeatedFailure, LowConfidence }
fn handoff(conf: f32, text: &str, refund: u64, fails: u32) -> Option<Why> { // example thresholds
    if text.to_lowercase().contains("human") { Some(Why::AskedHuman) }
    else if refund > 5000 { Some(Why::Policy) }
    else if fails >= 3 { Some(Why::RepeatedFailure) }
    else if conf < 0.5 { Some(Why::LowConfidence) }
    else { None }
}
fn main() {
    let rot = |m: Vec<&'static str>, a| Layer { start: 0, shift: 10080, members: m, active: a };
    let oncall = Schedule { layers: vec![rot(vec!["ana", "ben"], (0, 10080)), rot(vec!["cy"], (540, 1020))],
                            overrides: vec![(10230, 10280, "dee")] };
    assert_eq!([0, 600, 10130, 10250].map(|t| oncall.on_call(t)), [Some("ana"), Some("cy"), Some("ben"), Some("dee")]);
    let pol = Policy { levels: vec![(vec![Target::Sched(0)], 15), (vec![Target::User("lee")], 30), (vec![Target::User("vp")], 60)],
                       repeats: 1 };
    let mut s = Sys { pol, scheds: vec![oncall], incs: vec![], open: HashMap::new(), q: BinaryHeap::new(), seq: 0, log: vec![] };
    let a = s.trigger(10, "case-7", "refund 90, step 2", true);
    assert_eq!(s.trigger(12, "case-7", "refund 90, step 3", true), a); // merged
    assert_eq!(s.incs[a].count, 2);
    s.run_until(24); assert_eq!(s.paged(a), [(10, "ana")]);  s.run_until(25); assert_eq!(s.paged(a), [(10, "ana"), (25, "lee")]);  // level 2
    s.ack(26, a, 30); s.run_until(55); assert_eq!(s.incs[a].st, St::Acked);
    s.run_until(56); assert_eq!(s.incs[a].st, St::Triggered);             // 26 + 30
    assert_eq!(s.paged(a).last(), Some(&(56, "lee")));                    // same level again
    s.ack(57, a, 100); s.run_until(156); assert_eq!(s.incs[a].st, St::Acked); // snooze
    s.run_until(157); assert_eq!(s.incs[a].st, St::Triggered);
    s.resolve(a); let n = s.paged(a).len(); s.run_until(10_000);
    assert_eq!((s.incs[a].st, s.paged(a).len()), (St::Resolved, n)); // resolve cancels timers
    let b = s.trigger(1100, "case-7", "again", true);                // key free: new incident
    assert_ne!(a, b); s.run_until(2000);
    let who: Vec<_> = s.paged(b).iter().map(|e| e.1).collect();
    assert_eq!(who, ["ana", "lee", "vp", "ana", "lee", "vp"]);       // pass + 1 repeat
    assert_eq!(s.incs[b].st, St::Exhausted);
    let c = s.trigger(5000, "faq-1", "low urgency", false); s.run_until(9000);
    assert_eq!(s.paged(c).len(), 1);                               // no escalation
    let h = [handoff(0.9, "a human please", 0, 0), handoff(0.9, "refund", 12000, 0), handoff(0.9, "again", 0, 3), handoff(0.3, "hmm", 0, 0), handoff(0.9, "thanks", 100, 0)];
    assert_eq!(h, [Some(Why::AskedHuman), Some(Why::Policy), Some(Why::RepeatedFailure), Some(Why::LowConfidence), None]);
    println!("ok");
}

Compiled with rustc -O and run; every assert passes.

Complexity and constants

• A timer event costs O(log n) for the heap plus O(levels) to resolve targets; dedup is one hash lookup.
• Measured on this Mac (Apple M4 Pro): 200,000 incidents, 1,200,000 pages in about 171 ms (about 143 ns per page), in memory.

Efficiency

• Cost: about 143 ns per page, memory only (measured on this Mac (Apple M4 Pro)). In production the bound is I/O: the durable write and the notification provider.
• Levers:
  • Dedup before paging — one page per problem — Alertmanager holds the first notification for group_wait, default 30s.
  • Lazy cancellation by generation — no heap search on ack or resolve — always.
  • Timing wheel instead of heap — O(1) per timer — millions of timers (hierarchical-timing-wheel).
  • Fewer, better alerts (time-series-anomaly-detection) — the SRE book gives two incidents per 12-hour shift as the limit.
• Naive → standard → state of the art:
  • A sleeping thread per incident · lost on restart.
  • Row with next_fire_at plus an indexed poller · one range scan per tick, survives crashes.
  • Row as truth, heap or wheel rebuilt at start, one leased scheduler · O(log n) or O(1) per timer, survives crashes.

Implementation notes

• Compute on-call at page time so a new override applies to pending escalations.
• Guard ack and every timer fire with the expected state, so races lose quietly.
• Keep notifier retries (exponential-backoff-jitter) apart from escalation.

Pitfalls

• In-memory timers only: a restart loses every pending escalation.
• Timestamped dedup keys: an alert storm becomes a page storm.
• No loop cap pages forever; a cap with no final state ends silently.
• Coverage gaps: the last level must always resolve to a person.

Variants and alternatives

In production

PagerDuty, Opsgenie, GoAlert, Grafana OnCall OSS; Alertmanager for grouping and repeats. Hospital alarm and security-operations tiers use the same pattern (no source read).

Implementations

Sources

• PagerDuty, Escalation Policies and Schedules. docs · Schedule Basics. docs
• PagerDuty, Incidents docs, Configurable Service Settings docs, Snooze an incident docs, Send an alert event docs
• Atlassian, What is alert de-duplication? (Opsgenie). docs · opsgenie_escalation resource. docs
• Prometheus, Alertmanager configuration. docs
• Beyer et al., Site Reliability Engineering, O'Reilly 2016: ch. 11 Being On-Call online, ch. 14 Managing Incidents online.