Supervision trees

Deep dive

How it works

Parts:

• supervisor — starts children in order, watches them, applies strategy and intensity; no business logic
• child spec — start function; restart type (permanent: always, transient: after abnormal exit only, temporary: never); shutdown timeout (OTP default 5000 ms for workers, infinity for supervisors); worker or supervisor
• strategy — one_for_one restarts the failed child, one_for_all all children, rest_for_one the failed one and those started after it
• intensity and period — at most MaxR restarts within MaxT; one more and the supervisor stops its children and exits, a failure for its parent
• failure signal — a link or monitor delivers an exit notice with a reason (Armstrong: a linked process that traps exits gets {'EXIT', Pid, Why})

Operations:

1. start each child in spec order, one at a time.
2. watch: block on exit notices; ignore those from a generation already replaced.
3. decide: apply the restart type; if due, record the time and drop records older than MaxT.
4. escalate: more than MaxR records: stop all children in reverse order and exit with an error.
5. restart: stop the affected children in reverse start order (shutdown request, wait up to the timeout, then kill), start them again in order with fresh state.
6. shutdown: stop all children in reverse start order.

                  root supervisor            (rest_for_one: db first, then the rest)
                 /       |        \
            db-pool   session-sup   metrics         start order →
                      (one_for_one)                 stop order ←
                     /   |   |   \
                   s1   s2  s3  s4   one child per connection
 s2 crashes → only s2 restarts. Too many crashes in MaxT → session-sup exits;
 root restarts it and everything after it, or gives up in turn.

Design the tree by dependency: children others need go first under rest_for_one, independent ones under one_for_one, one_for_all only for a group that is meaningless when incomplete. Keep critical state near the root: Armstrong's error kernel is the part that must be correct (Akka's docs: "conscious application of the error kernel pattern"). Leaves hold cheap, rebuildable state.

Code

Supervisor thread plus child threads; the exit notice follows catch_unwind. Crashes come from messages, not timing, so asserts are deterministic. Mailbox survives restarts, state does not.

use std::{collections::VecDeque, panic::{catch_unwind, AssertUnwindSafe}};
use std::sync::{mpsc::{channel, Receiver, Sender}, Arc, Mutex};
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant};
#[derive(Clone, Copy)] pub enum Strategy { OneForOne, OneForAll, RestForOne }
#[derive(Clone, Copy, PartialEq)] pub enum Restart { Permanent, Transient, Temporary } // Temporary: never restarted
pub struct Spec {
    pub restart: Restart,
    pub start: Arc<dyn Fn(u64) + Send + Sync>, // child body; argument = generation
    pub stop: Box<dyn Fn(u64) + Send>,         // ask generation g to exit (a message, not a kill)
}
pub enum Event { Exit { idx: usize, generation: u64, panicked: bool }, Shutdown }
#[derive(Debug, PartialEq)] pub struct Escalate; // intensity exceeded: the parent must act
pub struct Supervisor {
    pub strategy: Strategy, pub max_restarts: usize, pub period: Duration,
    pub specs: Vec<Spec>, pub events: Receiver<Event>, pub tx: Sender<Event>,
}
impl Supervisor {
    fn spawn(&self, idx: usize, generation: u64) -> JoinHandle<()> {
        let (body, tx) = (self.specs[idx].start.clone(), self.tx.clone());
        thread::spawn(move || { // the monitor: report how the child ended
            let panicked = catch_unwind(AssertUnwindSafe(|| body(generation))).is_err();
            let _ = tx.send(Event::Exit { idx, generation, panicked });
        })
    }
    /// Reverse start order; `skip` = the child that already ended.
    fn stop(&self, alive: &mut [Option<JoinHandle<()>>], gen: &[u64], r: std::ops::Range<usize>, skip: Option<usize>) {
        for i in r.rev() {
            if let Some(h) = alive[i].take() {
                if Some(i) != skip { (self.specs[i].stop)(gen[i]); }
                let _ = h.join();
            }
        }
    }
    pub fn run(self) -> Result<(), Escalate> {
        let (n, mut gen) = (self.specs.len(), vec![0u64; self.specs.len()]);
        let mut alive: Vec<_> = (0..n).map(|i| Some(self.spawn(i, 0))).collect(); // start order
        let mut recent: VecDeque<Instant> = VecDeque::new();
        loop {
            match self.events.recv().unwrap() {
                Event::Shutdown => { self.stop(&mut alive, &gen, 0..n, None); return Ok(()); }
                Event::Exit { idx, generation, panicked } => {
                    if generation != gen[idx] { continue; } // stale notice from a child we stopped
                    let again = match self.specs[idx].restart { Restart::Permanent => true, Restart::Transient => panicked, _ => false };
                    if !again { let _ = alive[idx].take().map(|h| h.join()); continue; }
                    let now = Instant::now();
                    recent.push_back(now);
                    while now.duration_since(recent[0]) > self.period { recent.pop_front(); }
                    if recent.len() > self.max_restarts { // give up; the parent restarts or exits
                        self.stop(&mut alive, &gen, 0..n, Some(idx));
                        return Err(Escalate);
                    }
                    let r = match self.strategy { Strategy::OneForOne => idx..idx + 1, Strategy::OneForAll => 0..n, Strategy::RestForOne => idx..n };
                    self.stop(&mut alive, &gen, r.clone(), Some(idx));
                    for i in r {
                        gen[i] += 1;
                        if self.specs[i].restart != Restart::Temporary { alive[i] = Some(self.spawn(i, gen[i])); }
                    }
                }
            }
        }
    }
}
// test workers: state `n` lives in the thread, so a restart resets it
enum Cmd { Inc, Crash, Get(Sender<u64>), Stop(u64) }
fn worker() -> (Spec, Sender<Cmd>) {
    let (tx, rx) = channel::<Cmd>();
    let (rx, stopper) = (Arc::new(Mutex::new(rx)), tx.clone()); // mailbox outlives restarts
    let start = Arc::new(move |g: u64| {
        let rx = rx.lock().unwrap_or_else(|e| e.into_inner()); // a panic poisons the lock
        let mut n = 0u64;
        while let Ok(c) = rx.recv() {
            match c {
                Cmd::Inc => n += 1,
                Cmd::Get(r) => { let _ = r.send(n); }
                Cmd::Crash => panic!("boom"),
                Cmd::Stop(s) => if s == g { return }, // ignore Stops meant for older generations
            }
        }
    });
    (Spec { restart: Restart::Permanent, start, stop: Box::new(move |g| { let _ = stopper.send(Cmd::Stop(g)); }) }, tx)
}
fn get(t: &Sender<Cmd>) -> u64 { let (r, w) = channel(); t.send(Cmd::Get(r)).unwrap(); w.recv_timeout(Duration::from_secs(5)).unwrap() }
fn boot(strategy: Strategy, max_restarts: usize) -> (Sender<Event>, Sender<Cmd>, Sender<Cmd>, JoinHandle<Result<(), Escalate>>) {
    let ((a, ta), (b, tb), (tx, events)) = (worker(), worker(), channel());
    let sup = Supervisor { strategy, max_restarts, period: Duration::from_secs(60), specs: vec![a, b], events, tx: tx.clone() };
    (tx, ta, tb, thread::spawn(move || sup.run()))
}
fn main() {
    std::panic::set_hook(Box::new(|_| {})); // silence panics
    for _ in 0..200 { // repeat to shake out races
        for (s, a_after, b_after) in [(Strategy::OneForOne, 0, 3), (Strategy::OneForAll, 0, 0), (Strategy::RestForOne, 0, 0)] {
            let (ev, a, b, h) = boot(s, 5); // 5 restarts allowed per 60 s
            for _ in 0..2 { a.send(Cmd::Inc).unwrap(); b.send(Cmd::Inc).unwrap(); }
            b.send(Cmd::Inc).unwrap(); // a: 2, b: 3
            a.send(Cmd::Crash).unwrap(); // crash the first child
            assert_eq!((get(&a), get(&b)), (a_after, b_after)); // fresh state vs kept state
            if let Strategy::RestForOne = s { // crash the last child: earlier siblings keep their state
                a.send(Cmd::Inc).unwrap(); b.send(Cmd::Crash).unwrap();
                assert_eq!((get(&a), get(&b)), (1, 0));
            }
            ev.send(Event::Shutdown).unwrap();
            assert_eq!(h.join().unwrap(), Ok(()));
        }
        let (_ev, a, _b, h) = boot(Strategy::OneForOne, 2); // intensity: 2 restarts allowed in 60 s
        for _ in 0..2 { a.send(Cmd::Crash).unwrap(); assert_eq!(get(&a), 0); }
        a.send(Cmd::Crash).unwrap(); // the 3rd crash exceeds it
        assert_eq!(h.join().unwrap(), Err(Escalate));
    }
    let (ev, a, _b, h) = boot(Strategy::OneForOne, usize::MAX); // latency: Crash sent -> replacement answers
    let mut us: Vec<u128> = (0..1000).map(|_| { let t = Instant::now(); a.send(Cmd::Crash).unwrap(); get(&a); t.elapsed().as_micros() }).collect();
    us.sort();
    println!("restart latency us: min {} median {} p99 {}", us[0], us[500], us[990]);
    ev.send(Event::Shutdown).unwrap(); h.join().unwrap().unwrap();
}

Asserts: one_for_one restarts the crashed worker at 0 while its sibling keeps 3; one_for_all resets both; rest_for_one resets only later children; a third crash in the window returns Err(Escalate). Apple M4 Pro, release build, measured by me: 15 runs of 200 rounds, all asserts held. Restart latency (Crash sent until the replacement answers Get, 1,000 crashes per run): min 28-33 µs; median 31-41 µs and p99 50-470 µs on a quiet machine (15 runs), median 50-80 µs and p99 0.1-1 ms at load average about 9 (10 later runs). No backoff or kill timeout: std cannot kill a thread.

Complexity and constants

• Restart event: O(k) for k affected children; intensity window O(1) amortized.
• Cost drivers: child init, sequential stop of siblings (worst case the sum of their timeouts), crash logging.

Efficiency

• Cost: one restart, thread children, empty init, M4 Pro, measured here: min about 30 µs, median 30-80 µs depending on machine load, p99 up to about 1 ms. Real cost is the child's init and lost work. Bound: init and sibling stop time.
• Levers:
  • one_for_one instead of one_for_all — siblings keep state (OTP docs) — independent children
  • DynamicSupervisor for many identical children — no ordering, stops them concurrently (Elixir docs) — child per connection or device
  • backoff — Kubernetes: 10 s, 20 s, 40 s … capped at 300 s, reset after 10 minutes of success (k8s docs) — crash loops
  • MaxR and MaxT as a rate — 5 in 30 s allows at most one restart per 6 s sustained (OTP docs) — bounds logs and load
• Naive → standard → state of the art:
  • catch everything in the worker, repair in place · corrupt state survives, bugs hide
  • supervisor, immediate restart, intensity limit (this page) · tens of µs here; a crash loop ends only by escalation
  • tree plus backoff, jitter, an outer manager · Akka restartWithBackoff, systemd RestartSteps=, Kubernetes delays; the OS supervises the root

Implementation notes

• Count restarts per restart event, not per child: one_for_all over 10 children is one restart.
• Tag exit notices and stop messages with a generation; stale ones must be ignored or they kill the replacement.
• Children start one at a time: a blocking init stalls later siblings; push slow work behind a first message.
• OTP's and Elixir's supervisor docs describe no delay between restarts (searched); add one in init or a wrapper. Akka, systemd and Kubernetes have it built in.

Pitfalls

• State is gone after a restart. Rebuild it from a store (write-ahead log, event sourcing) or from the peer.
• Crash loop: the same cause every time (bad config, dead dependency). Intensity ends it only by escalating; add backoff.
• Restart storm: when a shared dependency returns, all children retry together; jitter spreads them (Akka's randomFactor, exponential backoff with jitter).
• Poison message: crashes every incarnation. Akka does not re-process the failing message; queue redelivery or log replay does, forever. Count attempts, then dead-letter.
• Intensity too tight: with the OTP default a second crash within 5 s kills the supervisor; correlated crashes restart the whole subtree. one_for_all everywhere turns a local fault into a full outage.
• Up is not ready: pair restarts with health checking and a circuit breaker.

Variants and alternatives

In production

Erlang/OTP (Ericsson telecom; the thesis calls the AXD301 one of Ericsson's most reliable products), Elixir (Discord's real-time back end), Akka and Pekko, systemd and Kubernetes as outer layers, Rust: ractor.

Implementations

Sources

• Joe Armstrong, Making reliable distributed systems in the presence of software errors, PhD thesis, KTH 2003 (error handling philosophy 4.3-4.4, supervision in ch. 6). thesis
• Erlang/OTP: Supervisor Behaviour, supervisor module.
• Elixir: Supervisor, DynamicSupervisor.
• Akka documentation: Fault Tolerance, Supervision and Monitoring, SupervisorStrategy API.
• systemd: systemd.service(5), systemd.unit(5).
• Kubernetes, Pod Lifecycle (restart policy, CrashLoopBackOff).
• Stanislav Vishnevskiy, How Discord Scaled Elixir to 5,000,000 Concurrent Users, 2017. post