Process supervision

Deep dive

How it works

In: a command line per service, a policy (restart condition, base delay, cap, burst, window, stop timeout), and a stop request · Out: one live child per service, a state (running, backoff, failed, stopped) and an exit log · State: per service: child handle, start times inside the window, next delay · Cost: O(1) per exit, one fork+exec per restart (order of 1 ms, measured below)

1. Spawn the command in its own process group (Linux: its own cgroup, see cgroups and namespaces); connect stdout/stderr to a pipe or file (systemd: the journal by default).
2. Notice the exit: waitpid (blocking, on SIGCHLD, or in a waiter thread; Linux also has pidfd). Collecting the status reaps the zombie and gives exit code or signal.
3. Decide: compare cause with the policy (no, on-failure, always, …). An exit caused by the supervisor's own stop never restarts.
4. Burst check: drop start times older than the window; if starts >= burst, mark failed and stop retrying.
5. Back off: wait delay, then delay = min(2·delay, cap); after a run longer than a "stable" time reset to the base.
6. Readiness (optional): started only after READY=1 arrives on $NOTIFY_SOCKET (systemd Type=notify); otherwise "started" means "exec'd".
7. Watchdog (optional): if the keep-alive WATCHDOG=1 misses its deadline, kill the service and treat it as a failure (step 3).
8. Stop: signal the whole group with SIGTERM, wait the timeout, then SIGKILL the group, so no child escapes.

Code

Std-only Rust. SIGTERM goes through kill(1) and process_group(0) lets one signal reach grandchildren. It polls try_wait every 5 ms; a production supervisor blocks instead.

use std::collections::VecDeque;
use std::os::unix::process::CommandExt;
use std::process::{Child, Command};
use std::time::{Duration, Instant};

#[derive(Debug, PartialEq, Clone, Copy)]
pub enum State { Running, Backoff(Instant), Failed, Stopped }

pub struct Policy { pub base: Duration, pub cap: Duration, pub burst: usize, pub window: Duration, pub stable: Duration }

pub struct Service {
    pub name: &'static str,
    script: &'static str,
    child: Option<Child>,
    pub state: State,
    starts: VecDeque<Instant>, // start times inside the window
    delay: Duration,           // next backoff
    pub spawns: u32,
    pub spawn_cost: Duration,  // last fork+exec time
}

/// std only has Child::kill (SIGKILL, one pid): other signals go through kill(1);
/// "-- -PID" signals the whole process group.
fn signal(pid: u32, sig: &str) {
    let _ = Command::new("kill").args([&format!("-{sig}"), "--", &format!("-{pid}")]).status();
}

impl Service {
    pub fn new(name: &'static str, script: &'static str, p: &Policy) -> Self {
        Service { name, script, child: None, state: State::Backoff(Instant::now()),
                  starts: VecDeque::new(), delay: p.base, spawns: 0, spawn_cost: Duration::ZERO }
    }

    /// One supervision step; call every few ms.
    pub fn tick(&mut self, p: &Policy) {
        let now = Instant::now();
        match self.state {
            State::Running => {
                // try_wait = waitpid(WNOHANG): detects the exit, reaps the zombie
                let Some(status) = self.child.as_mut().unwrap().try_wait().unwrap() else { return };
                self.child = None;
                if now - *self.starts.back().unwrap() >= p.stable { self.delay = p.base; } // healthy run
                while self.starts.front().map_or(false, |t| now - *t > p.window) { self.starts.pop_front(); }
                if self.starts.len() >= p.burst {
                    self.state = State::Failed; // burst limit: stop retrying
                    println!("{}: {status}, FAILED after {} starts", self.name, self.spawns);
                } else {
                    self.state = State::Backoff(now + self.delay);
                    println!("{}: {status}, restart in {:?}", self.name, self.delay);
                    self.delay = (self.delay * 2).min(p.cap);
                }
            }
            State::Backoff(at) if now >= at => {
                let t = Instant::now();
                // own process group, so stop() can signal the whole tree
                let c = Command::new("sh").args(["-c", self.script]).process_group(0).spawn().unwrap();
                self.spawn_cost = t.elapsed();
                self.child = Some(c);
                self.starts.push_back(now);
                self.spawns += 1;
                self.state = State::Running;
            }
            _ => {}
        }
    }
}

/// SIGTERM every group, wait up to `grace`, then SIGKILL what is left.
pub fn stop(svcs: &mut [Service], grace: Duration) {
    for s in svcs.iter() { if let Some(c) = &s.child { signal(c.id(), "TERM"); } }
    let deadline = Instant::now() + grace;
    while svcs.iter().any(|s| s.child.is_some()) {
        for s in svcs.iter_mut() {
            let Some(c) = s.child.as_mut() else { continue };
            if c.try_wait().unwrap().is_some() { s.child = None; }
            else if Instant::now() >= deadline { signal(c.id(), "KILL"); let _ = c.kill(); let _ = c.wait(); s.child = None; }
        }
        std::thread::sleep(Duration::from_millis(5));
    }
    for s in svcs.iter_mut() { s.state = State::Stopped; }
}

fn main() {
    let ms = Duration::from_millis;
    let p = Policy { base: ms(20), cap: ms(80), burst: 4, window: Duration::from_secs(5), stable: Duration::from_secs(1) };
    let mut svcs = vec![
        Service::new("crasher", "sleep 0.05; exit 1", &p),
        Service::new("healthy", "sleep 30", &p),
        Service::new("stubborn", "trap '' TERM; sleep 30", &p), // ignores SIGTERM
    ];
    let t0 = Instant::now();
    while svcs[0].state != State::Failed && t0.elapsed() < Duration::from_secs(5) {
        for s in svcs.iter_mut() { s.tick(&p); }
        std::thread::sleep(ms(5));
    }
    assert_eq!((svcs[0].state, svcs[0].spawns), (State::Failed, 4));
    assert!(svcs[1].state == State::Running && svcs[1].spawns == 1);
    let pids: Vec<u32> = svcs.iter().filter_map(|s| s.child.as_ref().map(|c| c.id())).collect();
    println!("restart spawn cost: {:?}", svcs[0].spawn_cost);
    let t = Instant::now();
    stop(&mut svcs, ms(300));
    assert!(t.elapsed() < ms(600), "stop took {:?}", t.elapsed());
    std::thread::sleep(ms(50));
    for pid in pids { // kill -0 succeeds only while a group member exists
        assert!(!Command::new("kill").args(["-0", "--", &format!("-{pid}")]).output().unwrap().status.success());
    }
    println!("ok, stop took {:?}", t.elapsed());
}

Measured on this Mac (Apple M4 Pro), six runs: Command::spawn for the last restart took 0.66 to 2.47 ms. The whole run (crash loop to Failed, then stop with one child ignoring SIGTERM) takes under half a second.

Complexity and constants

• Per exit: O(1) bookkeeping plus one fork+exec (order of ms, above). With doubling and cap C, the long-run rate is at most one restart per C.
• Detection lag: none with blocking waitpid/SIGCHLD; up to one poll interval (5 ms here) when polling.
• State: at most burst timestamps per service. Stop time: at most the grace timeout.

Efficiency

• Cost: one exit costs a few syscalls (waitpid, fork, exec; spawn call 0.66-2.47 ms measured on this Mac (Apple M4 Pro)); an idle supervisor sleeps in waitpid and uses no CPU. The real cost is downtime: delay + start-up + warm-up; the bound is the service's start-up, not the supervisor.
• Levers:
  • block in waitpid/SIGCHLD/pidfd instead of polling — no wake-ups, no poll lag (this page's code) — always
  • socket activation (systemd, launchd) — clients queue on the socket while the service restarts — services with a listening socket
  • readiness notification — dependents start when the service is ready, not after a guessed sleep — ordered start-up
  • delay cap and reset — bounds the rate (Kubernetes: 300 s cap, reset after 10 min healthy) — expensive dependencies
• Naive → standard → state of the art:
  • while true; do cmd; done · restarts instantly, no limit, signals do not reach the child
  • supervisor with delay, burst limit, group kill · O(1) per exit, bounded restart rate
  • cgroups + readiness + watchdog + ordering (systemd, s6, OTP) · catches hangs as well as exits

Implementation notes

• Run services in the foreground. A daemon that forks and exits loses its PID: systemd wants Type=forking plus PIDFile= (the manual discourages it).
• Game-server shape (systemd; values are examples): Type=notify (send READY=1 once the listener is up), Restart=on-failure, RestartSec=2s, WatchdogSec=30s (send WATCHDOG=1 from the tick loop), TimeoutStopSec= above the drain time (default 90 s).
• KillMode=control-group (default) kills every process of the unit; process lets children escape.
• The service must handle SIGTERM (hand off work, close sockets, exit) or SIGKILL follows. sh -c "cmd" may swallow it: use exec cmd.
• Drain the log pipe, or a full pipe blocks the service.
• As PID 1 in a container, an ordinary app gets no default signal handlers and never reaps orphans: put tini or dumb-init in front (docker run --init).

Pitfalls

• Restart=always without a burst limit turns a startup bug into an endless loop that looks like uptime; alert on restart count.
• No delay: a failing dependency (database down) is hit by every attempt at full speed.
• Nested restarters multiply: in OTP the restarts before the top supervisor gives up are the product of the intensities above the failing child.
• Two supervisors on one process (a unit running a container with its own restart policy) fight over it.
• A hung process never exits: exit detection misses it; add a watchdog or health probe.

Variants and alternatives

In production

systemd (most Linux distributions), launchd (macOS), Kubernetes kubelet, Erlang/OTP applications, ROS 2 launch (respawn, respawn_delay, respawn_max_retries, default unlimited), tini in Docker (--init).

Implementations

Sources

• systemd manuals: systemd.service(5) (Restart=, RestartSec=, RestartSteps=, Type=notify, WatchdogSec=, TimeoutStopSec=), systemd.unit(5) (StartLimit*), systemd-system.conf(5) (defaults), systemd.kill(5), sd_notify(3)
• Erlang/OTP, Supervisor Behaviour. doc · Joe Armstrong, Making reliable distributed systems in the presence of software errors, PhD thesis, KTH 2003. thesis
• Kubernetes, Pod Lifecycle. doc
• runsv(8), s6-supervise, Supervisor configuration, Tini README, launchd.plist(5) (macOS man page).