Health checking

Deep dive

How it works

In: probe results, real-request outcomes, a clock · Out: the hosts a request may go to · State: per host: up/down, run counters, error count, ejection multiplier and end time · Cost: O(1) per event; O(hosts) per pick.

1. Probe each host every interval with a timeout; add random jitter against synchronised bursts.
2. Active hysteresis: a failed probe resets the success run; fall failures in a row flip Up to Down. A good probe resets the failure run; rise successes in a row flip Down to Up. A strictly alternating host never flips.
3. Passive check: count consecutive failed real requests; at max_errs eject the host for base_eject x (times ejected), capped at max_eject.
4. Cap: refuse an ejection that would remove more than max_eject_pct of the pool.
5. Decay: a periodic sweep lowers the multiplier of well-behaved hosts.
6. Pick among hosts that are Up and not ejected.
7. Panic: if the available share is below panic_pct, ignore health and pick among all hosts.

Detection takes about fall x interval, recovery rise x interval: HAProxy defaults give about 6 s down, 4 s up.

Code

#[derive(Clone, Copy, PartialEq, Debug)]
pub enum Active { Up, Down }

#[derive(Clone, Debug)]
pub struct Backend {
    pub active: Active,
    ok_run: u32,         // consecutive good probes (counts toward `rise`)
    fail_run: u32,       // consecutive bad probes (counts toward `fall`)
    errs: u32,           // consecutive failed real requests (passive)
    ejections: u32,      // ejection multiplier, as in Envoy
    ejected_until: u64,  // ms; 0 = not ejected
}

pub struct Pool {
    pub be: Vec<Backend>,
    pub rise: u32, pub fall: u32,                     // active hysteresis
    pub max_errs: u32,                                // errors before ejection
    pub base_eject: u64, pub max_eject: u64,          // ms
    pub max_eject_pct: usize, pub panic_pct: usize,   // percent of the pool
    rr: usize,
}

impl Pool {
    pub fn new(n: usize) -> Pool {
        let b = Backend { active: Active::Up, ok_run: 0, fail_run: 0, errs: 0, ejections: 0, ejected_until: 0 };
        Pool { be: vec![b; n], rise: 2, fall: 3, max_errs: 5, base_eject: 30_000,
               max_eject: 300_000, max_eject_pct: 10, panic_pct: 50, rr: 0 }
    }

    /// One active probe; only an unbroken run flips the state.
    pub fn probe(&mut self, i: usize, ok: bool) {
        let (rise, fall, b) = (self.rise, self.fall, &mut self.be[i]);
        if ok {
            b.fail_run = 0; b.ok_run += 1;
            if b.active == Active::Down && b.ok_run >= rise { b.active = Active::Up; }
        } else {
            b.ok_run = 0; b.fail_run += 1;
            if b.active == Active::Up && b.fail_run >= fall { b.active = Active::Down; }
        }
    }

    /// Outcome of a real request (passive check).
    pub fn on_response(&mut self, i: usize, ok: bool, now: u64) {
        if ok { self.be[i].errs = 0; return; }
        self.be[i].errs += 1;
        let ejected = (0..self.be.len()).filter(|&j| self.be[j].ejected_until > now).count();
        let allowed = (ejected + 1) * 100 <= self.max_eject_pct * self.be.len(); // cap
        let b = &mut self.be[i];
        if b.errs >= self.max_errs && b.ejected_until <= now && allowed {
            b.ejections += 1;
            let dur = (self.base_eject * b.ejections as u64).min(self.max_eject);
            b.ejected_until = now + dur;
            b.errs = 0;
        }
    }

    /// Periodic sweep: well-behaved hosts shed one ejection step.
    pub fn sweep(&mut self, now: u64) {
        for b in &mut self.be {
            if b.ejected_until <= now && b.errs == 0 { b.ejections = b.ejections.saturating_sub(1); }
        }
    }

    fn available(&self, i: usize, now: u64) -> bool {
        self.be[i].active == Active::Up && self.be[i].ejected_until <= now
    }

    /// Round-robin over available hosts; in panic ignore health and use all.
    pub fn pick(&mut self, now: u64) -> Option<usize> {
        let n = self.be.len();
        let avail: Vec<usize> = (0..n).filter(|&i| self.available(i, now)).collect();
        let pool: Vec<usize> = if avail.len() * 100 < self.panic_pct * n { (0..n).collect() } else { avail };
        if pool.is_empty() { return None; } // panic_pct = 0 and nothing healthy
        self.rr += 1;
        Some(pool[self.rr % pool.len()])
    }
}

fn main() {
    // Hysteresis: a flapping backend never reaches `fall` = 3 in a row.
    let mut p = Pool::new(4);
    for k in 0..100 { p.probe(0, k % 2 == 0); }
    assert_eq!(p.be[0].active, Active::Up);
    for _ in 0..3 { p.probe(1, false); }
    assert_eq!(p.be[1].active, Active::Down);
    for k in 0..100 { p.probe(1, k % 2 == 0); }   // flapping while Down: stays Down
    assert_eq!(p.be[1].active, Active::Down);
    p.probe(1, true); p.probe(1, true);           // rise = 2
    assert_eq!(p.be[1].active, Active::Up);

    // Passive ejection: time grows per ejection; percent cap holds.
    let mut p = Pool::new(10);
    let mut now = 0;
    for _ in 0..5 { p.on_response(0, false, now); }
    assert_eq!(p.be[0].ejected_until, 30_000);
    now = 30_000;                                  // over; fails again
    for _ in 0..5 { p.on_response(0, false, now); }
    assert_eq!(p.be[0].ejected_until, now + 60_000);
    for _ in 0..5 { p.on_response(1, false, now); } // cap: 10 % of 10 = 1
    assert_eq!(p.be[1].ejected_until, 0);
    assert!((0..20).all(|_| p.pick(now) != Some(0)));
    p.sweep(now + 60_000);                          // multiplier decays
    assert_eq!(p.be[0].ejections, 1);

    // Panic: below 50 % available, route to all hosts.
    let mut p = Pool::new(4);
    for i in 0..3 { for _ in 0..3 { p.probe(i, false); } }
    let seen: std::collections::HashSet<usize> = (0..40).filter_map(|_| p.pick(0)).collect();
    assert_eq!(seen.len(), 4);
    p.panic_pct = 0;                                // panic off: only the survivor
    assert!((0..20).all(|_| p.pick(0) == Some(3)));
}

Complexity and constants

• Per probe or response O(1); the eject cap scans the pool, so a failed request costs O(hosts); keep a running count for big pools.
• Measured on this Mac (Apple M4 Pro), rustc -O, 100 hosts, 3 runs: probe 0.7 ns, on_response 3.7-3.8 ns, pick 235-251 ns (two Vec allocations per call). Loop averages.
• Probe traffic is the real cost: checkers x hosts / interval; 1,000 proxies probing 1,000 hosts every 2 s send 500,000 probes per second.

Efficiency

• Cost: a probe is one small request per checker per host per interval; the decision is nanoseconds. The bound is probe traffic.
• Levers:
  • Passive detection from real traffic — no extra requests, reacts at request rate (Envoy) — beside a slow active check.
  • Cached pass-through health endpoint — one local check feeds many probers (Envoy docs) — large meshes.
  • Jitter — removes probe bursts (HAProxy spread-checks, Envoy interval_jitter) — many checkers or hosts.
• Naive → standard → state of the art:
  • One TCP connect, first failure marks down · flaps on any blip.
  • Active rise/fall plus capped passive ejection · detection about fall x interval.
  • Shallow active plus passive and statistical outlier detection, panic, slow start (Envoy, HAProxy) · bad hosts leave fast, correlated failures never empty the pool.

Implementation notes

• Shallow vs deep. A shallow check (port open, HTTP 200) fails only for host-local reasons. A deep check that queries a shared database fails on every host at once when it is down. The Builders Library keeps dependency checks out of the load balancer and in external monitors.
• Fail open. AWS NLB, ALB and Route 53 send to all targets when none is healthy (Builders Library). Envoy's panic mode adds a percentage, and can send to all hosts or none.
• Envoy. Active: interval, timeout, unhealthy_threshold, healthy_threshold are required. A non-expected HTTP status marks unhealthy at once; at startup one success suffices; interval_jitter adds jitter. Outlier defaults: consecutive_5xx 5, interval 10 s, base_ejection_time 30 s, max_ejection_percent 10, max_ejection_time 300 s. A passing active check unejects a host unless successful_active_health_check_uneject_host is false.
• HAProxy. inter 2000 ms, rise 2, fall 3. observe layer7 adds passive checks (error-limit default 10). slowstart ramps a recovered server up.
• nginx. Open source is passive: max_fails (default 1) within fail_timeout (default 10 s) makes a server unavailable for fail_timeout.
• Kubernetes. periodSeconds 10, timeoutSeconds 1, failureThreshold 3, successThreshold 1 (must be 1 for liveness and startup). Liveness failure restarts the container; readiness failure removes the Pod from Service endpoints; a startup probe holds both back.
• gRPC. grpc.health.v1.Health: Check (set a deadline) and Watch (streams changes); empty service name = whole server; unknown name = NOT_FOUND.
• Route 53. Interval 10 or 30 s plus a failure threshold; healthy if more than 18 % of checkers say so, so one isolated region cannot declare it dead.
• Separate health from load. Answer probes from reserved capacity (extra workers, own port), or an overloaded host fails its check and shifts load onto the rest. Report draining (graceful draining).

Pitfalls

• Dependency check in a liveness probe: when the dependency fails, every container restarts.
• Deep check without fail-open or a cap: a shared outage empties the pool.
• max_ejection_percent 10 on fewer than ten hosts ejects nobody unless always_eject_one_host is set.
• Probe path unlike real traffic: Route 53 HTTPS checks skip certificate validation.

Variants and alternatives

In production

HAProxy, Envoy and meshes built on it, nginx, the Kubernetes kubelet, gRPC, Route 53 and AWS load balancers.

Implementations

Sources

• HAProxy, Configuration Manual 3.0. docs
• Envoy, architecture overview. health checking · outlier detection · panic threshold
• Envoy API, HealthCheck and OutlierDetection fields. health_check · outlier_detection
• nginx, ngx_http_upstream_module. docs
• Kubernetes, Configure Liveness, Readiness and Startup Probes; Pod API reference. task · Pod API
• gRPC, GRPC Health Checking Protocol. spec · health.proto
• Amazon Route 53 Developer Guide, How Amazon Route 53 determines whether a health check is healthy. docs
• David Yanacek, Implementing health checks, Amazon Builders' Library, 2019. article