Circuit breaker

Deep dive

How it works

In: the outcome (success, failure, slow) of each call; a clock · Out: allow or reject decision per call · State: closed/open/half-open, a sliding window of recent outcomes, the open-until time · Cost: O(1) per call; memory O(window).

1. Closed: every call proceeds; record success/failure (and optionally "slow" as failure) in a sliding window, count-based (last N calls) or time-based (last N seconds, in 1 s buckets).
2. If calls ≥ min_calls and failure_rate ≥ threshold → Open until now + open_duration.
3. Open: reject immediately (error or fallback). No load reaches the dependency.
4. After the cool-down → Half-open: allow up to k trial calls; reject the rest.
5. All k trials succeed → Closed with a fresh window. Any trial fails → Open again (optionally with a growing cool-down).

          failure rate ≥ threshold
 Closed ─────────────────────────► Open ──(cool-down elapsed)──► Half-open
   ▲                                 ▲                              │
   └──────── k trials succeed ───────┼──────── a trial fails ───────┘

Code

use std::collections::VecDeque;
use std::time::{Duration, Instant};

#[derive(Debug, Clone, Copy, PartialEq)]
pub enum State { Closed, Open { until: Instant }, HalfOpen { admitted: u32, succeeded: u32 } }

pub struct Breaker {
    pub state: State,
    window: VecDeque<bool>, // true = failure; count-based sliding window
    failures: usize,
    window_size: usize,
    min_calls: usize,
    threshold: f64,         // e.g. 0.5
    open_for: Duration,
    trials: u32,            // calls permitted in half-open
}

impl Breaker {
    pub fn new(window_size: usize, min_calls: usize, threshold: f64, open_for: Duration, trials: u32) -> Self {
        Breaker { state: State::Closed, window: VecDeque::with_capacity(window_size), failures: 0,
                  window_size, min_calls, threshold, open_for, trials }
    }

    /// Ask before calling. false = fail fast (use the fallback).
    pub fn allow(&mut self, now: Instant) -> bool {
        match self.state {
            State::Closed => true,
            State::Open { until } if now < until => false,
            State::Open { .. } => { self.state = State::HalfOpen { admitted: 1, succeeded: 0 }; true }
            State::HalfOpen { admitted, succeeded } if admitted < self.trials => {
                self.state = State::HalfOpen { admitted: admitted + 1, succeeded };
                true
            }
            State::HalfOpen { .. } => false,
        }
    }

    /// Report the outcome of a call that `allow` admitted. Timeouts count as failures.
    pub fn record(&mut self, ok: bool, now: Instant) {
        match self.state {
            State::Closed => {
                self.window.push_back(!ok);
                if !ok { self.failures += 1; }
                if self.window.len() > self.window_size && self.window.pop_front() == Some(true) {
                    self.failures -= 1;
                }
                let n = self.window.len();
                if n >= self.min_calls && self.failures as f64 >= self.threshold * n as f64 {
                    self.trip(now);
                }
            }
            State::HalfOpen { admitted, succeeded } => {
                if !ok { self.trip(now); return; }
                let succeeded = succeeded + 1;
                if succeeded >= self.trials { self.reset(); }
                else { self.state = State::HalfOpen { admitted, succeeded }; }
            }
            State::Open { .. } => {} // late result from before the trip: ignore
        }
    }

    fn trip(&mut self, now: Instant) { self.state = State::Open { until: now + self.open_for }; self.clear(); }
    fn reset(&mut self) { self.state = State::Closed; self.clear(); }
    fn clear(&mut self) { self.window.clear(); self.failures = 0; }
}

/// Google SRE client-side adaptive throttling ("soft" breaker); counts over ~2 minutes.
pub fn reject_probability(requests: f64, accepts: f64, k: f64) -> f64 {
    ((requests - k * accepts) / (requests + 1.0)).max(0.0) // k = 2: allow ~2x the accepted rate
}

fn main() {
    let t0 = Instant::now();
    let mut b = Breaker::new(20, 10, 0.5, Duration::from_secs(5), 3);
    for i in 0..10 { assert!(b.allow(t0)); b.record(i % 2 == 0, t0); } // 50 % failures
    assert!(matches!(b.state, State::Open { .. }) && !b.allow(t0));
    let later = t0 + Duration::from_secs(5);
    for _ in 0..3 { assert!(b.allow(later)); }
    assert!(!b.allow(later));                                   // only 3 trials
    for _ in 0..3 { b.record(true, later); }
    assert_eq!(b.state, State::Closed);
}

Complexity and constants

• Per call: O(1) (ring buffer push/pop, a compare). Time-based windows use ~10-60 one-second buckets.
• Detection delay: count-based needs min_calls outcomes; at 100 calls/s and min 100 that's ≈ 1 s, at 1 call/s ≈ 100 s. Size min_calls to traffic.
• Rejected calls cost order of 100 ns instead of a full timeout (often 1-30 s), which is the point: threads and connections stay free.
• Adaptive throttling with K = 2 lets about twice the accepted rate through, so the backend still sees enough traffic to signal recovery.

Efficiency

• Cost: Per call a ring-buffer push/pop and a compare: tens of ns, order of. A rejected call costs order of 100 ns instead of a timeout of 1-30 s. The bound is the caller's threads, connections and latency budget, not CPU.
• Levers:
  • Fail fast while open — a 1-30 s timeout becomes order of 100 ns per rejected call — dependency down.
  • Counters in 1 s buckets instead of per-call records — O(calls) → O(10-60 buckets) memory — high call rates.
  • Few half-open trials (Resilience4j default 10) — a recovering service is not re-flooded (Resilience4j docs) — always.
  • Adaptive throttling with K = 2 — no hard states, the backend still sees about twice the accepted rate (SRE book, ch. 21) — client-side overload control.
• Naive → standard → state of the art:
  • Every call waits for its full timeout, then retries · a thread and connection held 1-30 s per call.
  • Count-window breaker per dependency and operation class · O(1) per call, fail in order of 100 ns when open.
  • Adaptive client throttling plus per-host outlier ejection and bulkheads (SRE book, Envoy) · no hard state, sheds in proportion to observed rejections.

Implementation notes

• Count timeouts and "slow calls" (latency above a threshold) as failures; exclude client errors (4xx) and cancellations.
• One breaker per dependency and per operation class when they fail independently (reads vs writes), not one global breaker.
• Shared state across threads: atomics for counters, CAS for state transitions; in highly concurrent servers per-core breakers aggregated periodically avoid contention.
• Expose state as metrics and logs; an open breaker should page the owner of the dependency, not only the caller.
• Combine with bulkheads (bounded concurrency per dependency) and retries with backoff; never retry calls the breaker rejected.
• Envoy splits this into concurrency limits per upstream cluster (max_connections, max_pending_requests, max_requests default 1,024; max_retries 3) and outlier detection (eject a host after 5 consecutive 5xx for 30 s, at most 10 % of hosts).

Pitfalls

• Global breaker for a sharded/replicated backend: one bad shard opens it for all keys.
• Half-open letting all waiting traffic through at once: the recovering service falls over again. Limit trials.
• Thresholds tuned on low traffic: flapping between open and closed.
• No fallback: the breaker only converts slow failures into fast ones; the feature still breaks. Decide the degraded behaviour.
• Treating breaker-open as a dependency failure in your own health check: cascades the outage upward.

Variants and alternatives

In production

Resilience4j (Java), Netflix Hystrix (maintenance mode since 2018), Polly (.NET), Envoy (circuit breaking thresholds + outlier detection), Istio DestinationRule outlier detection, Finagle failure accrual.

Implementations

Sources

• Michael T. Nygard, Release It! Design and Deploy Production-Ready Software, Pragmatic Bookshelf, 2007 (2nd ed. 2018).
• Betsy Beyer et al. (eds.), Site Reliability Engineering, O'Reilly 2016, ch. 21 Handling Overload (client-side throttling). book
• Martin Fowler, CircuitBreaker, martinfowler.com, 2014. post
• Resilience4j, CircuitBreaker, documentation. docs · code