Config distribution

Deep dive

How it works

Parts:

• resource store — named resources with a global version counter and tombstones for deletions
• delta computer — "everything changed since v"; v = 0 means snapshot
• stream per node — long-lived gRPC or SSE, server pushes
• nonce — per-push id the node echoes, so a late ACK is not matched to a newer push
• validator and last-good store — checks a push whole, applies atomically, persists
• rollout controller — staged fan-out; reverts by publishing old content as a new version
• relay tier (optional) — caches serving child nodes

1. Node connects, sends its version and subscriptions.
2. The plane sends the delta with a nonce, clusters before routes.
3. The node applies it to a copy and checks every resource and reference.
4. Valid: swap in, persist, ACK (version, nonce). Invalid: NACK with the error; the old config keeps running.
5. The plane records the acked version per node. A stage advances only without NACK; otherwise the old content is republished to nodes that accepted.
6. On reconnect step 2 sends exactly the gap, or a snapshot if old tombstones were dropped.

 author -> [store v42] -> [rollout 1% -> 10% -> 100%] -> [relay/cache]
                                     node A   node B   node C   <- delta(v) / ACK|NACK(v, nonce)
                                     (disk: last good)

Code

Bad config: body with BAD (all reject) or new-field (strict nodes reject).

use std::collections::BTreeMap;
type Res = BTreeMap<String, String>;

/// One push. `full` = snapshot (node replaces everything), else delta.
struct Msg { version: u64, nonce: u64, full: bool, upserts: Vec<(String, String)>, removed: Vec<String> }
impl Msg {
    fn bytes(&self) -> usize {
        16 + self.upserts.iter().map(|(k, v)| k.len() + v.len()).sum::<usize>()
            + self.removed.iter().map(|k| k.len()).sum::<usize>()
    }
}
#[derive(Clone, Default)]
struct Plane { ver: u64, res: Res, changed: BTreeMap<String, u64>, tomb: BTreeMap<String, u64>, nonce: u64 }
impl Plane {
    fn set(&mut self, k: &str, v: &str) {
        self.ver += 1;
        self.res.insert(k.into(), v.into());
        self.changed.insert(k.into(), self.ver);
        self.tomb.remove(k);
    }
    fn remove(&mut self, k: &str) {
        if self.res.remove(k).is_some() { self.ver += 1; self.tomb.insert(k.into(), self.ver); }
    }
    fn revert_to(&mut self, old: &Res) {
        let gone: Vec<String> = self.res.keys().filter(|k| !old.contains_key(*k)).cloned().collect();
        for k in gone { self.remove(&k); }
        for (k, v) in old { if self.res.get(k) != Some(v) { self.set(k, v); } }
    }
    /// What a node at `have` is missing (0 = snapshot).
    fn since(&mut self, have: u64) -> Msg {
        self.nonce += 1;
        let mut upserts: Vec<_> = self.res.iter()
            .filter(|(k, _)| have == 0 || self.changed[*k] > have)
            .map(|(k, v)| (k.clone(), v.clone())).collect();
        upserts.sort_by_key(|(k, _)| k.starts_with("route/")); // clusters before routes
        let removed = self.tomb.iter().filter(|(_, v)| **v > have).map(|(k, _)| k.clone()).collect();
        Msg { version: self.ver, nonce: self.nonce, full: have == 0, upserts, removed }
    }
}
struct Node { version: u64, res: Res, strict: bool }
impl Node {
    /// All-or-nothing: on Err (NACK) the old state stays.
    fn apply(&mut self, m: &Msg) -> Result<u64, String> {
        let mut cand = if m.full { Res::new() } else { self.res.clone() };
        for k in &m.removed { cand.remove(k); }
        for (k, v) in &m.upserts {
            let bad = v.contains("BAD") || (self.strict && v.contains("new-field"));
            if bad { return Err(format!("{k}: rejected")); }
            if let Some(c) = v.strip_prefix("->") { // a route needs its cluster first
                if !cand.contains_key(c) { return Err(format!("{k}: unknown {c}")); }
            }
            cand.insert(k.clone(), v.clone());
        }
        self.res = cand; self.version = m.version;
        Ok(m.nonce) // ACK echoes the nonce
    }
}
#[derive(Default, Debug)]
struct Stats { msgs: usize, bytes: usize, rolled_back: bool, touched: usize }
/// Staged rollout 1/10/100 %; a NACK stops it and reverts nodes that took it.
fn rollout(p: &mut Plane, nodes: &mut [Node], change: impl Fn(&mut Plane)) -> Stats {
    let (before, mut s) = (p.res.clone(), Stats::default());
    change(p);
    let (mut done, mut nacked) = (0, false);
    for pct in [1, 10, 100] {
        let upto = (nodes.len() * pct / 100).max(1);
        for n in &mut nodes[done..upto] {
            let m = p.since(n.version);
            s.msgs += 1; s.bytes += m.bytes();
            nacked |= n.apply(&m).is_err();
        }
        done = upto;
        if nacked { break; }
    }
    s.touched = done;
    if nacked {
        s.rolled_back = true;
        p.revert_to(&before);
        for n in nodes[..done].iter_mut().filter(|n| n.res != before) {
            let m = p.since(n.version);
            s.msgs += 1; s.bytes += m.bytes();
            n.apply(&m).unwrap();
        }
    }
    s
}
fn main() {
    let mut p = Plane::default();
    for i in 0..1000 { p.set(&format!("cluster/c{i}"), &format!("lb=rr;endpoints=10.0.{}.{}", i / 250, i % 250)); }
    p.set("route/a", "->cluster/c1");
    let mut nodes: Vec<Node> = (0..100).map(|i| Node { version: 0, res: Res::new(), strict: i % 10 == 3 }).collect();
    let full = p.since(0);
    for n in &mut nodes { n.apply(&full).unwrap(); }
    assert!(nodes.iter().all(|n| n.res == p.res));
    // 1. good one-resource change: the delta is tiny
    let s = rollout(&mut p, &mut nodes, |p| p.set("cluster/c7", "lb=lc;endpoints=1.2.3.4"));
    assert!(!s.rolled_back && nodes.iter().all(|n| n.res == p.res && n.version == p.ver));
    let snap_total = p.since(0).bytes() * 100;
    assert!(s.bytes * 50 < snap_total);
    // 2. rejected by all: the 1 % canary NACKs after one message
    let s = rollout(&mut p, &mut nodes, |p| p.set("cluster/c8", "BAD"));
    assert!(s.rolled_back && s.msgs == 1 && s.touched == 1 && nodes.iter().all(|n| n.res == p.res));
    // 3. rejected only by strict nodes: caught at 10 %, reverted
    let s = rollout(&mut p, &mut nodes, |p| p.set("cluster/c9", "new-field=1"));
    assert!(s.rolled_back && s.touched == 10 && nodes.iter().all(|n| n.res == p.res && !n.res["cluster/c9"].contains("new")));
    println!("canary rollback: {} msgs, {} nodes touched of {}", s.msgs, s.touched, nodes.len());
    // 4. a reconnecting node gets exactly the gap
    let mut late = Node { version: p.ver, res: p.res.clone(), strict: false };
    p.set("cluster/new", "lb=rr"); p.remove("cluster/c3"); p.set("cluster/c4", "lb=lc");
    let m = p.since(late.version);
    assert_eq!(m.upserts.len(), 2); assert_eq!(m.removed, vec!["cluster/c3".to_string()]);
    late.apply(&m).unwrap(); assert!(late.res == p.res);
    // 5. a route before its cluster is NACKed, the sorted push ACKed
    p.set("cluster/z", "lb=rr"); p.set("route/z", "->cluster/z");
    let mut fresh = Node { version: p.ver - 2, res: late.res.clone(), strict: false };
    let mut m = p.since(fresh.version);
    m.upserts.reverse();
    assert!(fresh.apply(&m).is_err());
    m.upserts.reverse(); assert!(fresh.apply(&m).is_ok() && fresh.res == p.res);
}

Complexity and constants

A push costs O(changed resources) bytes plus a header; a rollout costs one message per node plus one per reverted node (toy: 19 for a failed 10-node canary). Cost drivers: bytes per push, per-stream serialization and encryption, reconnect storms. Toy, measured: 1,000 clusters (37,484 B), one changed, 100 nodes: deltas 4,900 B, snapshots 3,748,400 B.

Efficiency

• Cost: one delta is tens of bytes plus header on an open stream; the bound is control-plane egress and per-stream CPU, not the diff.
• Levers:
  • deltas instead of full state — toy 765x fewer bytes; Envoy contrasts delivering 100k clusters with delivering the one that changed (xDS) — big sets, small changes.
  • relay or cache tier — origin sends once per relay, not per node (Quicksilver, blog) — beyond a few thousand nodes.
  • batching edits into one version — one push instead of many, adds up to the batch window — bursts of edits.
  • stream compression — configs shrink several-fold (order of), costs CPU — snapshots and first connects.
  • ETag polling (If-None-Match, empty 304) — no held connection, staleness = poll interval — small fleets.
• Naive → standard → state of the art:
  • every node downloads the whole file every N s · nodes x file size per interval.
  • streams with State-of-the-World pushes · all resources of a type per change.
  • delta streams, relays, NACK-gated rollout · changed bytes; bad config stops at 1 %.

Implementation notes

• One monotonic counter; a revert is a new, higher version.
• Keep tombstones until all nodes are past them; else send a snapshot.
• ACK means "valid", not "traffic flows": gate on health too.
• Persist last good (temp file, fsync, rename); load it before dialing.
• Jitter reconnects (backoff) to avoid a snapshot stampede.

Pitfalls

• Ordering: a route can arrive before its cluster. Envoy's rule is make before break: clusters, endpoints, listeners, routes; remove old clusters last. One ADS stream lets one server sequence all types.
• Ignored NACKs: the plane thinks v43, the node runs v41. Alert on lag.
• An unrepresentative canary (one build, one region) passes 1 % and fails at 10 %.

Variants and alternatives

In production

• Meshes: Envoy takes config over xDS; Istio's istiod configures Envoy proxies; gRPC xDS clients use ADS.
• Flags: Unleash SDKs cache flags in memory; LaunchDarkly server SDKs stream updates (from memory).
• CDN: Cloudflare Quicksilver replicates config via primary and secondary nodes.
• Facebook Configerator, with canary testing (Tang et al.); etcd watches feed agents.

Implementations

Sources

• Tang, Kooburat, Venkatachalam, Chander, Wen, Narayanan, Dowell, Karl, Holistic Configuration Management at Facebook, SOSP 2015. paper
• Envoy, xDS protocol · Istio, architecture · etcd, API
• Cloudflare, Quicksilver · Unleash docs