BBR congestion control

Deep dive

How it works

In: per-ACK delivery and RTT samples · Out: pacing rate and cwnd · State: two windowed filters (BtlBw, RTprop), mode, cycle phase, per-packet delivered stamps · Cost: O(1) per ACK

1. Delivery rate sampling (per ACK): each sent packet records delivered and delivered_time of the connection. On its ACK, rate = (delivered_now − p.delivered) / max(send_interval, ack_interval). Samples taken while the app had nothing to send are marked app-limited and only count if they raise the max.
2. Filters: BtlBw = windowed max over 10 rounds; RTprop = windowed min over 10 s.
3. State machine (BBRv1):
   • Startup: pacing_gain = cwnd_gain = 2/ln2 ≈ 2.89 (doubles rate per round, like slow start). Exit when BtlBw grew < 25 % for 3 rounds.
   • Drain: pacing_gain = ln2/2 ≈ 0.35 until inflight ≤ BDP, emptying the queue Startup built.
   • ProbeBW: cycle pacing_gain through [1.25, 0.75, 1, 1, 1, 1, 1, 1], one phase per RTprop. The 1.25 phase probes for more bandwidth, the 0.75 phase drains what the probe queued. cwnd_gain = 2.
   • ProbeRTT: if RTprop was not refreshed for 10 s, drop cwnd to 4 packets for ≥ 200 ms plus one round to measure the empty-queue RTT. Costs about 2 % of throughput.
4. Sender paces every packet at pacing_rate and never exceeds cwnd.

 delivery rate                         RTT
   ^       ________ BtlBw                ^               /  loss-based
   |      /                              |              /   operates here
   |     /                               |  RTprop ____/
   |    /   BBR operates at the knee:    |  BBR here  ^
   +---|------------------> inflight     +-----------|-------> inflight
      BDP                                           BDP     BDP + buffer

Code

/// Windowed max filter keeping best, 2nd and 3rd best samples (Nichols; Linux win_minmax.c).
#[derive(Clone, Copy, Default)]
struct S { t: u64, v: f64 }
pub struct MaxFilter { s: [S; 3], win: u64 }

impl MaxFilter {
    pub fn new(win: u64) -> Self { MaxFilter { s: [S::default(); 3], win } }
    pub fn get(&self) -> f64 { self.s[0].v }
    pub fn update(&mut self, t: u64, v: f64) {
        let n = S { t, v };
        if v >= self.s[0].v || t - self.s[2].t > self.win { self.s = [n; 3]; return; }
        if v >= self.s[1].v { self.s[1] = n; self.s[2] = n; } else if v >= self.s[2].v { self.s[2] = n; }
        // Expire the best sample, promote the others.
        if t - self.s[0].t > self.win {
            self.s = [self.s[1], self.s[2], n];
            if t - self.s[0].t > self.win { self.s = [self.s[1], self.s[2], n]; }
        } else if self.s[1].t == self.s[0].t && t - self.s[0].t > self.win / 4 {
            self.s[1] = n; self.s[2] = n; // keep sub-window diversity
        } else if self.s[2].t == self.s[1].t && t - self.s[0].t > self.win / 2 {
            self.s[2] = n;
        }
    }
}

#[derive(PartialEq)]
enum Mode { Startup, Drain, ProbeBw, ProbeRtt }
const GAINS: [f64; 8] = [1.25, 0.75, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0];

pub struct Bbr {
    mode: Mode,
    btlbw: MaxFilter,          // bytes/s, window in rounds
    rtprop: f64, rtprop_stamp: f64, // seconds
    full_bw: f64, full_cnt: u32,
    cycle: usize, cycle_stamp: f64,
    probe_rtt_done: f64,
    pub pacing_rate: f64, pub cwnd: f64,
}

impl Bbr {
    pub fn new() -> Self {
        Bbr { mode: Mode::Startup, btlbw: MaxFilter::new(10), rtprop: f64::INFINITY, rtprop_stamp: 0.0,
              full_bw: 0.0, full_cnt: 0, cycle: 0, cycle_stamp: 0.0, probe_rtt_done: 0.0,
              pacing_rate: 0.0, cwnd: 10.0 * 1448.0 }
    }
    fn bdp(&self) -> f64 { self.btlbw.get() * self.rtprop }

    /// Call once per ACK with a delivery-rate sample. `new_round` = a packet sent after the
    /// previous round start was acked.
    pub fn on_ack(&mut self, now: f64, round: u64, new_round: bool, rate: f64, app_limited: bool,
                  rtt: f64, inflight: f64) {
        if !app_limited || rate > self.btlbw.get() { self.btlbw.update(round, rate); }
        if rtt <= self.rtprop || now - self.rtprop_stamp > 10.0 { self.rtprop = rtt; self.rtprop_stamp = now; }
        match self.mode {
            Mode::Startup if new_round && !app_limited => {
                let bw = self.btlbw.get();
                if bw >= self.full_bw * 1.25 { self.full_bw = bw; self.full_cnt = 0; }
                else { self.full_cnt += 1; if self.full_cnt >= 3 { self.mode = Mode::Drain; } }
            }
            Mode::Drain if inflight <= self.bdp() => { self.mode = Mode::ProbeBw; self.cycle_stamp = now; }
            Mode::ProbeBw if now - self.cycle_stamp > self.rtprop => {
                self.cycle = (self.cycle + 1) % GAINS.len(); self.cycle_stamp = now;
            }
            Mode::ProbeRtt if now >= self.probe_rtt_done => {
                self.rtprop_stamp = now; self.mode = Mode::ProbeBw; self.cycle_stamp = now;
            }
            _ => {}
        }
        if self.mode != Mode::ProbeRtt && now - self.rtprop_stamp > 10.0 {
            self.mode = Mode::ProbeRtt; self.probe_rtt_done = now + 0.2_f64.max(self.rtprop);
        }
        let (pg, cg) = match self.mode {
            Mode::Startup => (2.885, 2.885),
            Mode::Drain => (1.0 / 2.885, 2.885),
            Mode::ProbeBw => (GAINS[self.cycle], 2.0),
            Mode::ProbeRtt => (1.0, 0.0),
        };
        self.pacing_rate = pg * self.btlbw.get();
        self.cwnd = if self.mode == Mode::ProbeRtt { 4.0 * 1448.0 } else { (cg * self.bdp()).max(4.0 * 1448.0) };
    }
}

Complexity and constants

• O(1) per ACK; state is a few dozen bytes plus per-packet delivered/delivered_time (16 bytes) in the retransmit queue.
• Steady-state queue: about 0-1 BDP in ProbeBW (the 2·BDP cwnd absorbs ACK aggregation); ProbeRTT costs order of 2 % of time.
• Loss tolerance: BBRv1 keeps near line rate up to order of 15 % random loss; CUBIC collapses well below 1 % on long paths.
• Startup reaches full bandwidth in about log2(BDP/IW) rounds, like slow start, but overshoots to roughly 2.89x BDP queued before Drain.

Efficiency

• Cost: O(1) per ACK on the sender, a few dozen bytes of state per flow plus 16 bytes per in-flight packet for delivery-rate sampling. The bound is the network, not the CPU: BBR's job is to fill the bottleneck without filling its queue. CPU cost comes from pacing, not from the model.
• Levers:
  • Pacing in the fq qdisc or TSO/GSO autosized bursts (~1 ms of data) — avoids per-packet timers (Implementation notes) — above a few Gbit/s.
  • Loss-tolerant window — keeps high throughput under random loss where CUBIC collapses (Cardwell et al., ACM Queue 2016) — lossy long-haul paths (WAN, wireless).
  • Short queues — lower median RTT for YouTube traffic (same paper) — shared bottlenecks with deep buffers.
  • BBRv3 loss/ECN bounds — gentler to coexisting CUBIC flows (IETF CCWG draft) — shared links with loss-based traffic.
• Naive → standard → state of the art:
  • Loss-based AIMD (Reno) · halves its window on any loss, so long-RTT paths stay far below capacity.
  • CUBIC · window grows by a cubic function; still fills the buffer and treats every loss as congestion.
  • BBRv1 → BBRv3 · model-based (bandwidth x RTT), paced, with loss/ECN bounds in v2/v3; deployed at Google.

Implementation notes

• Linux: sysctl net.ipv4.tcp_congestion_control=bbr (since 4.9). Since 4.13 TCP has internal pacing; the fq qdisc is still the efficient pacer (tc qdisc replace dev eth0 root fq).
• Pacing granularity: release packets in small bursts (TSO/GSO autosizing ~1 ms of data) rather than per packet; per-packet timers cost too much CPU above a few Gbit/s.
• ACK aggregation (Wi-Fi, cellular, GRO) makes delivery rate bursty; BBRv1 in Linux adds an extra cwnd allowance for measured aggregation ("extra_acked").
• In QUIC you own all of this: delivery-rate sampling must use QUIC packet numbers and must skip ACK-only packets.

Pitfalls

• No pacer → BBR sends cwnd-sized bursts and behaves badly. Always verify pacing is active (ss -ti shows pacing_rate).
• Policers (token bucket rate limiters on ISP links) look like bandwidth until they drop; v1 gets a high loss rate. Linux BBR has a policer-detection mode ("lt_bw").
• Mixed with CUBIC in a deep buffer, BBRv1 can be starved (its 2·BDP cap leaves room CUBIC fills); in a shallow buffer, BBRv1 starves CUBIC.
• ProbeRTT synchronizes across flows sharing a bottleneck; fine, but it causes periodic 200 ms throughput dips visible in dashboards.
• App-limited senders (RPC, chat) rarely produce valid max-bandwidth samples; BtlBw can stay stale for the filter window.

Variants and alternatives

In production

Google (google.com, YouTube, B4 WAN, Google Cloud), Linux mainline (tcp_bbr.c, BBRv1), many CDNs on edge servers. QUIC stacks quiche, msquic, quinn, mvfst offer BBR variants. Launch note for google.com, YouTube and Google Cloud: google/bbr.

Implementations

Sources

• Cardwell, Cheng, Gunn, Hassas Yeganeh, Jacobson, BBR: Congestion-Based Congestion Control, ACM Queue, 2016 (CACM 2017). paper · code
• Cheng, Cardwell, Hassas Yeganeh, Jacobson, Delivery Rate Estimation, IETF draft, 2017+. draft
• Hock, Bless, Zitterbart, Experimental Evaluation of BBR Congestion Control, IEEE ICNP, 2017. paper
• Ware, Mukerjee, Seshan, Sherry, Modeling BBR's Interactions with Loss-Based Congestion Control, ACM IMC, 2019. paper
• Cardwell et al., BBR Congestion Control, IETF CCWG draft. draft