Jitter buffer

Deep dive

How it works

Parts:

• Packet store — packets ordered by sequence number; drops duplicates and late ones.
• Delay estimator — arrival delay against sender timestamps; EWMA, histogram or Kalman filter.
• Target policy — statistic to playout delay (late loss against latency).
• Scheduler — per output tick: play, wait, conceal or skip.
• Time-scale control (audio) — stretches or shrinks by pitch periods to move the delay without clicks.
• Feedback (video) — NACK and keyframe requests; RTT enters the target.

Operations

1. Insert: transit = arrival - rtp_timestamp in one clock unit (RFC 3550 §6.4.1); store; drop duplicates and packets whose slot has passed.
2. Estimate: RFC 3550 jitter J += (|D| - J)/16, D = change of transit between consecutive packets (Appendix A.8): a mean absolute difference, not a delay level.
3. Choose the target: fixed, a quantile of recent delays, or mean delay + k·J.
4. Move toward it slowly: stretch or shrink audio, or adjust at a talkspurt start (RFC 3551 §4.1 sets the marker bit there and allows the adjustment).
5. Each tick: play the next packet if present, else conceal (PLC) and discard it if it arrives later.
6. Video: assemble frames, decode when complete and decodable, schedule by render time; request a keyframe if a gap cannot be repaired.

 network ──> [packet store] ──> [scheduler] ──> decoder / PLC ──> sink
                  │ transit            ^
                  v                    │ accelerate / expand
        [delay estimator] ──> [target policy]      NACK, PLI <── [feedback]

Audio. Ramjee et al. (1994) adjust the delay per talkspurt with delay-spike detection; Moon, Kurose and Towsley (1998) bound the best achievable delay for a given loss and track delay percentiles of recent packets. NetEq follows the percentile line: its UnderrunOptimizer keeps a 100-bucket histogram of 20 ms with exponential forgetting, feeds it the largest delay of each 500 ms interval and reads the 0.95 quantile (delay_manager.h: quantile = 0.95, forget_factor = 0.983); a ReorderOptimizer weighs reordering against loss. Each tick it returns an Operation: kNormal; kAccelerate (remove pitch periods, buffer too full); kPreemptiveExpand (add them, too empty); kExpand (no packet: concealment); kMerge (cross-fade from expanded audio into the first real packet).

Video. PacketBuffer assembles frames and flags a cleared buffer (send a keyframe request). JitterEstimator fits frame delay against frame size with a two-state Kalman filter (inverse bandwidth, queueing delay), adds kNoiseStdDevs = 2.33 noise deviations and 10 ms for the OS, and after kNackLimit = 3 NACKs adds min(0.9 × RTT, 200 ms). VCMTiming sets the target to jitter delay + decode time (95th percentile) + render delay.

Drift and repair. A 100 ppm clock offset moves the delay 6 ms per minute (arithmetic) and looks like a trend: compensate it separately (resample, or accelerate/expand). FEC repairs without a round trip but needs its whole group buffered; NACK needs delay above jitter plus one RTT, else concealment replaces repair. Lip sync adds a constraint (A/V sync).

Code

Rust, std only: 1000 packets of 20 ms (base delay 20 ms, exponential jitter, two 150 ms spikes) through a fixed, a quantile and a mean + k·J buffer. Adaptive delay may grow 10 ms and shrink 1 ms per packet (simulation parameters). main asserts the RFC estimator on a hand-worked vector and the outcomes.

/// RFC 3550 A.8 interarrival jitter.
struct Rfc3550 { transit: Option<f64>, j: f64 }
impl Rfc3550 {
    fn update(&mut self, arrival: f64, ts: f64) -> f64 {
        let transit = arrival - ts;
        if let Some(prev) = self.transit {
            self.j += ((transit - prev).abs() - self.j) / 16.0;
        }
        self.transit = Some(transit);
        self.j
    }
}

#[derive(Clone, Copy)]
enum Policy { Fixed(f64), Quantile(f64), JitterK(f64) }

const PTIME: f64 = 20.0;   // ms of audio per packet
const BASE: f64 = 20.0;    // network delay without jitter, ms
const UP: f64 = 10.0;      // max growth of the playout delay per packet (inserted audio)
const DOWN: f64 = 1.0;     // max shrink per packet (5 % faster playout)
const WINDOW: usize = 100; // 2 s of delay history
const SPIKES: [usize; 2] = [300, 700];

/// BASE + exponential jitter (mean 4 ms, cap 40 ms) + spikes of 150 ms draining 10 ms per packet.
fn trace(n: usize) -> Vec<f64> {
    let mut s = 0x9E3779B97F4A7C15u64;
    (0..n).map(|i| {
        s ^= s << 13; s ^= s >> 7; s ^= s << 17; // xorshift64
        let u = (s >> 11) as f64 / (1u64 << 53) as f64;
        let spike = SPIKES.iter().filter(|&&p| i >= p)
            .map(|&p| (150.0 - 10.0 * (i - p) as f64).max(0.0)).fold(0.0, f64::max);
        BASE + (-4.0 * (1.0 - u).ln()).min(40.0) + spike
    }).collect()
}

/// (late packets, late outside the spikes, mean playout delay in ms after send time).
/// Late: network delay exceeds the playout delay in force when the packet was due.
fn simulate(delays: &[f64], policy: Policy) -> (usize, usize, f64) {
    let mut est = Rfc3550 { transit: None, j: 0.0 };
    let (mut d, mut mean_delay) = (delays[0], delays[0]);
    if let Policy::Fixed(f) = policy { d = f; }
    let (mut late, mut late_calm, mut sum_d) = (0, 0, 0.0);
    for (i, &net) in delays.iter().enumerate() {
        let j = est.update(i as f64 * PTIME + net, i as f64 * PTIME);
        sum_d += d;
        if net > d { // missed its slot: concealed, then discarded on arrival
            late += 1;
            late_calm += !SPIKES.iter().any(|&p| (p..p + 16).contains(&i)) as usize;
        }
        mean_delay += (net - mean_delay) / 16.0;
        let target = match policy {
            Policy::Fixed(f) => f,
            Policy::Quantile(q) => {
                let mut w = delays[i.saturating_sub(WINDOW - 1)..=i].to_vec();
                w.sort_by(|a, b| a.partial_cmp(b).unwrap());
                w[((w.len() - 1) as f64 * q).ceil() as usize]
            }
            Policy::JitterK(k) => mean_delay + k * j, // mean delay plus k times the RFC 3550 jitter
        };
        d = if target > d { (d + UP).min(target) } else { (d - DOWN).max(target) };
    }
    (late, late_calm, sum_d / delays.len() as f64)
}

fn main() {
    // RFC 3550 by hand: transit 0, 10, 4, 20 give |D| = 10, 6, 16: J = 0.625,
    // 0.625 + 5.375/16 = 0.9609375, 0.9609375 + 15.0390625/16.
    let mut e = Rfc3550 { transit: None, j: 0.0 };
    let js: Vec<f64> = [0.0, 10.0, 4.0, 20.0].iter().map(|&t| e.update(t + 100.0, 100.0)).collect();
    assert_eq!(js, [0.0, 0.625, 0.9609375, 1.90087890625]);

    let t = trace(1000); // 20 s of audio
    let run = |p| simulate(&t, p);
    let (f40, f60, f120) = (run(Policy::Fixed(40.0)), run(Policy::Fixed(60.0)), run(Policy::Fixed(120.0)));
    let (q95, q99, jk4) = (run(Policy::Quantile(0.95)), run(Policy::Quantile(0.99)), run(Policy::JitterK(4.0)));
    for (name, r) in [("fixed 40", f40), ("fixed 60", f60), ("fixed 120", f120),
                      ("quantile 0.95", q95), ("quantile 0.99", q99), ("mean + 4 J", jk4)] {
        println!("{name:14} late {:3} (calm {:3})  mean playout delay {:5.1} ms, added {:5.1} ms", r.0, r.1, r.2, r.2 - BASE);
    }
    // Outcomes of this trace (asserted as measured).
    assert_eq!([f40.0, f60.0, f120.0, q95.0, q99.0, jk4.0], [33, 24, 12, 54, 21, 21]);
    assert!(jk4.2 < f60.2 && jk4.0 < f40.0);            // mean + 4 J: 21 late at 46.7 ms; fixed 40 loses 33
    assert!(q95.0 > f60.0 && (q95.2 - f60.2).abs() < 1.0); // q 0.95 at the same mean delay as fixed 60 loses more
    assert!(q95.1 > 25 && f60.1 < 5);                    // a 0.95 target gives up about 5 % in calm periods
    assert!(f120.0 < jk4.0 && f120.2 > 2.0 * jk4.2);     // fixed 120 loses least, at 2.6x the delay (spike peak 170 ms)
}

In this simulation (late packets, of which outside the spikes; mean playout delay): fixed 40 ms 33 (5), 40.0 ms; fixed 60 ms 24 (0), 60.0 ms; fixed 120 ms 12 (0), 120.0 ms; quantile 0.95 54 (36), 60.1 ms; quantile 0.99 21 (6), 79.9 ms; mean + 4J 21 (7), 46.7 ms. Surprise: the 0.95 quantile has the mean delay of fixed 60 ms but loses more than twice as many packets: it accepts 5 % loss in calm periods and keeps the spike in its window.

Complexity and constants

• Per packet: O(1) for fixed, EWMA and histogram targets; O(W log W) for the sorted window. Cost drivers: window length, estimator, one stretch operation per tick.

Efficiency

• Cost: measured on this Mac (Apple M4 Pro), best of 7, 3 repeats, final page code: 2.1-2.2 ns per packet for fixed and mean + k·J (both include the RFC 3550 update); 550-555 ns for the sorted 100-entry quantile, negligible at 50 packets/s. Bound: the latency budget, not compute; decoding and stretching cost more (not measured).
• Levers:
  • Histogram quantile instead of a sort — O(1) per packet (NetEq: 100 buckets) — long windows.
  • Largest delay per interval before the update (NetEq: 500 ms) — an outlier counts once — bursty links.
  • Grow fast, shrink slowly (here 10 vs 1 ms per packet) — hides changes, limits oscillation — audio with stretching.
  • RTT headroom only when NACKs are in use (libwebrtc: after 3, capped at 200 ms) — repair delay paid on demand — video.
• Naive → standard → state of the art:
  • Fixed buffer sized for the worst delay · constant high latency (fixed 120 ms: 100 ms above the base)
  • Mean + k·deviation, adjusted at talkspurt starts · low delay, lags behind spikes
  • Quantile histogram with forgetting, spike handling and time-scale stretching (NetEq, Moon et al.) · delay follows the network without audible jumps

Implementation notes

• One clock unit; wraparound-safe 32-bit timestamp differences.
• Treat the first packet after a silence gap as a new baseline, not as jitter.
• Report nominal, maximum and adaptive mode (RFC 3611 §4.7.7).

Pitfalls

• J used as a delay: it measures change; add a mean delay (as the code does).
• Quantile equals loss rate: pick it from the loss budget.
• Spike blindness: a sudden 150 ms queue hits an adaptive buffer like a small fixed one.
• Drift read as congestion: the target grows until playout lags.
• Keyframes inflate delay variation: video estimators must use frame size.

Variants and alternatives

In production

libwebrtc (NetEq for audio; PacketBuffer, JitterEstimator, VCMTiming for video); GStreamer; PJSIP, Asterisk and FreeSWITCH in telephony. str0m's README lists an adaptive jitter buffer as unsupported.

Implementations

Sources

• R. Ramjee, J. Kurose, D. Towsley, H. Schulzrinne, Adaptive playout mechanisms for packetized audio applications in wide-area networks, IEEE INFOCOM 1994. DOI
• S. B. Moon, J. Kurose, D. Towsley, Packet audio playout delay adjustment: performance bounds and algorithms, Multimedia Systems 6(1), 1998. DOI
• RFC 3550 (RTP), §6.4.1 and Appendix A.8; RFC 3551 §4.1 (talkspurt marker); RFC 3611 §4.7.7 (jitter buffer parameters). 3550 · 3551 · 3611
• WebRTC source: delay_manager.h · underrun_optimizer.cc · jitter_estimator.h · timing.h