Selective forwarding unit

Deep dive

How it works

Parts:

• Ingest transport — ICE and DTLS-SRTP per participant (SRTP, WebRTC); parses RTP and header extensions (MID, RID, dependency descriptor, transport sequence number, audio level).
• Stream registry — incoming SSRC to participant, track and layer (from SDP a=rid/a=simulcast, SDP); keyframe and dependency state.
• Packet cache — ring buffer of recent packets per incoming stream, to answer NACK locally.
• Forwarder (per subscriber and track) — layer choice and sequence/timestamp rewriter (code below).
• Estimator (per downlink) — bitrate from transport-wide feedback or REMB (delay-based control).
• Feedback router — terminates RTCP per hop; turns subscriber PLI/FIR/NACK into upstream requests or local answers.
• Speaker selector — audio levels (RFC 6464) decide who is active: last-N or dominant speaker.
• Egress transport — SRTP with the subscriber's own hop key, pacing, send.

Operations

1. Negotiate: senders announce simulcast encodings by RID (RFC 8851/8852/8853), bundle tracks by MID (RFC 9143); AV1 adds a dependency descriptor extension (AV1 RTP spec, Appendix A) that exposes frame dependencies without parsing the payload.
2. Receive: decrypt, identify the layer, cache, note keyframes.
3. Estimate each downlink from feedback (transport-wide CC draft: RTCP type 205, FMT 15; RFC 8888: FMT 11; REMB: type 206, FMT 15); add wanted size and active speakers.
4. Choose a target layer: drop at once when the estimate falls, rise only with headroom.
5. On a target change, request a keyframe upstream (PLI: PSFB FMT 1, RFC 4585; FIR: PSFB FMT 4, RFC 5104). Keep forwarding the current layer meanwhile.
6. When a keyframe of the target layer arrives, switch: continue the output sequence after the last number sent; keep timestamps monotone on the common clock (RTCP Sender Reports). Never switch on a delta frame.
7. Forward: rewrite SSRC, sequence number, timestamp (and codec fields such as the VP8 picture ID, RFC 7741), re-encrypt, send.
8. Answer a subscriber's NACK: map the output sequence number back to the source, resend from the cache (RTX, RFC 4588, or plain).

 pub A ──(3 layers, 1 uplink)──┐                      ┌─ forwarder A→B ─ SRTP ─> sub B (layer 1)
 pub B ──(3 layers, 1 uplink)──┼─> ingest ─> cache ───┼─ forwarder A→C ─ SRTP ─> sub C (layer 0)
 pub C ──(3 layers, 1 uplink)──┘     │  keyframe state └─ ...                       │
                                     ^                          estimator <── TWCC ──┘
                                     └──── PLI / FIR / NACK (feedback router) <─ subscribers

Topologies (RFC 7667). Mesh: N-1 uploads per peer. MCU: decode, mix, re-encode; high CPU and delay. SFU (the RFC's "selective forwarding middlebox", §3.7): separate SSRC space per leg, no signal processing, so very low delay and cascadable.

Congestion. Each downlink has its own estimate. Google Congestion Control (Carlucci et al.) uses delay gradients; with transport-wide feedback the SFU estimates itself, with REMB the receiver does.

Cascading and E2EE. Large rooms span several SFUs, each forwarding only the layers its subscribers need (Jitsi relays, LiveKit multi-region). Hop keys let the server read media; SFrame encrypts whole frames end to end and leaves RTP headers and extensions readable.

Code

Rust, std only: one subscriber's forwarder over three simulcast layers. main runs 900 frames (downlink 3000, 400, 3000 kbit/s), asserts continuous sequence numbers, monotone timestamps and keyframe-only switches, then times the rewrite path.

#[derive(Clone, Copy)]
struct Pkt { layer: usize, seq: u16, ts: u32, key: bool } // key: first packet of a keyframe

const RATE: [f64; 3] = [150., 500., 1500.]; // kbit/s per simulcast layer
const PKTS: [usize; 3] = [1, 2, 3]; // packets per frame and layer
const OUT_BASE: u32 = 1_000_000; // this subscriber's timestamp origin
const TICK: u32 = 3000; // 90 kHz at 30 fps

/// One per subscriber. `sr_off[l]`: layer l's RTP clock minus the common clock (known from RTCP SR).
struct Forwarder { cur: usize, target: usize, started: bool, last_seq: u16, last_ts: u32,
                   seq_off: u16, ts_off: u32, sr_off: [u32; 3] }

impl Forwarder {
    fn new(sr_off: [u32; 3]) -> Self {
        Forwarder { cur: 0, target: 0, started: false, last_seq: 65_500, last_ts: 0, seq_off: 0, ts_off: 0, sr_off }
    }
    /// Target layer from the downlink estimate (kbit/s): down at once to the best fit under 85 %,
    /// up one layer at a time with 40 % headroom. Returns the layer to send a PLI for.
    fn on_estimate(&mut self, est: f64) -> Option<usize> {
        let mut t = self.target;
        if RATE[t] > 0.85 * est {
            t = (0..3).rev().find(|&l| RATE[l] <= 0.85 * est).unwrap_or(0);
        } else if t < 2 && RATE[t + 1] <= 0.6 * est {
            t += 1;
        }
        (t != self.target).then(|| { self.target = t; t })
    }
    fn forward(&mut self, p: Pkt) -> Option<Pkt> {
        let m = p.ts.wrapping_sub(self.sr_off[p.layer]).wrapping_add(OUT_BASE); // common timeline
        if p.layer == self.target && p.key && (p.layer != self.cur || !self.started) {
            // Switch point (keyframe of the target layer): continue the sequence after the last one sent.
            self.seq_off = self.last_seq.wrapping_add(1).wrapping_sub(p.seq);
            // Timestamps follow the common clock, shifted by the minimum that keeps the output monotone.
            self.ts_off = if !self.started || (m.wrapping_sub(self.last_ts) as i32) > 0 { 0 }
                          else { self.last_ts.wrapping_add(1).wrapping_sub(m) };
            self.cur = p.layer;
            self.started = true;
        } else if !self.started || p.layer != self.cur {
            return None; // not the layer this subscriber receives
        }
        let out = Pkt { layer: p.layer, seq: p.seq.wrapping_add(self.seq_off), ts: m.wrapping_add(self.ts_off), key: p.key };
        self.last_seq = out.seq;
        self.last_ts = out.ts;
        Some(out)
    }
}

const SR: [u32; 3] = [0x1000_0000, 0x7000_0000, 0xF000_0000];
fn emit(i: u32, key: [bool; 3], seq: &mut [u16; 3], out: &mut Vec<Pkt>) {
    for l in 0..3 {
        for k in 0..PKTS[l] {
            out.push(Pkt { layer: l, seq: seq[l], ts: i * TICK + SR[l], key: key[l] && k == 0 });
            seq[l] = seq[l].wrapping_add(1);
        }
    }
}

fn main() {
    // 900 frames, downlink 3000 -> 400 -> 3000 kbit/s; a PLI is answered 3 frames (100 ms) later.
    let (mut f, mut seq, mut pli) = (Forwarder::new(SR), [65_000u16, 10, 40_000], [u32::MAX; 3]);
    let (mut buf, mut prev, mut switches) = (vec![], None::<Pkt>, vec![]);
    for i in 0..900u32 {
        if i % 30 == 0 {
            let est = if (300..600).contains(&i) { 400. } else { 3000. };
            if let Some(l) = f.on_estimate(est) { pli[l] = i + 3; }
        }
        buf.clear();
        emit(i, [0, 1, 2].map(|l| i == 0 || pli[l] == i), &mut seq, &mut buf);
        for o in buf.iter().filter_map(|&p| f.forward(p)) {
            match prev {
                None => switches.push((i, o.layer)),
                Some(q) => {
                    assert_eq!(o.seq, q.seq.wrapping_add(1), "gap or jump in sequence numbers");
                    assert!(o.ts.wrapping_sub(q.ts) <= TICK, "timestamp backwards or jump");
                    if o.layer != q.layer { assert!(o.key, "switched without a keyframe"); switches.push((i, o.layer)); }
                }
            }
            prev = Some(o);
        }
    }
    assert_eq!(switches.iter().map(|s| s.1).collect::<Vec<_>>(), [1, 2, 0, 1, 2]); // start, up, down, up, up
    println!("switches (frame, layer) {switches:?}");

    // Speed of the rewrite path on a pre-generated stream (keyframes every 30 frames).
    let (mut all, mut seq) = (vec![], [0u16; 3]);
    for i in 0..100_000u32 { emit(i, [i % 30 == 0; 3], &mut seq, &mut all); }
    let (t, mut f, mut n, mut sent) = (std::time::Instant::now(), Forwarder::new(SR), 0u64, 0u64);
    for _ in 0..10 {
        for (idx, &p) in all.iter().enumerate() {
            if idx % 540 == 0 { f.on_estimate(if (idx / 540) % 2 == 0 { 3000. } else { 400. }); }
            n += 1;
            sent += std::hint::black_box(f.forward(p)).is_some() as u64;
        }
    }
    let s = t.elapsed().as_secs_f64();
    println!("{n} offered, {sent} forwarded: {:.0} M/s offered, {:.0} M/s forwarded", n as f64 / s / 1e6, sent as f64 / s / 1e6);
}

Run (cargo run --release), asserts pass. Switches (frame, layer): [(0, 1), (33, 2), (303, 0), (603, 1), (633, 2)]. The down-switch after the frame-300 drop waits 3 frames for the simulated PLI answer; the old layer flows meanwhile, where a real SFU would pause it under overuse.

Measured on this Mac (Apple M4 Pro), 6 runs (5 on a draft of this code, 1 on the final code), one thread, 6 M pre-generated packets per run: 640-730 M offered packets/s, 160-183 M forwarded packets/s (5.5-6.3 ns per forwarded packet; the 75 % of packets from other layers are dropped in under 2 ns). Rewrite path only: no sockets, SRTP or feedback.

Complexity and constants

• Per packet and subscriber: O(1) state (a few integers), no allocation, no payload copy except descriptor edits.
• Fan-out: one published packet becomes up to N-1 sends. Switch latency: PLI round trip plus the wait for the keyframe.

Efficiency

• Cost: the rewrite path costs 5.5-6.3 ns per forwarded packet (measured on this Mac, Apple M4 Pro); per-subscriber SRTP encryption and the send syscall cost more (not measured here). Bound: network egress and per-packet crypto, not decoding; an MCU adds a decode and encode per participant.
• Levers:
  • Simulcast or SVC — each receiver gets the layer its link carries (no figure) — mixed bandwidths.
  • Forward only audible or visible tracks (last-N) — egress saved in proportion to what is skipped (no figure) — large rooms.
  • Batch sends (sendmmsg, UDP GSO) — fewer syscalls per packet (no figure) — high packet rates.
  • Serve NACK from the cache — repair stays local — lossy downlinks.
  • Cascade SFUs near users — shorter round trips per leg — rooms spanning regions.
• Naive → standard → state of the art:
  • Mesh · N-1 uploads per client
  • MCU · one stream per client, decode and encode per participant
  • SFU with simulcast, per-subscriber selection, transport-wide feedback · one uplink, quality per receiver

Implementation notes

• Find keyframes in the payload descriptor (VP8/VP9/H.264) or, for AV1 and E2EE, the dependency descriptor.
• Pace the egress; probe before an up-switch.

Pitfalls

• Switching on a delta frame: corrupt picture until the next keyframe.
• PLI storms: every new subscriber wants a keyframe; coalesce and rate-limit requests.
• Stale NACK mapping: a subscriber may NACK a number sent before the last switch; keep old offsets for a while.
• Estimate noise: switching up on every spike oscillates; add headroom and dwell time.
• Codec mismatch: an SFU cannot transcode; receivers of a track must share a codec.
• E2EE without layer metadata: no dependency descriptor, no keyframe detection.

Variants and alternatives

In production

LiveKit, mediasoup, Jitsi Videobridge (Jitsi Meet), Janus (VideoRoom), Galene; the str0m chat example shows the pattern in Rust.

Implementations

Sources

• M. Westerlund, S. Wenger, RTP Topologies, RFC 7667, 2015. RFC
• RFC 8851 (RID), 8852 (RID in SDES), 8853 (simulcast in SDP and RTP). 8851 · 8852 · 8853
• S. Holmer et al., RTP Extensions for Transport-wide Congestion Control, draft-holmer-rmcat-transport-wide-cc-extensions-01. draft
• H. Alvestrand (ed.), RTCP message for Receiver Estimated Maximum Bitrate, draft-alvestrand-rmcat-remb. draft
• RFC 8888 (congestion control feedback), 4585 (PLI, Generic NACK), 5104 (FIR), 9143 (BUNDLE, MID), 6464 (audio level). 8888 · 4585 · 5104 · 9143 · 6464
• RTP Payload Format for AV1, Appendix A: Dependency Descriptor. spec
• RFC 9605, Secure Frame (SFrame), 2024. RFC
• G. Carlucci, L. De Cicco, S. Holmer, S. Mascolo, Analysis and design of the Google congestion control for web real-time communication (WebRTC), MMSys 2016. DOI