IP multicast

Deep dive

How it works

In: datagrams to a group G from one or two redundant feeds · Out: every message once, in order, plus holes to re-request · State: next expected sequence number, a map of packets ahead of a hole · Cost: O(1) per in-order packet, O(log g) with g packets ahead of a hole

1. Pick an address. 224.0.0.0/24 is link-local control traffic, never forwarded by routers; 232.0.0.0/8 is source-specific (SSM, RFC 4607); 239.0.0.0/8 is administratively scoped (RFC 2365), reusable inside your own domain. IPv6 carries a 4-bit scope in ff0s::/8 (RFC 4291): 2 link, 5 site, 8 organization, e global.
2. The receiver joins (IP_ADD_MEMBERSHIP): the host sends an IGMP report. The querier re-asks periodically; IGMPv3 defaults are a 125 s query interval and robustness 2, so membership lapses 2·125 + 10 = 260 s after the last report. IGMPv2 (RFC 2236) adds leave messages, IGMPv3 (RFC 3376) source filters, MLDv2 (RFC 3810) does both for IPv6.
3. Layer 2 maps the group to a MAC (code below; IPv6 uses 33:33 plus the last 4 address bytes, RFC 2464). An IGMP-snooping switch forwards frames only to ports that joined; without snooping it floods them to all.
4. Routers join subnets with PIM-SM (RFC 7761): shared trees rooted at a rendezvous point, optionally switching to per-source shortest-path trees. SSM needs no rendezvous point.
5. The sender sets IP_MULTICAST_TTL (each router hop decrements it, so it is a scope) and sends. IP_MULTICAST_LOOP lets receivers on the sender's host hear it.
6. Each datagram carries a sequence number; the receiver delivers in order, drops duplicates and records holes.
7. With two feeds A and B carrying identical sequence numbers, the first copy wins and the second is a duplicate. A packet lost on one line costs nothing.
8. Holes still open go to a repair server as (first sequence, count) over unicast. Heartbeats carrying the next sequence number expose tail loss.

Code

use std::collections::BTreeMap;
use std::net::Ipv4Addr;

/// RFC 1112: the low 23 bits of the group go into 01:00:5e:00:00:00.
pub fn group_mac(g: Ipv4Addr) -> [u8; 6] {
    assert!(g.is_multicast(), "not class D");
    let o = g.octets();
    [0x01, 0x00, 0x5e, o[1] & 0x7f, o[2], o[3]]
}

/// MoldUDP64 header: session[10], first seq u64 BE, count u16 BE; then (len u16, bytes) blocks.
pub fn parse(p: &[u8]) -> Option<(u64, u16, &[u8])> {
    if p.len() < 20 { return None; }
    Some((u64::from_be_bytes(p[10..18].try_into().ok()?), u16::from_be_bytes([p[18], p[19]]), &p[20..]))
}

fn skip(mut b: &[u8], k: u16) -> Option<&[u8]> { // drop the first k message blocks
    for _ in 0..k {
        let n = u16::from_be_bytes([*b.first()?, *b.get(1)?]) as usize;
        b = b.get(2 + n..)?;
    }
    Some(b)
}

/// Feed packets from line A and line B alike; each message comes out once, in order.
pub struct Arbiter {
    next: u64,                            // next sequence number owed to the application
    ahead: BTreeMap<u64, (u16, Vec<u8>)>, // packets beyond a hole, by first seq
}

impl Arbiter {
    pub fn new(first: u64) -> Self { Arbiter { next: first, ahead: BTreeMap::new() } }

    pub fn on_packet(&mut self, p: &[u8], out: &mut impl FnMut(u64, u16, &[u8])) {
        let Some((seq, count, body)) = parse(p) else { return };
        if count == 0 || count == 0xFFFF { return; } // heartbeat / end of session
        if seq > self.next { self.ahead.entry(seq).or_insert((count, body.to_vec())); return; }
        self.deliver(seq, count, body, out);
        while let Some(e) = self.ahead.first_entry() { // a hole may just have closed
            if *e.key() > self.next { break; }
            let (seq, (count, body)) = e.remove_entry();
            self.deliver(seq, count, &body, out);
        }
    }

    fn deliver(&mut self, seq: u64, count: u16, body: &[u8], out: &mut impl FnMut(u64, u16, &[u8])) {
        let end = seq + count as u64;
        if end <= self.next { return; }   // duplicate from the other line
        let k = (self.next - seq) as u16; // overlap with what we already delivered
        if let Some(rest) = skip(body, k) { out(self.next, count - k, rest); self.next = end; }
    }

    /// Holes as (first_seq, count) re-requests; `upto` = next seq a heartbeat announced.
    pub fn gaps(&self, upto: u64) -> Vec<(u64, u16)> {
        let (mut v, mut at) = (vec![], self.next);
        for (s, c) in self.ahead.iter().map(|(&s, (c, _))| (s, *c)).chain([(upto, 0)]) {
            while at < s { let n = (s - at).min(u16::MAX as u64) as u16; v.push((at, n)); at += n as u64; }
            at = at.max(s + c as u64);
        }
        v
    }
}

#[cfg(test)]
mod tests {
    use super::*;
    use std::net::{SocketAddrV4, UdpSocket};
    fn pkt(seq: u64, n: u16) -> Vec<u8> { // n one-byte messages, payload = seq
        let mut p = [b"SESSION001".as_slice(), &seq.to_be_bytes(), &n.to_be_bytes()].concat();
        for i in 0..n as u64 { p.extend([0, 1, (seq + i) as u8]); }
        p
    }
    #[test]
    fn mac_mapping() {
        assert_eq!(group_mac("224.0.0.251".parse().unwrap()), [1, 0, 0x5e, 0, 0, 0xfb]);
        // 224.1.1.1 and 239.129.1.1 differ only in the 5 bits the MAC drops
        assert_eq!(group_mac(Ipv4Addr::new(224, 1, 1, 1)), group_mac(Ipv4Addr::new(239, 129, 1, 1)));
    }
    #[test]
    fn a_b_arbitration() {
        let (mut got, mut a) = (vec![], Arbiter::new(1));
        let mut out = |first: u64, n: u16, b: &[u8]| {
            for i in 0..n as usize { assert_eq!(b[3 * i + 2] as u64, first + i as u64); got.push(first + i as u64); }
        };
        a.on_packet(&pkt(1, 2), &mut out); // line A: 1,2
        a.on_packet(&pkt(1, 2), &mut out); // line B: same packet, dropped
        a.on_packet(&pkt(6, 2), &mut out); // A: 6,7 (3..5 lost on A)
        assert_eq!(a.gaps(8), vec![(3, 3)]); // re-request 3 messages from 3
        a.on_packet(&pkt(2, 4), &mut out); // B: 2..5 overlaps 2, fills the hole
        assert_eq!(a.gaps(8), vec![]);
        drop(out);
        assert_eq!(got, (1..=7).collect::<Vec<u64>>());
    }
    #[test]
    fn real_group_on_this_host() {
        let g = Ipv4Addr::new(239, 255, 42, 99);
        let rx = UdpSocket::bind("0.0.0.0:34567").unwrap();
        rx.join_multicast_v4(&g, &Ipv4Addr::UNSPECIFIED).unwrap();
        rx.set_read_timeout(Some(std::time::Duration::from_secs(1))).unwrap();
        let tx = UdpSocket::bind("0.0.0.0:0").unwrap();
        tx.set_multicast_ttl_v4(1).unwrap(); // stays on this link
        tx.set_multicast_loop_v4(true).unwrap();
        tx.send_to(&pkt(1, 1), SocketAddrV4::new(g, 34567)).unwrap();
        let mut buf = [0u8; 64];
        let (n, _) = rx.recv_from(&mut buf).unwrap();
        assert_eq!(parse(&buf[..n]).unwrap().0, 1);
    }
}

The format is MoldUDP64; Arbiter implements steps 6-8. The third test joined 239.255.42.99 with std's join_multicast_v4 on this Mac and passed. SO_REUSEADDR (several listeners on one port) needs socket2; std has no setter. Two feeds with sequence numbers is common practice, not part of MoldUDP64.

Complexity and constants

• Router and snooping-switch state: one entry per active group (per source and group with SSM).
• 28 significant address bits, 23 mapped: 2^5 = 32 groups per MAC. The NIC cannot tell them apart; the IP layer filters.
• Sender: one sendto per datagram for any receiver count. Keep datagrams under the path MTU; a lost fragment kills the datagram.

Efficiency

• Cost: a MoldUDP64 transmitter sends each packet once regardless of the number of listeners (Nasdaq spec). Measured here on an Apple M4 Pro, one core, release build: 7.9 M datagrams (4 M one-message packets, both lines losing 1 % independently, B three packets behind A) took 0.09-0.11 s, about 70-90 M datagrams/s over three runs, every message delivered once, all holes repaired. Through a real multicast socket pair the same Mac moved 500,000 of 500,000 64-byte datagrams in 7.2 s (about 70 k/s): the bound is the per-datagram socket path, not the arbitration.
• Levers:
  • several messages per datagram — MoldUDP64 aggregates messages to cut traffic (Nasdaq spec) — small messages
  • SSM instead of ASM — no shared trees or rendezvous point (RFC 4607) — one known source per stream
  • IGMP snooping — frames only to joined ports (RFC 4541) — any switched LAN
  • re-request servers near remote receivers — less repair latency and WAN bandwidth (Nasdaq spec) — receivers behind WAN links
  • kernel-bypass receive — removes the syscall that bounded the socket test above — gap-sensitive feeds
• Naive → standard → state of the art:
  • N unicast copies · sender cost and uplink grow with N
  • one multicast stream, sequence numbers, unicast repair · one send, loss repaired per receiver
  • redundant A/B lines with arbitration, NACK-based repair as in PGM (RFC 3208) · loss on one line is free; repair only for double loss

Implementation notes

• On multi-homed hosts join with the interface address, not 0.0.0.0, or the kernel picks one by routing table.
• Linux IP_MULTICAST_ALL defaults to 1: a socket receives every group joined anywhere on the host; set 0 for only its own joins (man-pages). IP_MULTICAST_TTL defaults to 1: a feed crossing a router must raise it.
• Size SO_RCVBUF for bursts; kernel drops are invisible to the application.
• Cap re-requests per hole and per second, or one outage floods the repair server.

Pitfalls

• Reordered is not lost: wait a short grace period before re-requesting what the other line may still deliver.
• A hole never repaired blocks in-order delivery forever: bound the ahead map and resync from a snapshot.
• Snooping switch without a querier: memberships expire and the stream stops (about 260 s with defaults).
• Groups in 224.0.0.0/24 and MACs 01:00:5e:00:00:xx: snooping switches may flood them (RFC 4541 asks vendors about it); keep data out of them.
• Public Internet: ISPs generally do not route multicast between each other (from memory).

Variants and alternatives

In production

• Exchange data feeds: Nasdaq's MoldUDP64 sends a multicast stream with a heartbeat about once per second and a unicast re-request server.
• LAN protocols: mDNS (224.0.0.251, port 5353, RFC 6762), VRRP (224.0.0.18, TTL 255, RFC 9568), SSDP (239.255.255.250:1900, from memory).
• IPTV channels as groups joined on channel change, in operator networks (from memory).

Implementations

Sources

• Deering, Host Extensions for IP Multicasting, RFC 1112, 1989. RFC
• Fenner, IGMP Version 2, RFC 2236, 1997. RFC
• Cain et al., IGMP Version 3, RFC 3376, 2002. RFC
• Vida, Costa (eds.), MLD Version 2 for IPv6, RFC 3810, 2004. RFC
• Holbrook, Cain, Source-Specific Multicast for IP, RFC 4607, 2006. RFC
• Christensen et al., IGMP and MLD Snooping Switches, RFC 4541, 2006. RFC
• Meyer, Administratively Scoped IP Multicast, RFC 2365, 1998. RFC
• Fenner et al., PIM-SM Specification (Revised), RFC 7761, 2016. RFC
• Speakman et al., PGM Reliable Transport Protocol Specification, RFC 3208, 2001. RFC
• Cheshire, Krochmal, Multicast DNS, RFC 6762, 2013. RFC
• Lindem, Dogra, VRRP Version 3 for IPv4 and IPv6, RFC 9568, 2024. RFC
• Crawford, Transmission of IPv6 Packets over Ethernet Networks, RFC 2464, 1998. RFC
• Hinden, Deering, IP Version 6 Addressing Architecture, RFC 4291, 2006. RFC
• IANA, IPv4 Multicast Address Space Registry. registry
• Nasdaq, MoldUDP64 Protocol Specification V1.00. PDF
• Linux man-pages, IP_MULTICAST_ALL(2const). man page