Reliable UDP

Deep dive

How it works

In: payload messages with a reliability class, plus received packet headers · Out: packets with (seq, ack, ack_bits), acked messages, RTT and loss stats · State: send ring and receive ring of 1024 entries, message queues · Cost: O(1) per packet plus 33 ring lookups.

1. Send: seq = local_seq++; header ack = newest received remote seq; bit i of ack_bits = received(ack − 1 − i). Record sent[seq] = {time, message ids} in a ring of 1024.
2. Receive: drop the packet if it is older than the receive window (1024) or a duplicate; record received[seq]; update newest remote seq if it's greater (wrap-aware).
3. Process acks: for ack and each set bit, if sent[s] exists and isn't acked yet, mark it acked, update RTT from its send time, and mark its reliable messages delivered.
4. Reliable messages: queue with message ids. When building a packet, include unacked messages not sent in the last ~RTT (or every packet for lowest latency), up to the size budget. The receiver delivers reliable-ordered messages by message id from a receive buffer, ignores duplicates.
5. Loss/RTT stats: a sent packet not acked after ~1 s counts as lost; smooth RTT with an EWMA (10 %).

packet 100: seq=100 ack=57 bits=...1111   (I got 57, 56, 55, ...)
packet 101: seq=101 ack=58 bits=...1110   (got 58; 57 still acked via bit 0... etc.)
receiver of 101 learns that 57 and 58 arrived even if packet 100 was lost

Code

/// Wrap-aware "a is newer than b" for u16 sequence numbers.
pub fn seq_gt(a: u16, b: u16) -> bool { a != b && a.wrapping_sub(b) < 32768 }

const N: usize = 1024; // ring size: window of sequence numbers tracked

#[derive(Clone, Copy, Debug, PartialEq)]
pub struct Header { pub seq: u16, pub ack: u16, pub ack_bits: u32 }

impl Header {
    pub fn write(&self, out: &mut Vec<u8>) {
        out.extend_from_slice(&self.seq.to_le_bytes());
        out.extend_from_slice(&self.ack.to_le_bytes());
        out.extend_from_slice(&self.ack_bits.to_le_bytes());
    }
    pub fn read(b: &[u8]) -> Option<Header> {
        Some(Header { seq: u16::from_le_bytes(b.get(0..2)?.try_into().ok()?),
                      ack: u16::from_le_bytes(b.get(2..4)?.try_into().ok()?),
                      ack_bits: u32::from_le_bytes(b.get(4..8)?.try_into().ok()?) })
    }
}

/// Fixed-size map from sequence number to T, valid only for the last N sequences.
pub struct SeqBuf<T> { slots: Vec<Option<(u16, T)>> }
impl<T> SeqBuf<T> {
    pub fn new() -> Self { SeqBuf { slots: (0..N).map(|_| None).collect() } }
    pub fn insert(&mut self, s: u16, v: T) { self.slots[s as usize % N] = Some((s, v)); }
    pub fn get_mut(&mut self, s: u16) -> Option<&mut T> {
        match &mut self.slots[s as usize % N] { Some((k, v)) if *k == s => Some(v), _ => None }
    }
    pub fn contains(&self, s: u16) -> bool { matches!(&self.slots[s as usize % N], Some((k, _)) if *k == s) }
}

pub struct Sent { pub time: f64, pub acked: bool, pub msg_ids: Vec<u16> }

pub struct Endpoint {
    local_seq: u16,
    remote_seq: Option<u16>,  // newest received
    received: SeqBuf<()>,
    pub sent: SeqBuf<Sent>,
    pub rtt: f64,
}

impl Endpoint {
    pub fn new() -> Self {
        Endpoint { local_seq: 0, remote_seq: None, received: SeqBuf::new(), sent: SeqBuf::new(), rtt: 0.1 }
    }

    /// Build the header for an outgoing packet carrying `msg_ids` reliable messages.
    pub fn send(&mut self, now: f64, msg_ids: Vec<u16>) -> Header {
        let seq = self.local_seq;
        self.local_seq = seq.wrapping_add(1);
        self.sent.insert(seq, Sent { time: now, acked: false, msg_ids });
        // Nothing received yet: ack 65535, which matches no `sent` entry while local seqs start at 0.
        let ack = self.remote_seq.unwrap_or(u16::MAX);
        let mut bits = 0u32;
        for i in 0..32u16 {
            if self.received.contains(ack.wrapping_sub(1 + i)) { bits |= 1 << i; }
        }
        Header { seq, ack, ack_bits: bits }
    }

    /// Returns reliable message ids that are now known delivered; None = drop packet.
    pub fn receive(&mut self, now: f64, h: Header) -> Option<Vec<u16>> {
        if let Some(r) = self.remote_seq {
            if self.received.contains(h.seq) { return None; } // duplicate
            if seq_gt(r, h.seq) && r.wrapping_sub(h.seq) as usize >= N { return None; } // too old
        }
        self.received.insert(h.seq, ());
        if self.remote_seq.map_or(true, |r| seq_gt(h.seq, r)) { self.remote_seq = Some(h.seq); }
        let mut delivered = Vec::new();
        for i in 0..33u16 {
            let s = h.ack.wrapping_sub(i);
            if i > 0 && h.ack_bits & (1 << (i - 1)) == 0 { continue; }
            if let Some(p) = self.sent.get_mut(s) {
                if !p.acked {
                    p.acked = true;
                    self.rtt += (now - p.time - self.rtt) * 0.1; // EWMA
                    delivered.extend(p.msg_ids.drain(..));
                }
            }
        }
        Some(delivered)
    }
}

#[test]
fn acks_survive_loss_and_wrap() {
    assert!(seq_gt(0, 65535) && !seq_gt(65535, 0) && seq_gt(5, 1));
    let (mut a, mut b) = (Endpoint::new(), Endpoint::new());
    a.local_seq = 65530; // exercise wraparound
    let mut delivered = Vec::new();
    for t in 0..20u16 {
        let h = a.send(t as f64, vec![t]);           // message t rides in packet t
        if t % 3 != 1 { b.receive(t as f64, h); }    // lose every third packet a -> b
        let back = b.send(t as f64, vec![]);
        delivered.extend(a.receive(t as f64 + 0.05, back).unwrap());
    }
    // Every packet that arrived was acked back, even across the u16 wrap.
    let expect: Vec<u16> = (0..19).filter(|t| t % 3 != 1).collect();
    assert_eq!(delivered, expect);
}

Complexity and constants

• O(1) per packet send/receive plus 33 ring lookups for acks; ring memory = 1024 × entry size per direction.
• Header overhead: 8 bytes + UDP/IP 28 bytes (IPv4) or 48 bytes (IPv6); at 60 packets/s that is 17-27 kbit/s before payload. Keep packet rates modest (20-60 Hz) and batch messages.
• Resend latency for reliable messages: one RTT plus the send interval with "resend every packet until acked"; versus TCP's RTO (≥ 200 ms on Linux) on tail loss.
• Safe payload: about 1200 bytes (QUIC's minimum path MTU); Ethernet allows 1472 for IPv4 UDP, but tunnels and mobile networks shrink it.

Efficiency

Cost: O(1) per packet; per-direction ring of 1024 entries. Overhead per packet is 8 B header plus 28 B UDP/IPv4 (48 B IPv6): 17-27 kbit/s at 60 packets/s before payload. Reliable data resent every packet until acked recovers in about one RTT plus the send interval, versus TCP's RTO of at least 200 ms on Linux for tail loss. Bound: RTT and packet rate.

Levers:

• Piggyback acks via the 32-bit history — no ack-only packets, each ack repeated up to 33 times — always.
• Keep payload near 1200 B — avoids IP fragmentation (QUIC's minimum path MTU) — mobile and tunnelled paths.
• Selective retransmission plus a faster RTO growth (x1.5, not x2) — KCP reports about 30-40% lower average latency for 10-20% more bandwidth than TCP (KCP README) — latency-sensitive links.
• recvmmsg/sendmmsg or GSO — fewer syscalls per packet — servers with many clients.
• Receive-side latency window sized to the retransmission time — hides loss in streams (SRT) — live media.

Naive → standard → state of the art:

• TCP for everything · head-of-line stalls of at least an RTT per loss
• Sequence + ack bitfield with per-message reliability · one RTT recovery, 8 B header
• QUIC with DATAGRAM frames or kernel-bypass UDP messaging (Aeron) · standard, encrypted, congestion-controlled

Implementation notes

• Fragment large reliable blocks (map data, initial state) into ≤ 1 KB slices with a slice bitfield and resend only missing slices; cap in-flight slices to avoid flooding.
• Message ids are u16 with the same wrap logic; reliable-ordered receive window must exceed the max in-flight messages.
• Congestion: games run at fixed rates, but still back off on sustained loss or RTT growth (drop send rate from 60 to 30 Hz, reduce snapshot size); Gaffer's simple "good/bad mode" toggle is a reasonable floor.
• Security: add a connect handshake with a server-issued token (netcode.io connect tokens) to prevent IP spoofing and amplification; encrypt with an AEAD per packet (ChaCha20-Poly1305 or AES-GCM) using the sequence number as part of the nonce.
• Keepalive: send at least one packet every 100-250 ms even when idle so acks flow and NAT bindings stay open; time out after 5-10 s of silence.
• Use recvmmsg/sendmmsg or GSO on servers with many clients; one socket per server process is common.

Pitfalls

• Comparing sequence numbers with >: breaks after 65 536 packets (about 18 minutes at 60 Hz).
• Acking only via dedicated ack packets: one lost ack resends data needlessly; piggyback acks with the 32-bit history.
• Retransmitting stale state (old snapshots) reliably: bandwidth spirals under loss. Only events need reliability; state should be superseded.
• Treating "not acked yet" as "lost" too early triggers spurious resends; use RTT + jitter margin.
• Unbounded reliable queues when a client's link dies: cap and disconnect.

Variants and alternatives

In production

• Games: Quake 3 / id Tech (sequenced unreliable + reliable commands), Valve Source and Steam (GameNetworkingSockets), Unreal Engine (own UDP NetDriver with acks and reliable bunches), ENet-based games, Overwatch, Rocket League.
• Robotics: DDS (the ROS 2 middleware) is implemented on UDP by default and reimplements reliability; ROS 2 offers best-effort and reliable QoS and defaults to reliable.
• Media: SRT is a transport for ultra-low-latency live video with ARQ as its main error recovery; the IETF draft is draft-sharabayko-srt.
• Finance: Aeron, a messaging transport for reliable UDP unicast, multicast and IPC, aims for the lowest and most predictable latency.
• Cloud and web: QUIC (RFC 9000) provides reliable streams over UDP, and RFC 9221 adds unreliable datagrams.

Implementations

Sources

• Fiedler, Reliability and Congestion Avoidance over UDP, gafferongames.com, 2008; Reliable Ordered Messages, 2016; Packet Fragmentation and Reassembly, 2016. post · post · post
• Valve, GameNetworkingSockets, code
• Salzman, ENet documentation, enet.bespin.org. code
• Fiedler, netcode.io specification, code
• Iyengar, Thomson (eds.), QUIC: A UDP-Based Multiplexed and Secure Transport, RFC 9000, 2021. RFC
• Pauly, Kinnear, Schinazi, An Unreliable Datagram Extension to QUIC, RFC 9221, 2022. RFC
• ROS 2 design, Quality of Service policies and ROS on DDS. qos · dds
• The SRT Protocol, IETF Internet-Draft draft-sharabayko-srt-01, 2022. draft · code
• Real Logic, Aeron. code
• skywind3000, KCP, README. code