Zero-copy I/O

Deep dive

How it works

In: source file or buffer, destination socket, offset and length · Out: the bytes on the wire with no CPU copy · State: per connection file offset; for MSG_ZEROCOPY the range of sends awaiting completion · Cost: 0 CPU copies per byte; O(pages) reference counting per call.

read+write:  disk -DMA-> page cache -copy-> user buf -copy-> skb -DMA-> NIC   (2 copies, 2 syscalls)
sendfile:    disk -DMA-> page cache ----page refs in skb frags----DMA-> NIC   (0 copies, 1 syscall)
MSG_ZEROCOPY: user pages pinned -> skb frags -DMA-> NIC; kernel notifies when pages are free

1. sendfile(out, in, &off, n): kernel builds skbs whose fragments reference page-cache pages (refcount++), up to 0x7ffff000 bytes per call. Requires a NIC with scatter-gather and checksum offload (all modern NICs).
2. splice/tee/vmsplice: move page references between a pipe and any fd; build socket→file or socket→socket proxies (HAProxy) without user copies.
3. MSG_ZEROCOPY: enable SO_ZEROCOPY, then send(..., MSG_ZEROCOPY). The kernel pins your pages; you must not modify the buffer until a notification on MSG_ERRQUEUE reports that send's sequence number range as done. If the kernel had to copy anyway (loopback, no SG), the notification says so.
4. kTLS: user space does the TLS handshake, then hands keys to the socket (setsockopt(TCP_ULP, "tls"), TLS_TX). The kernel encrypts records during sendfile; with NIC TLS offload the NIC encrypts and the path is copy-free again.
5. Receive side: TCP_ZEROCOPY_RECEIVE (4.18) maps received payload pages into user memory; needs page-sized payloads (4 KB MTU or header-split NICs) and pays TLB shootdowns, so it only wins for large flows.

Code

// Cargo: libc = "0.2". Linux only.
use std::{fs::File, io, net::TcpStream, os::fd::AsRawFd};

/// Send `len` bytes of `file` starting at `off` (blocking socket). Returns the new offset.
pub fn send_file(sock: &TcpStream, file: &File, mut off: i64, len: usize) -> io::Result<i64> {
    let mut left = len;
    while left > 0 {
        let chunk = left.min(0x7fff_f000); // kernel max per call
        let n = unsafe { libc::sendfile(sock.as_raw_fd(), file.as_raw_fd(), &mut off, chunk) };
        if n < 0 {
            let e = io::Error::last_os_error();
            if e.kind() == io::ErrorKind::Interrupted { continue; }
            return Err(e); // non-blocking socket: on EAGAIN wait for EPOLLOUT and resume at `off`
        }
        if n == 0 { return Err(io::ErrorKind::UnexpectedEof.into()); } // file was truncated
        left -= n as usize; // kernel already advanced `off`
    }
    Ok(off)
}

const SO_ZEROCOPY: i32 = 60;
const MSG_ZEROCOPY: i32 = 0x400_0000;
const SO_EE_ORIGIN_ZEROCOPY: u8 = 5;
const SO_EE_CODE_ZEROCOPY_COPIED: u8 = 1;

#[repr(C)]
struct SockExtendedErr { errno: u32, origin: u8, ty: u8, code: u8, pad: u8, info: u32, data: u32 }

pub fn enable_zerocopy(fd: i32) -> io::Result<()> {
    let one: libc::c_int = 1;
    let r = unsafe {
        libc::setsockopt(fd, libc::SOL_SOCKET, SO_ZEROCOPY, &one as *const _ as *const _, 4)
    };
    if r < 0 { Err(io::Error::last_os_error()) } else { Ok(()) }
}

/// Each successful call gets the next sequence number (0, 1, 2, ...). `buf` must stay
/// untouched until `reap` reports that number as completed.
pub fn send_zc(fd: i32, buf: &[u8]) -> io::Result<usize> {
    let n = unsafe { libc::send(fd, buf.as_ptr() as *const _, buf.len(), MSG_ZEROCOPY) };
    if n < 0 { Err(io::Error::last_os_error()) } else { Ok(n as usize) }
}

/// Returns Some((first, last, copied)): sends first..=last are done and their buffers reusable.
/// `copied` = the kernel fell back to copying, so zero-copy is not helping on this path.
pub fn reap(fd: i32) -> io::Result<Option<(u32, u32, bool)>> {
    let mut ctrl = [0u64; 16]; // u64 for cmsg alignment
    let mut msg: libc::msghdr = unsafe { std::mem::zeroed() };
    msg.msg_control = ctrl.as_mut_ptr() as *mut _;
    msg.msg_controllen = std::mem::size_of_val(&ctrl) as _;
    if unsafe { libc::recvmsg(fd, &mut msg, libc::MSG_ERRQUEUE | libc::MSG_DONTWAIT) } < 0 {
        let e = io::Error::last_os_error();
        return if e.kind() == io::ErrorKind::WouldBlock { Ok(None) } else { Err(e) };
    }
    let cm = unsafe { libc::CMSG_FIRSTHDR(&msg) };
    if cm.is_null() { return Ok(None); }
    let ee = unsafe { &*(libc::CMSG_DATA(cm) as *const SockExtendedErr) };
    if ee.errno != 0 || ee.origin != SO_EE_ORIGIN_ZEROCOPY {
        return Err(io::Error::from_raw_os_error(ee.errno as i32));
    }
    Ok(Some((ee.info, ee.data, ee.code & SO_EE_CODE_ZEROCOPY_COPIED != 0)))
}

Complexity and constants

• memcpy of cold data: order of 10-20 GB/s per core; a 2-copy read/write path at 100 Gbit/s needs order of 2 cores just for copies, plus cache pollution that slows everything else.
• sendfile cost is per page (refcount, skb frag): order of 10s of ns per 4 KB, vs order of 200-400 ns to copy 4 KB.
• MSG_ZEROCOPY adds page pinning plus one notification per send (notifications coalesce ranges); break-even around 10 KB per call, per the kernel docs.
• The optmem_max sysctl bounds outstanding notifications; exceeding it returns ENOBUFS.

Efficiency

• Cost: read + write copies each byte twice; memcpy of cold data runs at order of 10-20 GB/s per core, so 100 Gbit/s needs order of 2 cores just for copies, plus cache pollution. sendfile costs order of 10s of ns per 4 KB page against order of 200-400 ns to copy it. The bound moves to NIC and memory bandwidth, and per-call overhead for small sends.
• Levers:
  • sendfile with TCP_CORK / tcp_nopush — header and file data share full packets (nginx) — static files
  • kTLS, with NIC TLS offload — keeps the copy-free path under TLS; Netflix about 400 Gbit/s per box (Gallatin 2021) — HTTPS and video
  • MSG_ZEROCOPY only for sends of about 10 KB and up — below that, pinning and completions cost more than the copy (kernel docs) — large user buffers
  • io_uring SEND_ZC — completions on the completion queue, fewer syscalls — high send rates
  • splice between sockets — proxying without user copies (HAProxy) — L4/L7 relays
• Naive → standard → state of the art:
• read + write loop · 2 CPU copies and 2 syscalls per chunk
• sendfile with cork · 0 copies, 1 syscall per up to 0x7ffff000 bytes
• sendfile + kTLS with NIC TLS offload · 0 CPU copies and no CPU encryption

Implementation notes

• nginx: sendfile on; tcp_nopush on; (TCP_CORK so headers and file data share full packets); sendfile_max_chunk 512k stops one fast client from monopolizing a worker.
• Headers + file: write the header with MSG_MORE (or cork), then sendfile, so the first packet is full.
• Non-blocking sockets: sendfile returns partial counts and EAGAIN; keep off per connection and resume on EPOLLOUT.
• io_uring offers IORING_OP_SPLICE, SEND_ZC and SENDMSG_ZC with completions in the CQ instead of the error queue.
• copy_file_range (4.5) is the file→file analogue; on XFS/Btrfs it reflinks and on NFS it does a server-side copy.
• Kafka's consumer path relies on sendfile (FileChannel.transferTo); with SSL enabled the JVM encrypts in user space and the zero-copy path is lost.

Pitfalls

• Reusing a buffer after send_zc returns but before its notification: the peer receives the new contents.
• sendfile reads the live page cache: if the file is modified during transmission, the client gets a mix. Serve immutable files (write new, rename).
• Truncated files make sendfile return 0 bytes; with mmap + write the same race is a SIGBUS.
• MSG_ZEROCOPY on loopback or a NIC without SG silently copies (the copied flag); measure before keeping the complexity.
• TLS in user space (OpenSSL in the app) forces a read into user memory; use kTLS (SSL_OP_ENABLE_KTLS in OpenSSL 3) to keep sendfile.

Variants and alternatives

In production

nginx, Apache (EnableSendfile), Kafka, Varnish, HAProxy (splice), Netflix Open Connect (FreeBSD sendfile + kTLS), Linux kTLS offload on Mellanox/NVIDIA ConnectX NICs.

Implementations

Sources

• de Bruijn, Dumazet, sendmsg copy avoidance with MSG_ZEROCOPY, Netdev 2.1, 2017; Linux docs Documentation/networking/msg_zerocopy.rst. docs
• Linux man-pages, sendfile(2), splice(2), vmsplice(2), copy_file_range(2). sendfile · splice · vmsplice · copy_file_range
• Gallatin, Serving Netflix Video at 400Gb/s on FreeBSD, EuroBSDCon, 2021. slides
• Pai, Druschel, Zwaenepoel, IO-Lite: A Unified I/O Buffering and Caching System, OSDI, 1999. paper