Buffer pools for zero-allocation I/O

Deep dive

How it works

In: a get request, later the release of the handle · Out: an aligned &mut [u8] buffer, or None when the pool is empty · State: a free list (tagged head index) over N preallocated buffers · Cost: O(1); order of 10-30 ns uncontended.

one allocation:  [buf0][buf1][buf2]...[bufN-1]     each `size` bytes, 4 KiB-aligned
free stack:      head(tag|idx) → idx → next[idx] → ... → NIL
get():  pop index (CAS on head)  → Buf{idx}       put(): push index (CAS)  ← Buf dropped

1. Startup: allocate size × count bytes in one aligned block. Zero it once, which also prefaults it. Push all indices.
2. get: pop with compare-exchange. The head also carries a tag that changes on every push and pop. Without it, a thread that read head = A, next = B could succeed after others popped A and B and pushed A back, corrupting the list (ABA).
3. Use: the handle derefs to &mut [u8] for exactly one owner. It can be moved to another thread with the I/O.
4. put: Drop pushes the index back. Release ordering on the push and Acquire on the pop hand the buffer's contents to the next owner without data races.
5. Empty pool: return None. The caller queues the request, applies back pressure to the socket, or sheds load.

Code

Lock-free fixed pool with RAII handles and a read loop that does no per-I/O allocation. Stress-tested with 8 threads × 500 k get/put on macOS, type-checked for Linux.

use std::alloc::{alloc_zeroed, dealloc, Layout};
use std::ops::{Deref, DerefMut};
use std::sync::atomic::{AtomicU32, AtomicU64, Ordering::*};
use std::sync::Arc;

const NIL: u32 = u32::MAX;

/// `count` buffers of `size` bytes carved from one aligned allocation, zeroed once.
/// Free list: a Treiber stack of slot indices. `head` packs (tag << 32 | index); the tag
/// changes on every push/pop, so a stale compare-exchange fails (no ABA).
pub struct BufPool {
    base: *mut u8,
    size: usize,
    layout: Layout,
    next: Box<[AtomicU32]>, // next[i] = slot below i on the free stack
    head: AtomicU64,
}
unsafe impl Send for BufPool {}
unsafe impl Sync for BufPool {} // slots are handed out exclusively, see get/put

impl BufPool {
    /// align = 4096 makes every buffer valid for O_DIRECT if size is a multiple of 4096.
    pub fn new(size: usize, count: u32, align: usize) -> Arc<BufPool> {
        assert!(count > 0 && count < NIL && size > 0 && size % align == 0);
        let layout = Layout::from_size_align(size * count as usize, align).unwrap();
        let base = unsafe { alloc_zeroed(layout) };
        assert!(!base.is_null(), "out of memory");
        let next = (0..count).map(|i| AtomicU32::new(if i + 1 < count { i + 1 } else { NIL })).collect();
        Arc::new(BufPool { base, size, layout, next, head: AtomicU64::new(0) })
    }

    /// A free buffer, or None when all are in use: the caller waits or sheds load
    /// instead of allocating, which bounds memory.
    pub fn get(self: &Arc<Self>) -> Option<Buf> {
        let mut h = self.head.load(Acquire);
        loop {
            let idx = h as u32;
            if idx == NIL { return None; }
            // May read a slot another thread just took; then the tag differs and the CAS fails.
            let below = self.next[idx as usize].load(Relaxed);
            let new = ((h >> 32) + 1) << 32 | below as u64;
            match self.head.compare_exchange_weak(h, new, Acquire, Acquire) {
                Ok(_) => return Some(Buf { pool: Arc::clone(self), idx }), // refcount bump, no allocation
                Err(cur) => h = cur,
            }
        }
    }

    fn put(&self, idx: u32) {
        let mut h = self.head.load(Relaxed);
        loop {
            self.next[idx as usize].store(h as u32, Relaxed);
            let new = ((h >> 32) + 1) << 32 | idx as u64;
            // Release: our writes to the buffer happen-before the next owner's Acquire pop.
            match self.head.compare_exchange_weak(h, new, Release, Relaxed) {
                Ok(_) => return,
                Err(cur) => h = cur,
            }
        }
    }
}

impl Drop for BufPool {
    fn drop(&mut self) { unsafe { dealloc(self.base, self.layout) } } // runs after the last Buf is gone
}

/// Exclusive handle to one slot; returns it to the pool on drop. Send: hand it to an
/// I/O thread or across a channel and the buffer travels with it.
pub struct Buf { pool: Arc<BufPool>, idx: u32 }

impl Deref for Buf {
    type Target = [u8];
    fn deref(&self) -> &[u8] {
        let p = &*self.pool;
        unsafe { std::slice::from_raw_parts(p.base.add(self.idx as usize * p.size), p.size) }
    }
}
impl DerefMut for Buf {
    fn deref_mut(&mut self) -> &mut [u8] {
        let p = &*self.pool;
        unsafe { std::slice::from_raw_parts_mut(p.base.add(self.idx as usize * p.size), p.size) }
    }
}
impl Drop for Buf {
    fn drop(&mut self) { self.pool.put(self.idx) }
}

/// Steady-state read loop with zero heap allocations per I/O.
pub fn checksum_file(f: &std::fs::File, pool: &Arc<BufPool>) -> std::io::Result<u64> {
    use std::os::unix::fs::FileExt;
    let (mut off, mut sum) = (0u64, 0u64);
    loop {
        let mut b = pool.get().expect("pool exhausted"); // a server would queue the request instead
        let n = f.read_at(&mut b, off)?;
        if n == 0 { return Ok(sum); }
        sum = b[..n].iter().fold(sum, |s, &x| s.wrapping_mul(31).wrapping_add(x as u64));
        off += n as u64;
    } // b dropped here: slot back on the free stack
}

Complexity and constants

• Uncontended get + put: two atomic RMWs plus an Arc refcount increment and decrement, order of 10-30 ns. Under heavy multi-core contention the head line ping-pongs: order of 100 ns per operation. Shard it (see below).
• Memory: exactly size × count plus 4 B per slot. Peak memory is fixed at startup.
• A fresh mmap-backed 1 MiB allocation (above the malloc mmap threshold) instead costs 256 page faults with zeroing (4 KiB pages), order of 100 µs, plus a TLB shootdown on free.

Efficiency

• Cost: get plus put is two atomic read-modify-writes and an Arc count: order of 10-30 ns uncontended, order of 100 ns under heavy multi-core contention. A fresh mmap-backed 1 MiB buffer costs 256 page faults, order of 100 µs, plus a TLB shootdown on free. Memory is size × count plus 4 B per slot. Bound: latency of the head cache line.
• Levers:
  • preallocated pool instead of per-I/O allocation — order of 100 µs → order of 10-30 ns per 1 MiB buffer (figures above) — buffers above the malloc mmap threshold
  • per-core pools or a per-thread cache — removes the order-of-100-ns contention on the shared head — many cores
  • io_uring registered buffers (READ_FIXED) — kernel skips page pinning on every I/O (io_uring_register(2)) — Linux, high I/O rates
  • huge pages for the pool (MADV_HUGEPAGE) — fewer TLB misses — large pools
• Naive → standard → state of the art:
• naive: fresh Vec per read · order of 100 µs per 1 MiB when malloc uses mmap
• standard: malloc with thread caches, or a reused Vec · order of 10-50 ns for small buffers
• state of the art: sharded, aligned, prefaulted pool with registered buffers · zero allocations, order of 10 ns per get and put

Implementation notes

• Sizing: count ≥ maximum in-flight I/Os (queue depth) + buffers held downstream. size = your I/O unit: 4-16 KiB for random page I/O, 64 KiB-1 MiB for sequential streams.
• Sharding: one pool per core or thread (no atomics needed), or a small per-thread cache in front of the shared stack. This is the magazine layer of the slab allocator. Pad shared heads to 128 B (false sharing).
• NUMA: allocate and first-touch each pool on the node where its threads run.
• Huge pages: back a large pool with mmap + madvise(MADV_HUGEPAGE) to cut TLB misses (virtual memory).
• io_uring fixed buffers: register the whole pool once (IORING_REGISTER_BUFFERS, each iovec up to 1 GiB) and use READ_FIXED/WRITE_FIXED with the slot index. The kernel then skips pinning pages on every I/O. Registered memory counts against RLIMIT_MEMLOCK. For receives on many sockets, provided buffer rings (IORING_REGISTER_PBUF_RING, Linux 5.19+) let the kernel pick a buffer only when data arrives (io_uring).
• Variable sizes: several pools with power-of-two size classes, or a slab allocator. For per-request scratch memory freed all at once, use an arena.
• Rust idioms: reusing one Vec<u8> with clear() keeps its capacity, the simplest pool of size 1. bytes::BytesMut reclaims its storage on reserve when no frozen views remain.

Pitfalls

• Leaking stale data: a recycled buffer holds the previous request's bytes. Track the valid length and never send buf[..] blindly. Zero buffers if they cross trust boundaries.
• Holding buffers too long: a slow consumer drains the pool and stalls all I/O. Export "free buffers" as a metric and bound queues downstream.
• Unbounded fallback: "allocate if the pool is empty" turns overload into an OOM. Prefer waiting or rejecting.
• ABA in a hand-written lock-free free list without a tag or epoch-based reclamation.
• Use after return: with raw indices instead of RAII handles, a buffer returned while an async I/O still writes into it corrupts the next owner. Return buffers only on completion.
• Alignment: Vec<u8> is 1-byte aligned. O_DIRECT with it fails with EINVAL or silently falls back.

Variants and alternatives

In production

Netty PooledByteBufAllocator, NGINX buffer chains, SPDK and DPDK mempools (hugepage-backed, per-core caches), TigerBeetle (all memory allocated at startup, none afterwards), ScyllaDB/Seastar per-core memory, io_uring-based servers with registered buffers.

Implementations

Sources

• Jeff Bonwick, The Slab Allocator: An Object-Caching Kernel Memory Allocator, USENIX Summer, 1994. paper
• Jeff Bonwick, Jonathan Adams, Magazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources, USENIX ATC, 2001. paper
• R. Kent Treiber, Systems Programming: Coping with Parallelism, IBM Research Report RJ 5118, 1986.
• Jens Axboe, Efficient IO with io_uring, 2019. pdf
• Linux man pages: mallopt(3) (M_MMAP_THRESHOLD), io_uring_register(2). mallopt · io_uring_register