Low-latency hot path

Deep dive

Decision table

Code

// Linux, x86-64. Cargo: libc = "0.2"
use std::cell::UnsafeCell;
use std::io;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering::{Acquire, Relaxed, Release}};

pub fn pin_to_cpu(cpu: usize) -> io::Result<()> {
    unsafe {
        let mut set: libc::cpu_set_t = std::mem::zeroed();
        libc::CPU_ZERO(&mut set);
        libc::CPU_SET(cpu, &mut set);
        if libc::sched_setaffinity(0, std::mem::size_of::<libc::cpu_set_t>(), &set) != 0 {
            return Err(io::Error::last_os_error());
        }
    }
    Ok(())
}

pub fn lock_all_memory() -> io::Result<()> {
    if unsafe { libc::mlockall(libc::MCL_CURRENT | libc::MCL_FUTURE) } != 0 { return Err(io::Error::last_os_error()); }
    Ok(())
}

/// Touch every page so the first real write doesn't page-fault.
pub fn prefault(buf: &mut [u8]) {
    for i in (0..buf.len()).step_by(4096) { unsafe { std::ptr::write_volatile(&mut buf[i], 0) } }
}

#[repr(align(128))] // x86 adjacent-line prefetch works on 128-byte pairs; Apple M-series lines are 128 B
struct Line<T>(T);

/// Single-producer single-consumer ring, preallocated at startup.
pub struct Ring<T> { head: Line<AtomicUsize>, tail: Line<AtomicUsize>, mask: usize, buf: Box<[UnsafeCell<T>]> }
unsafe impl<T: Send> Sync for Ring<T> {}

impl<T: Copy + Default> Ring<T> {
    pub fn new(cap: usize) -> Self {
        assert!(cap.is_power_of_two());
        Ring { head: Line(AtomicUsize::new(0)), tail: Line(AtomicUsize::new(0)), mask: cap - 1,
               buf: (0..cap).map(|_| UnsafeCell::new(T::default())).collect() }
    }
    pub fn push(&self, v: T) -> bool {                       // producer thread only
        let t = self.tail.0.load(Relaxed);
        if t.wrapping_sub(self.head.0.load(Acquire)) == self.buf.len() { return false; }
        unsafe { *self.buf[t & self.mask].get() = v; }
        self.tail.0.store(t.wrapping_add(1), Release);       // publish the slot
        true
    }
    pub fn pop(&self) -> Option<T> {                         // consumer thread only
        let h = self.head.0.load(Relaxed);
        if h == self.tail.0.load(Acquire) { return None; }
        let v = unsafe { *self.buf[h & self.mask].get() };
        self.head.0.store(h.wrapping_add(1), Release);       // free the slot
        Some(v)
    }
}

/// Fixed log2 histogram of cycle counts: no allocation, one increment per sample.
pub struct Hist { pub buckets: [u64; 64] }
impl Hist {
    #[inline] pub fn record(&mut self, cycles: u64) { self.buckets[((64 - cycles.leading_zeros()) as usize).min(63)] += 1; }
}

#[inline] fn cycles() -> u64 { unsafe { core::arch::x86_64::_rdtsc() } }

/// The hot loop: pinned, never sleeps, never allocates.
pub fn run<T: Copy + Default>(cpu: usize, input: &Ring<T>, stop: &AtomicBool, hist: &mut Hist, mut on_msg: impl FnMut(&T)) -> io::Result<()> {
    pin_to_cpu(cpu)?;
    while !stop.load(Relaxed) {
        match input.pop() {
            Some(m) => { let t0 = cycles(); on_msg(&m); hist.record(cycles() - t0); }
            None => std::hint::spin_loop(),                  // PAUSE
        }
    }
    Ok(())
}

Startup order: parse config → allocate every pool, ring, book and buffer for the peak → prefault → mlockall → pin threads → warm up (run synthetic messages through the full path so code and data are cached and branch predictors trained) → open for business.

Efficiency

Cost: the bound is tail latency from unpredictable events, not average compute. Order of: a DRAM miss 100 ns against 1-2 ns for L1; a minor page fault or context switch 1 µs or more; a syscall 100 ns before mitigations; kernel UDP receive several µs per packet against order of 1 µs wire-to-application with kernel bypass; an IRQ on a hot core is a multi-µs stall.

Levers

• Pinned threads on isolated cores, IRQs steered away — no preemption or timer ticks — stages with a microsecond budget.
• Preallocate, prefault, mlockall, huge pages — no page faults or allocator locks — from startup on.
• Kernel bypass (ef_vi/Onload, DPDK, AF_XDP) — several µs down to order of 1 µs — packet-rate-bound paths.
• One writer per cache line (SPSC rings) — no locks, no false sharing — stage-to-stage hand-off.
• Binary logging into a ring, formatting off-path — no format! or write() on the hot path.

Cost of the levers: busy polling holds a core at 100 % and C-state limits raise idle power.

Naive → standard → state of the art

• Naive: thread per connection, blocking sockets, malloc per message · tail of 10-100× the median
• Standard: epoll or io_uring with normal threads · fine for millisecond budgets
• State of the art: isolated pinned busy-polling threads, kernel bypass, SPSC rings · order of 1 µs wire-to-application

Implementation notes

• Numbers to budget with (order of, modern x86 server at 3 GHz): L1 hit 1-2 ns, L2 4-5 ns, L3 15-40 ns, DRAM 80-120 ns; uncontended atomic RMW 5-20 ns, contended cache line transfer between cores 50-100+ ns; vDSO clock_gettime 20 ns; syscall 100 ns-1 µs; context switch 2-5 µs including cache refill.
• Cache discipline: one writer per cache line; keep the hot working set (book top, open orders, risk counters for the symbols you trade) in L1/L2; use struct-of-arrays and indices instead of pointers; no HashMap with SipHash on the hot path.
• No allocation: preallocated pools and arenas reset per message or per day; reuse output buffers; avoid Vec growth, String, Box::new, format!. Check with a counting global allocator in tests that the steady state allocates zero bytes.
• Huge pages: reserve explicit 2 MB pages (vm.nr_hugepages, MAP_HUGETLB) for big tables; transparent huge pages can trigger compaction stalls, so many shops set THP to madvise or never.
• Kernel isolation: isolcpus= (documented as deprecated in favour of cpusets, still widely used), nohz_full= (needs CONFIG_NO_HZ_FULL; a residual tick of about 1 Hz remains), rcu_nocbs=; disable irqbalance or exclude hot cores from it.
• PAUSE cost: on Intel Skylake-generation and later cores PAUSE takes order of 100+ cycles (was about 10); spin on a plain load first, then PAUSE, if the reaction time matters at the cycle level.
• NUMA: NIC, memory and hot cores on the same socket; allocate with first-touch from the pinned thread.
• Branching: keep the common case straight-line; move error handling to #[cold] functions; precompute message templates (FIX/binary) and patch only price, quantity and sequence fields.
• Measurement: hardware timestamps on the NIC for wire-to-wire; rdtsc (check constant_tsc and nonstop_tsc in /proc/cpuinfo) for in-process spans; HDR or log histograms; record max per second, not only percentiles.

Pitfalls

• A "temporary" debug println! or log formatting call on the hot path: syscall + lock + allocation.
• Pinning two busy-polling threads to SMT siblings of one core: they halve each other's throughput and share L1.
• Allocating lazily on first use (HashMap growth at the market open, first order of the day page-faulting).
• Using Mutex "because it's uncontended": the slow path still parks the thread on contention spikes.
• Timer or watchdog threads landing on isolated cores because their affinity was inherited from the process; set affinity per thread.
• Benchmarks without warm-up, or measuring averages: the cold first message after a quiet period is the one that matters.
• SCHED_FIFO busy loops on non-isolated cores starving kernel threads (RCU stalls, hung tasks).

Variants and alternatives

In production

Trading firms' feed handlers and order gateways; DPDK, which its site lists as running in cloud infrastructure, telecom networks and financial exchanges; JACK and PortAudio ring buffers for real-time audio; LMAX Disruptor (busy-spin wait strategy); Solarflare/AMD Onload and ef_vi, DPDK; Red Hat's tuned latency-performance and network-latency profiles automate much of the OS tuning.

Implementations

Sources

• Ulrich Drepper, What Every Programmer Should Know About Memory, 2007. pdf
• Linux kernel documentation, NO_HZ: Reducing Scheduling-Clock Ticks and kernel-parameters (isolcpus, nohz_full, rcu_nocbs, irqaffinity). NO_HZ · kernel-parameters
• Intel, Intel 64 and IA-32 Architectures Optimization Reference Manual (PAUSE latency, cache hierarchy).
• Brendan Gregg, Systems Performance: Enterprise and the Cloud, 2nd ed., Addison-Wesley, 2020. book
• Martin Thompson et al., Disruptor: High performance alternative to bounded queues for exchanging data between concurrent threads, LMAX, 2011. paper · code