Zero-copy serialization

Deep dive

How it works

In: a received or memory-mapped byte buffer plus its schema · Out: typed read access to fields, borrowed from the buffer · State: none (the buffer is the object) · Cost: decode O(1); validation O(n) for untrusted input; field access one bounds check and one load.

1. Fixed part: scalars at offsets fixed by the schema (SBE: whole message is fixed-offset; FlatBuffers: a vtable maps field id → offset so absent fields cost nothing).
2. Variable part: strings, vectors and sub-messages live elsewhere in the buffer; the fixed part stores a relative offset (FlatBuffers u32 offsets, Cap'n Proto 64-bit pointers in 8-byte words, rkyv relative pointers).
3. Access: read_u32_le(buf, base + off). With aligned buffers and matching endianness this compiles to one load.
4. Validation (untrusted): one pass checks every offset and length stays inside the buffer, UTF-8 where required, and no cycles; then accessors can skip checks.

SBE-style message (little-endian):
 0       4       8                              8+24n
 |magic  |count n| rec0 (24 B) | rec1 | ... | string table: [u16 len][bytes]...
 rec = | id u64 | price i64 (1e-8 fixed point) | qty u32 | sym_off u32 |

Code

/// Minimal fixed-layout format in the SBE style: write once, read fields in place.
const MAGIC: u32 = 0x4f52_4431; // "ORD1"
const REC: usize = 24;

pub fn write_orders(orders: &[(u64, i64, u32, &str)]) -> Vec<u8> {
    let mut b = Vec::with_capacity(8 + REC * orders.len() + 16 * orders.len());
    b.extend_from_slice(&MAGIC.to_le_bytes());
    b.extend_from_slice(&(orders.len() as u32).to_le_bytes());
    let strings_at = 8 + REC * orders.len();
    let mut strings = Vec::new();
    for &(id, price, qty, sym) in orders {
        b.extend_from_slice(&id.to_le_bytes());
        b.extend_from_slice(&price.to_le_bytes());
        b.extend_from_slice(&qty.to_le_bytes());
        b.extend_from_slice(&((strings_at + strings.len()) as u32).to_le_bytes());
        strings.extend_from_slice(&(sym.len() as u16).to_le_bytes());
        strings.extend_from_slice(sym.as_bytes());
    }
    b.extend_from_slice(&strings);
    b
}

#[derive(Debug)]
pub struct Invalid(&'static str);

/// A validated view over borrowed bytes; no copies, no allocation.
#[derive(Clone, Copy)]
pub struct Orders<'a> { buf: &'a [u8], n: usize }

#[derive(Clone, Copy)]
pub struct Order<'a> { buf: &'a [u8], at: usize }

fn u16_at(b: &[u8], i: usize) -> u16 { u16::from_le_bytes(b[i..i + 2].try_into().unwrap()) }
fn u32_at(b: &[u8], i: usize) -> u32 { u32::from_le_bytes(b[i..i + 4].try_into().unwrap()) }
fn u64_at(b: &[u8], i: usize) -> u64 { u64::from_le_bytes(b[i..i + 8].try_into().unwrap()) }

impl<'a> Orders<'a> {
    /// One O(n) pass so that accessors can never go out of bounds or see bad UTF-8.
    pub fn new(buf: &'a [u8]) -> Result<Self, Invalid> {
        if buf.len() < 8 || u32_at(buf, 0) != MAGIC { return Err(Invalid("header")); }
        let n = u32_at(buf, 4) as usize;
        let end = n.checked_mul(REC).and_then(|x| x.checked_add(8)).ok_or(Invalid("count"))?;
        if end > buf.len() { return Err(Invalid("records")); }
        for i in 0..n {
            let s = u32_at(buf, 8 + i * REC + 20) as usize;
            if s < end || s + 2 > buf.len() { return Err(Invalid("string offset")); }
            let len = u16_at(buf, s) as usize;
            let bytes = buf.get(s + 2..s + 2 + len).ok_or(Invalid("string length"))?;
            std::str::from_utf8(bytes).map_err(|_| Invalid("utf8"))?;
        }
        Ok(Orders { buf, n })
    }
    pub fn len(&self) -> usize { self.n }
    pub fn get(&self, i: usize) -> Order<'a> {
        assert!(i < self.n);
        Order { buf: self.buf, at: 8 + i * REC }
    }
}

impl<'a> Order<'a> {
    pub fn id(&self) -> u64 { u64_at(self.buf, self.at) }
    pub fn price(&self) -> i64 { u64_at(self.buf, self.at + 8) as i64 }
    pub fn qty(&self) -> u32 { u32_at(self.buf, self.at + 16) }
    pub fn symbol(&self) -> &'a str {
        let s = u32_at(self.buf, self.at + 20) as usize;
        let len = u16_at(self.buf, s) as usize;
        // Validated in Orders::new, so this cannot fail.
        std::str::from_utf8(&self.buf[s + 2..s + 2 + len]).unwrap()
    }
}

#[test]
fn roundtrip() {
    let bytes = write_orders(&[(1, 101_25000000, 10, "AAPL"), (2, 99_00000000, 5, "MSFT")]);
    let v = Orders::new(&bytes).unwrap();
    assert_eq!(v.get(1).symbol(), "MSFT");
    assert_eq!(v.get(0).price(), 101_25000000);
    assert!(Orders::new(&bytes[..bytes.len() - 1]).is_err());
}

from_le_bytes on byte slices is alignment-free and compiles to single loads on x86 and ARM64. The symbol accessor re-checks UTF-8; production code stores the validated flag in the type and uses from_utf8_unchecked.

Complexity and constants

• Field access: a bounds check plus one load, order of 1 ns from cache; random access to a cold buffer costs a cache miss (order of 100 ns) like any struct.
• Validation: O(n) over offsets, order of GB/s; skipped entirely for trusted data (same process, signed files).
• Size overhead vs protobuf: fixed-width ints and 4-8 byte offsets; for small-integer-heavy messages order of 1.5-3x larger before compression.
• FlatBuffers offsets are 32-bit: a single buffer is limited to 2 GB.

Efficiency

• Cost: field access order of 1 ns from cache, order of 100 ns on a cold miss; validation O(n) at order of GB/s; messages order of 1.5-3x larger than protobuf for small-integer data. The bound is memory bandwidth and cache misses, not decode CPU.
• Levers:
  • skip validation for trusted data (same process, signed or checksummed files) — removes the O(n) pass — local mmaped files
  • hot fixed-size fields first, records a multiple of 8 bytes — aligned loads, fewer cache lines (this page) — arrays of records
  • LZ4 / zstd on the buffer, or Cap'n Proto packing — recovers size — bandwidth-bound links
  • decode once into a plain struct — avoids repeated offset math and bounds checks — every field touched many times
  • read into an aligned receive buffer after a length prefix — whole path copy-free — RPC and IPC
• Naive → standard → state of the art:
• JSON parse into objects · order of 100s of MB/s plus allocation
• Protobuf decode · order of 1-2 GB/s plus allocation
• Fixed-layout format (SBE, FlatBuffers, rkyv), validated once or not at all when trusted · O(1) decode, a few ns per field

Implementation notes

• Alignment: casting &[u8] to &T needs the buffer aligned for T (UB otherwise). Either read with from_le_bytes (as above), allocate aligned buffers (rkyv's AlignedVec), or use zerocopy/bytemuck which check alignment and layout at compile time and run time.
• Endianness: all mainstream formats are little-endian; big-endian hosts pay a byte swap per load.
• Put fixed-size, hot fields first and keep records a multiple of 8 bytes so arrays of records stay aligned.
• For RPC, frame each message with a length prefix and read directly into an aligned receive buffer to keep the whole path copy-free.
• Schema evolution: FlatBuffers add fields only at the end of a table, mark old ones deprecated; SBE uses a schema version in the header and appends fields to the block; Cap'n Proto assigns ordinal numbers (@0, @1) that must never change.

Pitfalls

• Skipping validation on network input: a crafted offset reads outside the buffer (out-of-bounds read in unsafe accessors, panics in safe ones).
• Cap'n Proto pointer loops and amplification: readers enforce a traversal limit (default 8M words = 64 MB) and nesting limit (64); keep them.
• Holding a view keeps the whole receive buffer alive: copy out small fields you keep long-term.
• Hot loops over many fields can be slower than a decoded struct: every access repeats offset math and bounds checks. Decode once if you touch everything many times.
• Floats compared bit-wise across platforms: NaN payloads and -0.0 differ.

Variants and alternatives

In production

FlatBuffers (TensorFlow Lite .tflite models, Facebook's mobile apps), Cap'n Proto (Cloudflare Workers runtime and RPC), SBE (CME MDP 3.0 market data, many exchange gateways, Aeron), Apache Arrow (Spark, pandas, Polars, DuckDB interchange), rkyv (Rust caches and game assets).

Implementations

Sources

• van Oortmerssen et al., FlatBuffers documentation and Internals, google.github.io/flatbuffers. docs · code
• Varda, Cap'n Proto: Encoding Spec, capnproto.org. spec · code
• FIX Trading Community, Simple Binary Encoding (SBE) Technical Specification. spec · code
• Apache Arrow, Arrow Columnar Format specification, arrow.apache.org. spec
• Rkyv, The rkyv book. docs · code