SSD and NVMe internals

Deep dive

How it works

In: NVMe read, write, flush and deallocate commands in submission queues · Out: completions; data stored in NAND pages · State: the FTL logical-to-physical map, valid-page counts per erase block, queue pairs · Cost: 4 KiB QD1 read order of 50-100 µs; write amplification about 1× sequential, 2-5× random on a nearly full drive.

host LBA (4 KiB) ──FTL map──► (channel, die, plane, block, page)
block: [p0 p1 p2 ... p255+]   program pages in order, erase whole block
GC: pick block with fewest valid pages → copy valid pages → erase → free list

1. Geometry: NAND page order of 16 KiB. Erase block = hundreds to thousands of pages, order of MiB. A drive has order of 8-16 channels with several dies each. Every die works independently.
2. Timings (TLC, order of): read tens of µs, program hundreds of µs, erase several ms. Writes are acknowledged from DRAM or an SLC-mode cache, so they look faster than program time.
3. FTL map: page-level, about 4 B per 4 KiB → about 1 GiB DRAM per 1 TiB of flash. DRAM-less drives cache the map in a small SRAM or in host memory (NVMe HMB) and slow down on random access over large spans.
4. Over-provisioning: spare flash that is never exposed. "GB vs GiB" alone gives 7.4 %. Write-heavy enterprise drives expose less capacity. More spare space lets GC find emptier blocks and lowers write amplification.
5. TRIM / Deallocate: tells the FTL which LBAs are dead, so GC doesn't copy them. Linux: fstrim (periodic) or the discard mount option.
6. SLC cache: consumer TLC/QLC drives write first in 1-bit mode. After order of tens of GB of sustained writes, speed drops to native TLC/QLC rate. QLC can drop to a few hundred MB/s.
7. NVMe: submission/completion queue pairs in host memory, doorbell registers, MSI-X interrupts. The spec allows up to 65 535 I/O queues of up to 65 536 entries. Linux blk-mq maps one queue per CPU.

Decision table

Code

Random 4 KiB reads that bypass the page cache, with one blocking pread per thread, so threads = queue depth. Reports IOPS and latency percentiles. Numbers in Key facts are from this program on an M4 Pro. Tested on macOS, type-checked for Linux.

// Cargo.toml: libc = "0.2". Random 4 KiB reads at queue depth = number of threads.
use std::alloc::{alloc, dealloc, Layout};
use std::fs::{File, OpenOptions};
use std::io::{self, Seek, SeekFrom};
use std::os::unix::fs::FileExt;
use std::time::{Duration, Instant};

const BS: usize = 4096;

/// Uncached file handle: O_DIRECT on Linux, F_NOCACHE on macOS.
fn open_uncached(path: &str) -> io::Result<File> {
    let mut o = OpenOptions::new();
    o.read(true);
    #[cfg(target_os = "linux")]
    { use std::os::unix::fs::OpenOptionsExt; o.custom_flags(libc::O_DIRECT); }
    let f = o.open(path)?;
    #[cfg(target_os = "macos")]
    { use std::os::fd::AsRawFd; unsafe { libc::fcntl(f.as_raw_fd(), libc::F_NOCACHE, 1) }; }
    Ok(f)
}

/// One worker = one outstanding I/O. Returns per-I/O latencies.
fn worker(f: &File, blocks: u64, seed: u64, deadline: Instant) -> Vec<Duration> {
    let layout = Layout::from_size_align(BS, BS).unwrap(); // O_DIRECT needs an aligned buffer
    let buf = unsafe { std::slice::from_raw_parts_mut(alloc(layout), BS) };
    let (mut s, mut lat) = (seed | 1, Vec::new());
    while Instant::now() < deadline {
        s ^= s << 13; s ^= s >> 7; s ^= s << 17;
        let off = (s % blocks) * BS as u64; // 4 KiB-aligned offset
        let t = Instant::now();
        f.read_exact_at(buf, off).expect("pread");
        lat.push(t.elapsed());
    }
    unsafe { dealloc(buf.as_mut_ptr(), layout) };
    lat
}

fn main() -> io::Result<()> {
    let path = std::env::args().nth(1).expect("usage: qd <big file or /dev/nvme0n1 (read-only)>");
    let f = open_uncached(&path)?;
    let blocks = (&f).seek(SeekFrom::End(0))? / BS as u64; // metadata().len() is 0 for block devices
    println!("{:>4} {:>10} {:>9} {:>9}", "QD", "IOPS", "p50 µs", "p99 µs");
    for qd in [1usize, 2, 4, 8, 16, 32, 64] {
        let deadline = Instant::now() + Duration::from_secs(3);
        let mut lat: Vec<Duration> = std::thread::scope(|sc| {
            let hs: Vec<_> = (0..qd).map(|i| { let f = &f; sc.spawn(move || worker(f, blocks, 0x9E37 * (i as u64 + 1), deadline)) }).collect();
            hs.into_iter().flat_map(|h| h.join().unwrap()).collect()
        });
        lat.sort_unstable();
        let us = |q: f64| lat[((lat.len() - 1) as f64 * q) as usize].as_secs_f64() * 1e6;
        println!("{qd:>4} {:>10.0} {:>9.1} {:>9.1}", lat.len() as f64 / 3.0, us(0.5), us(0.99));
    }
    Ok(())
}

The same measurement with fio: fio --name=rr --filename=/dev/nvme0n1 --readonly --rw=randread --bs=4k --direct=1 --ioengine=io_uring --iodepth=32 --runtime=30 --time_based.

Complexity and constants

• Little's law: QD1 at 75 µs → 13 k IOPS. 1 M IOPS at 100 µs needs 100 I/Os in flight.
• PCIe link: Gen4 x4 = 16 GT/s × 4 × 128/130 / 8 ≈ 7.9 GB/s. Gen5 x4 ≈ 15.8 GB/s. Drives reach order of 7 and 14 GB/s sequential.
• Endurance: TLC order of 1000-3000 P/E cycles, QLC order of hundreds to 1000. Rated as TBW or DWPD. Host writes allowed ≈ capacity × P/E cycles / WA.
• Write amplification = NAND bytes written / host bytes written. Read it from SMART/vendor logs (nvme smart-log, vendor-specific "NAND writes") vs "Data Units Written".

Efficiency

• Cost: a QD1 4 KiB random read takes order of 50-100 µs on TLC NVMe (74 µs p50, 13 k IOPS on an M4 Pro). The same drive reaches about 230 k IOPS at QD 32-64 but p50 latency rises to 271 µs at QD 64. Bound: latency at low queue depth, then NAND parallelism and the PCIe link.
• Levers:
  • queue depth 32-128 (io_uring, many threads) — IOPS ≈ QD / latency, 13 k → 230 k (above) — random reads
  • batch small updates into large sequential writes — write amplification 2-5× → about 1× — update-heavy workloads (LSM, log-structured)
  • keep 10-20 % free space or buy extra over-provisioning — fewer valid-page copies in garbage collection — nearly full drives
  • 4 KiB-aligned multiples — avoids read-modify-write in the FTL — on-disk formats
• Naive → standard → state of the art:
• naive: one synchronous request at a time · 13 k IOPS (74 µs each)
• standard: 32-64 requests in flight · about 230 k IOPS
• state of the art: user-space or polled NVMe with per-core queues (SPDK, io_uring) · 1 M IOPS needs about 100 requests in flight at 100 µs (Little's law)

Implementation notes

• Inspect: nvme id-ctrl (VWC = volatile write cache present), nvme id-ns -H (LBA formats 512 vs 4096), /sys/block/nvme0n1/queue/{logical_block_size,physical_block_size,optimal_io_size,scheduler}. NVMe usually runs with scheduler none.
• Switch a namespace to 4 KiB LBAs with nvme format --lbaf=<n>. This erases the namespace.
• Align partitions to 1 MiB (the default in modern partitioners) and filesystem blocks to 4 KiB.
• One thread with io_uring and SQPOLL can drive hundreds of thousands of IOPS. Blocking pread needs one thread per outstanding I/O (io_uring).
• Mixed read/write: reads stuck behind a program or erase on the same die see ms-scale tails. Drives with program/erase suspend help. Rate-limit background compaction.
• Hot/cold separation: data with similar lifetimes in the same erase blocks lowers GC copying. The host can steer this with ZNS or FDP.

Pitfalls

• Benchmarking reads on a freshly written file: the page cache (or the drive's DRAM) serves them. Use O_DIRECT, drop caches, or read a large old file.
• Reading unwritten or trimmed LBAs: many drives return zeros without touching flash, so you get fantasy IOPS.
• QD1 microbenchmarks that call the drive "slow" when it is idle 95 % of the time.
• Assuming sequential write speed holds forever on consumer drives (SLC cache exhaustion).
• Ignoring GC: steady-state random-write IOPS can be order of 3-10× below fresh-out-of-box numbers.

Variants and alternatives

In production

RocksDB (LSM design for flash, ZNS support via ZenFS), Aerospike and ScyllaDB (direct I/O, per-core queues), SPDK (user-space NVMe driver), Linux blk-mq, fio for measurement.

Implementations

Sources

• Nitin Agrawal et al., Design Tradeoffs for SSD Performance, USENIX ATC, 2008. paper
• Xiao-Yu Hu et al., Write Amplification Analysis in Flash-Based Solid State Drives, SYSTOR, 2009.
• Jun He, Sudarsun Kannan, Andrea C. Arpaci-Dusseau, Remzi H. Arpaci-Dusseau, The Unwritten Contract of Solid State Drives, EuroSys, 2017.
• Matias Bjørling et al., ZNS: Avoiding the Block Interface Tax for Flash-based SSDs, USENIX ATC, 2021. paper
• NVM Express, NVM Express Base Specification, revision 2.0. spec
• SNIA, Solid State Storage Performance Test Specification (PTS). spec