GPU command buffers

Deep dive

How it works

In: draw, dispatch, copy and barrier commands recorded by one or more threads, plus a frame index · Out: command buffers submitted to a queue, and a fence or timeline value when the GPU has finished them · State: per-slot pool or allocator, the fence value of each slot · Cost: record O(commands), order of 1 ns per command in the model below; real cost is driver calls

1. Pool per (thread, frame slot). Vulkan VkCommandPool, D3D12 ID3D12CommandAllocator; Metal hides this inside the queue. Pools are externally synchronized, so one thread records into one pool at a time.
2. Record into a command buffer/list from that pool: state commands (pipeline, descriptor sets), draws, barriers. Nothing executes; the driver validates and translates, and the app can dedupe redundant state.
3. Close the buffer and submit several at once to the queue (vkQueueSubmit, ExecuteCommandLists, MTLCommandBuffer.commit). The call returns immediately and signals a fence or timeline value on completion.
4. Ring: frame f uses slot f mod N. Before recording, wait until the slot's fence value is reached, then reset the pool and reuse its memory.
5. Never wait for the frame you just submitted: that serializes CPU and GPU.

CPU:  rec f0 | rec f1 | rec f2 (wait fence of f0) | rec f3 (wait f1) ...
GPU:      run f0   | run f1   | run f2          | run f3 ...
slot:   0        1        0                 1            (N = 2)

Code

A std-only model of the CPU side: a flat u32 command stream with redundant-state elimination, a simulated queue that signals a timeline fence, and the frame ring that blocks only when the GPU is behind. It does not touch a real GPU; the encode/reset/fence structure is what Vulkan and D3D12 impose.

use std::collections::VecDeque;

// One u32 opcode, then a fixed number of u32 payload words per opcode.
const SET_PIPELINE: u32 = 1; // [op, pipeline_id]
const BIND_SET: u32 = 2;     // [op, slot, descriptor_set_id]
const DRAW: u32 = 3;         // [op, vertex_count, instance_count]
const BARRIER: u32 = 4;      // [op, src_stage, dst_stage]
const WORDS: [usize; 5] = [0, 2, 3, 3, 3];

/// One recording arena: a linear Vec<u32>. `reset` keeps the capacity (no free/malloc).
#[derive(Default)]
pub struct Encoder {
    words: Vec<u32>,
    pipeline: Option<u32>,
    sets: [Option<u32>; 4],
    pub skipped: u32, // redundant state commands never written
}

impl Encoder {
    pub fn set_pipeline(&mut self, id: u32) {
        if self.pipeline == Some(id) { self.skipped += 1; return; }
        self.pipeline = Some(id);
        self.words.extend_from_slice(&[SET_PIPELINE, id]);
    }
    pub fn bind_set(&mut self, slot: u32, id: u32) {
        if self.sets[slot as usize] == Some(id) { self.skipped += 1; return; }
        self.sets[slot as usize] = Some(id);
        self.words.extend_from_slice(&[BIND_SET, slot, id]);
    }
    pub fn draw(&mut self, vertices: u32, instances: u32) {
        self.words.extend_from_slice(&[DRAW, vertices, instances]);
    }
    pub fn barrier(&mut self, src: u32, dst: u32) {
        self.words.extend_from_slice(&[BARRIER, src, dst]);
        self.pipeline = None; self.sets = [None; 4]; // be conservative: forget cached state
    }
    pub fn finish(&mut self) -> Vec<u32> { std::mem::take(&mut self.words) }
    pub fn reset(&mut self, mut recycled: Vec<u32>) {
        recycled.clear(); // length 0, capacity kept
        self.words = recycled;
        self.pipeline = None; self.sets = [None; 4]; self.skipped = 0;
    }
}

/// Simulated GPU queue: executes buffers in order and signals a monotonic timeline fence.
#[derive(Default)]
pub struct Gpu {
    pending: VecDeque<(u64, Vec<u32>)>,
    pub completed: u64, // timeline value, like a Vulkan timeline semaphore
    pub draws: u64,
    pub stalls: u32,    // times the CPU had to block
    retired: Vec<(u64, Vec<u32>)>, // finished buffers, memory returned to the app
}

impl Gpu {
    pub fn submit(&mut self, serial: u64, cb: Vec<u32>) { self.pending.push_back((serial, cb)); }
    /// The GPU makes progress: run one buffer, return its (now reusable) memory.
    pub fn step(&mut self) {
        let Some((serial, cb)) = self.pending.pop_front() else { return };
        let mut i = 0;
        while i < cb.len() {
            let op = cb[i] as usize;
            if op == DRAW as usize { self.draws += cb[i + 2] as u64; }
            i += WORDS[op];
        }
        self.completed = serial;
        self.retired.push((serial, cb));
    }
}

/// Ring of N frames in flight: each slot owns an arena and remembers the fence value of its last submit.
pub struct FrameRing { arenas: Vec<Option<Vec<u32>>>, fence: Vec<u64>, next_serial: u64 }

impl FrameRing {
    pub fn new(n: usize) -> Self {
        FrameRing { arenas: (0..n).map(|_| Some(Vec::with_capacity(1 << 16))).collect(), fence: vec![0; n], next_serial: 1 }
    }
    /// Wait until the GPU has finished the slot's previous use, then hand its memory to the encoder.
    pub fn begin(&mut self, frame: usize, gpu: &mut Gpu, enc: &mut Encoder) {
        let slot = frame % self.arenas.len();
        while gpu.completed < self.fence[slot] {
            gpu.stalls += 1; // the CPU blocks here (vkWaitSemaphores, WaitForSingleObject, ...)
            gpu.step();
            self.collect(gpu);
        }
        self.collect(gpu);
        enc.reset(self.arenas[slot].take().expect("arena still owned by the GPU"));
    }
    pub fn submit(&mut self, frame: usize, gpu: &mut Gpu, enc: &mut Encoder) {
        let serial = self.next_serial;
        self.next_serial += 1;
        self.fence[frame % self.arenas.len()] = serial;
        gpu.submit(serial, enc.finish());
    }
    /// Take back the memory of every buffer the GPU has finished.
    fn collect(&mut self, gpu: &mut Gpu) {
        for (serial, mem) in gpu.retired.drain(..) {
            let slot = self.fence.iter().position(|&f| f == serial).unwrap();
            self.arenas[slot] = Some(mem);
        }
    }
}

fn main() {
    let (mut gpu, mut enc, mut ring) = (Gpu::default(), Encoder::default(), FrameRing::new(2));
    for f in 0..6 {                       // the GPU never advances unless the CPU has to wait for it
        ring.begin(f, &mut gpu, &mut enc);
        for i in 0..64 {                  // 64 draws sorted by pipeline, 8 per pipeline
            enc.set_pipeline(i / 8);
            enc.bind_set(0, 7);
            enc.draw(36, 1);
        }
        if f == 5 { assert_eq!(enc.skipped, 56 + 63); } // redundant state never written
        ring.submit(f, &mut gpu, &mut enc);
    }
    assert_eq!(gpu.stalls, 4);            // frames 0, 1 are free; each later frame waits once
    while !gpu.pending.is_empty() { gpu.step(); }
    assert_eq!(gpu.draws, 6 * 64);
}

begin is where vkWaitSemaphores / ID3D12Fence wait goes in a real renderer; Vec::clear stands in for vkResetCommandPool / ID3D12CommandAllocator::Reset.

Complexity and constants

• Record: O(commands). In the model above 3.25 u32 words (13 bytes) per draw with pipeline sorting (8 draws per pipeline); measured on an Apple-silicon Mac, release build: order of 1 ns per draw to encode. Real drivers add validation and state translation, so cost per call is far higher and is the thing to count.
• Cost drivers: API calls (each one crosses into the driver), bytes written, state changes, and the number of submits (each vkQueueSubmit can involve a kernel transition).
• Memory: N frames x peak commands per frame. Allocators grow but do not shrink (D3D12 documents this), so size by the worst frame.
• Fence wait: zero when the GPU is at most N-1 frames behind, otherwise the CPU blocks until it catches up.

Efficiency

• Cost: the bound is CPU time in the driver and latency, not bandwidth. Encoding in the model costs order of 1 ns per draw (measured above, 13 bytes per draw); a real driver call costs far more, so count calls. The model skips 119 of 128 state commands in its 64-draw test (93 %) before the driver sees them.
• Levers:
  • fewer, larger submits — one call for a frame's buffers instead of one per buffer (each vkQueueSubmit can cross into the kernel) — always
  • reset the pool, not buffers — vkResetCommandPool recycles every buffer at once; D3D12 recommends pooling allocators and reusing lists right after ExecuteCommandLists (docs) — per-frame recording
  • drop redundant state (pipeline, descriptor set) — 93 % of state commands in the model — scenes sorted by pipeline
  • record in parallel, one pool per thread — no locks on the hot path — multi-core CPUs
  • reuse recordings (D3D12 bundles, Vulkan secondary buffers, CUDA graphs) — amortizes driver work — repeated sequences
• Naive → standard → state of the art:
  • one submit and one wait per draw · CPU and GPU serialized
  • ring of N frames, per-thread pools, batched submit · wait only when the GPU is N-1 frames behind
  • record once and drive variation from data with indirect draws · near-zero per-frame recording for static scenes

Implementation notes

• Vulkan: create the pool with VK_COMMAND_POOL_CREATE_TRANSIENT_BIT for short-lived buffers; add RESET_COMMAND_BUFFER_BIT only if you reset buffers individually (prefer pool reset). ONE_TIME_SUBMIT_BIT tells the driver each recording is submitted once. Use a timeline semaphore (Vulkan 1.2) or one fence per slot.
• D3D12: ID3D12CommandAllocator::Reset fails while the GPU still executes lists recorded on it. Command lists can be Reset right after ExecuteCommandLists. Allocator pool size is about lists x MaxFrameLatency.
• Metal: queue and command buffers are cheap and single-use; the queue has a fixed cap of uncompleted command buffers (makeCommandQueue(maxCommandBufferCount:)) and makeCommandBuffer() blocks when none is free, so that cap is your frame ring.
• Keep per-frame uploads in the same slot: a per-slot staging/uniform ring freed by the same fence.
• Wait with a timeout and log it: a device-lost GPU never signals, and an unbounded wait hangs the process.
• Frame count N is a latency/throughput knob; pair it with frame pacing.

Pitfalls

• Resetting a pool or allocator that the GPU still uses: Vulkan undefined behavior, a D3D12 error. Always check the slot fence first.
• Freeing or overwriting a resource (buffer, texture, descriptor) referenced by an in-flight command buffer; retire resources by the same fence.
• Sharing one pool between threads without a lock.
• Waiting on the fence of the frame just submitted, which collapses the pipeline to 1 frame in flight.
• One vkQueueSubmit per command buffer: submit overhead dominates with many small buffers.
• Growing N to hide spikes: it hides them behind latency; fix the stall instead.

Variants and alternatives

In production

• Rendering engines: the Vulkan, D3D12 and Metal documentation patterns above (Sources): per-thread recording, per-frame pools or allocators, fence-guarded reuse.
• Browsers and portable graphics: WebGPU's GPUCommandEncoder records commands that are finished into a command buffer and submitted to a queue (spec); wgpu implements it over Vulkan, Metal and D3D12.
• GPU compute and ML: a CUDA graph is defined separately from its execution, so it is defined once and launched repeatedly, which cuts per-launch CPU cost (docs).

Implementations

Sources

• Khronos, The Vulkan Specification, command buffers and command pools chapters. spec
• Microsoft, Creating and recording command lists and bundles, Direct3D 12 docs. docs
• Apple, MTLCommandQueue and MTLCommandBuffer, Metal documentation. docs