Group commit

Deep dive

How it works

In: commit requests from many concurrent writers, each with a log record · Out: each writer told its record is durable · State: shared buffer, flushed LSN, flushing flag, one mutex and condvar · Cost: one fsync per batch; commits/s ≈ batch size / fsync latency.

t0  A appends, becomes leader ── write+fsync(A) ──────────► wake A
t1    B, C, D append, wait               (batch 2 builds up)
t2                                 B becomes leader ── write+fsync(B,C,D) ──► wake B,C,D

1. Under a mutex, append the record to the buffer. Its LSN is the buffer's new end offset.
2. Loop: if flushed ≥ my_lsn, return. If no flush is in progress, become leader: take the whole buffer, note batch_end, set flushing, release the mutex, then write and fsync.
3. The leader re-locks, sets flushed = batch_end, clears flushing and wakes all waiters.
4. Waiters re-check. Those not yet covered (they appended during the flush) elect the next leader.

Code

use std::fs::File;
use std::io::{self, Write};
use std::sync::{Condvar, Mutex};

struct State { buf: Vec<u8>, appended: u64, flushed: u64, flushing: bool, failed: bool }

pub struct GroupLog { st: Mutex<State>, cv: Condvar, file: Mutex<File> }

impl GroupLog {
    pub fn new(file: File) -> Self {
        let st = State { buf: Vec::new(), appended: 0, flushed: 0, flushing: false, failed: false };
        GroupLog { st: Mutex::new(st), cv: Condvar::new(), file: Mutex::new(file) }
    }

    /// Returns once `rec` is durable. Called concurrently by many threads.
    pub fn commit(&self, rec: &[u8]) -> io::Result<u64> {
        let mut st = self.st.lock().unwrap();
        st.buf.extend_from_slice(rec);
        st.appended += rec.len() as u64;
        let my_lsn = st.appended;
        loop {
            if st.failed { return Err(io::Error::other("log write failed; restart and recover")); }
            if st.flushed >= my_lsn { return Ok(my_lsn); }
            if !st.flushing {
                st.flushing = true; // become leader for everything appended so far
                let batch = std::mem::take(&mut st.buf);
                let batch_end = st.appended;
                drop(st); // others append into the next batch while we wait on the disk
                let res = {
                    let mut f = self.file.lock().unwrap(); // only one leader at a time
                    f.write_all(&batch).and_then(|_| f.sync_data())
                };
                st = self.st.lock().unwrap();
                st.flushing = false;
                match res {
                    Ok(()) => st.flushed = batch_end,
                    Err(_) => st.failed = true, // never retry fsync: the kernel may have dropped the pages
                }
                self.cv.notify_all();
                continue;
            }
            st = self.cv.wait(st).unwrap();
        }
    }
}

The failed flag matters. Without it, the waiters whose bytes were in the failed batch would elect a new leader, flush an empty buffer and report success.

Complexity and constants

• Throughput ≈ min(concurrency / fsync latency, device bandwidth / record size, CPU). Example: 0.5 ms fsync and 64 concurrent committers give an order of 100k commits/s ceiling from the log alone.
• Latency: 1-2 fsyncs plus wakeup. A condvar wakeup is order of 5-50 µs, which matters on fast NVMe with power-loss protection (fsync order of 10-50 µs).
• Batch size follows load: about 1 at low load (no added latency), large under load (high throughput).

Efficiency

• Cost: one write plus one fsync per batch, shared by all waiters; a commit waits at most about two fsync latencies. Commits/s ≈ batch size / fsync latency (0.5 ms fsync and 64 committers: order of 100k commits/s ceiling). Bound: fsync latency.
• Levers:
  • no lock held during I/O — next batch forms while the leader waits, no timer, no delay at low load — always
  • overlap stages — copy to the file on one thread, fsync the previous range on another (MySQL binlog flush, sync, commit) — high concurrency
  • page-aligned buffers with O_DIRECT|O_DSYNC, or io_uring linked write+fsync — write plus fsync in one submission — Linux hot logs
  • power-loss-protected NVMe or NVRAM — fsync order of 10-50 µs, batching matters less — latency-critical logs
  • asynchronous commit (PostgreSQL synchronous_commit = off) or epoch commit (Silo) — bounded loss window or latency traded for throughput — loss-tolerant writes
• Naive → standard → state of the art:
  • fsync per commit · 1 ms fsync gives 1000 commits/s
  • leader-based group commit · throughput about N× with N waiters
  • pipelined flush on PLP NVMe or epochs · fsync 10-50 µs, no wait for batching

Implementation notes

• Never hold the append lock during I/O. That is the whole trick. If the lock is held, the next batch cannot form.
• Split write and fsync stages: one thread copies the buffer to the file (write) while another fsyncs the previous range. Pipelining increases overlap (MySQL binlog group commit has flush, sync and commit stages).
• Thundering herd: notify_all with thousands of waiters wakes everyone. Use per-waiter slots or futexes keyed by LSN, or a sorted wait list where the leader wakes only the covered waiters.
• Preallocated, page-aligned log buffers with O_DIRECT|O_DSYNC turn write+fsync into one I/O. io_uring can link a write and an fsync into one submission.
• RocksDB write groups: the leader merges the followers' write batches into one WAL write and memtable insert, then followers apply in parallel.
• Early lock release: release row locks when the commit record is in the buffer, but acknowledge the client only after the flush. This raises hot-row throughput. Dependent transactions must not acknowledge before their predecessors are durable.

Pitfalls

• Timer-based batching ("flush every 10 ms") adds the delay even at low load. Leader/follower batching adds none.
• Retrying a failed fsync and reporting success (see failed above).
• Two-phase commit across an external binlog serialized fsyncs. MySQL before 5.6 lost group commit when the binlog was enabled.
• Asynchronous commit (PostgreSQL synchronous_commit = off) can lose up to 3 × wal_writer_delay (default 200 ms) of acknowledged commits, but never corrupts data.
• Replication: if commit also waits for replicas, batch the network round trips too. Raft leaders batch and pipeline AppendEntries for the same reason.

Variants and alternatives

In production

• Databases: PostgreSQL (commit_delay, commit_siblings), MySQL/InnoDB and binlog group commit, SQL Server and Oracle log writers.
• Key-value stores: RocksDB write groups.
• Coordination and consensus: etcd batches Raft log entries per fsync.
• File systems: the ext4 journal (jbd2) commits many operations in one transaction.

Implementations

Sources

• D. DeWitt et al., Implementation Techniques for Main Memory Database Systems, SIGMOD, 1984. paper
• R. Johnson et al., Aether: A Scalable Approach to Logging, VLDB, 2010. paper
• S. Tu et al., Speedy Transactions in Multicore In-Memory Databases (Silo), SOSP, 2013. paper · code
• PostgreSQL documentation, WAL Configuration and Asynchronous Commit. WAL configuration · asynchronous commit