← Back
Home › Systems › Data processing and pipelines

commit-log

distributed commit log / event streaming

Kafka, Redpanda

SystemTechniques · 15Primitives · 51 commit-log Consumer group rebalancingdistributed · medium: Assigning partitions to group members with generations, sticky/cooperative rebalanceDeterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Exactly once processingdistributed · medium: Idempotent producers plus transactional offset+output commits for effectively-once resultsLog compactiondistributed · medium: Keep only the latest record per key in a log, with tombstones and a cleanerPrimary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Raftdistributed · medium: Leader-based consensus on a replicated log; one leader per term, majority commit; the default for a strongly consistent replicated state machine.›Coverage-guided fuzzingsecurity · medium: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.›Consumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Idempotency keysfinance · low: Client-chosen unique key per logical operation, stored with a request fingerprint and the final response, so retries replay the first result instead of acting twice.›Segmented logdistributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›Fast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›LZ4compression · low: Byte-aligned LZ77 with no entropy stage; compresses at hundreds of MB/s and decompresses at several GB/s per core, for when speed beats ratio.›Zstandard (zstd)compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.›Hdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesCRC32C checksumshardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.›mmap and the page cachehardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.›Process supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›epoll event loopnetworking · low: One thread waits on thousands of non-blocking sockets with epoll_wait and handles only ready ones; the core of nginx, Redis and tokio.›Length-prefix framingnetworking · low: Split a byte stream (TCP, TLS, pipes) into messages by writing each message's length before it; O(1) boundary detection with no escaping.›Zero-copy I/Onetworking · low: Move bytes between files, sockets and NICs by passing page references instead of copying through user space: sendfile, splice, MSG_ZEROCOPY, kTLS.›Mutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Group commitstorage · low: Let concurrent committers share one log write and fsync; throughput grows with concurrency while each commit still waits at most about two fsyncs.›+ 8 more techniques+ 35 more primitives

Similar systems

Share the most building blocks with this one.

mqtt-broker 34 %customer-data-platform 32 %robot-middleware 31 %telecom-charging 29 %e2e-cloud-backup 28 %