mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
Databases and storage engines
block-storage
networked block storage
EBS, Ceph RBD
System
Techniques · 8
Primitives · 36
block-storage
Chain replication
distributed · medium: Replicas in a line: writes enter at the head and commit at the tail, reads hit the tail; f+1 nodes tolerate f failures.
›
Quorum replication
distributed · medium: Leaderless replication: write to W of N replicas, read from R; overlap (R + W > N) makes reads see the latest completed write.
›
Raft
distributed · medium: Leader-based consensus on a replicated log; one leader per term, majority commit; the default for a strongly consistent replicated state machine.
›
Erasure coding
storage · medium: Reed-Solomon k+m coding to survive m lost shards at lower overhead than replication
Log-structured storage
storage · medium: Make the log the store: append every write, point an index at the newest copy, reclaim space by copying live records forward.
›
Anti-entropy repair
distributed · medium: Replicas compare hash trees of a key range, descend only into differing subtrees and stream just those ranges, healing silent divergence in the background.
›
Hinted handoff
distributed · medium: Park a write meant for a down replica as a tagged hint on a healthy node, replay it when the replica returns, and count it toward the write quorum only if availability matters more than consistency.
›
Write-ahead log
storage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.
›
CRC32C checksums
hardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.
›
Copy on write snapshots
storage · low: Point-in-time snapshots by sharing pages/extents and copying only on write
Fencing tokens
distributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.
›
fsync, O_DSYNC and O_DIRECT
hardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.
›
Nvme over fabrics
hardware-os · low: NVMe-oF over TCP/RDMA: remote block access with NVMe queue semantics
SSD and NVMe internals
hardware-os · low: NAND pages and erase blocks, the FTL, garbage collection and write amplification, NVMe queues and queue depth, and 4 KiB alignment, with the numbers that matter.
›
io_uring
networking · low: Linux async I/O through two shared-memory rings (submissions in, completions out), batching or eliminating syscalls for sockets and files alike.
›
Kernel-bypass networking
networking · low: User space (DPDK, AF_XDP) or an in-driver eBPF hook (XDP) handles packets straight from NIC rings, skipping sockets, for 10-40 Mpps per core.
›
Aes xts disk encryption
security · low: AES-XTS tweakable sector encryption for block devices; why no integrity and how sector tweaks work
Extent trees
storage · low: Map file ranges to disk extents in a shallow B-tree (ext4, XFS)
Leases
distributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.
›
Health checking
systems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.
›
Consistent hashing
distributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.
›
Vector clocks
distributed · low: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.
›
Exponential backoff with jitter
systems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.
›
+ 1 more techniques
+ 20 more primitives
System
Techniques · 8
Primitives · 36
block-storage
Chain replication
distributed · medium: Replicas in a line: writes enter at the head and commit at the tail, reads hit the tail; f+1 nodes tolerate f failures.
›
Quorum replication
distributed · medium: Leaderless replication: write to W of N replicas, read from R; overlap (R + W > N) makes reads see the latest completed write.
›
Raft
distributed · medium: Leader-based consensus on a replicated log; one leader per term, majority commit; the default for a strongly consistent replicated state machine.
›
Erasure coding
storage · medium: Reed-Solomon k+m coding to survive m lost shards at lower overhead than replication
Log-structured storage
storage · medium: Make the log the store: append every write, point an index at the newest copy, reclaim space by copying live records forward.
›
Anti-entropy repair
distributed · medium: Replicas compare hash trees of a key range, descend only into differing subtrees and stream just those ranges, healing silent divergence in the background.
›
Hinted handoff
distributed · medium: Park a write meant for a down replica as a tagged hint on a healthy node, replay it when the replica returns, and count it toward the write quorum only if availability matters more than consistency.
›
State-machine snapshots
distributed · medium: Periodically save the state a log has produced at index i so the log prefix can be deleted; recovery and catch-up become load snapshot plus replay suffix.
›
Write-ahead log
storage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.
›
CRC32C checksums
hardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.
›
Copy on write snapshots
storage · low: Point-in-time snapshots by sharing pages/extents and copying only on write
Fencing tokens
distributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.
›
fsync, O_DSYNC and O_DIRECT
hardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.
›
Nvme over fabrics
hardware-os · low: NVMe-oF over TCP/RDMA: remote block access with NVMe queue semantics
SSD and NVMe internals
hardware-os · low: NAND pages and erase blocks, the FTL, garbage collection and write amplification, NVMe queues and queue depth, and 4 KiB alignment, with the numbers that matter.
›
io_uring
networking · low: Linux async I/O through two shared-memory rings (submissions in, completions out), batching or eliminating syscalls for sockets and files alike.
›
Kernel-bypass networking
networking · low: User space (DPDK, AF_XDP) or an in-driver eBPF hook (XDP) handles packets straight from NIC rings, skipping sockets, for 10-40 Mpps per core.
›
Aes xts disk encryption
security · low: AES-XTS tweakable sector encryption for block devices; why no integrity and how sector tweaks work
Extent trees
storage · low: Map file ranges to disk extents in a shallow B-tree (ext4, XFS)
Leases
distributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.
›
Health checking
systems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.
›
Consistent hashing
distributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.
›
Vector clocks
distributed · low: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.
›
Exponential backoff with jitter
systems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
Finite field arithmetic
numerics · low: Prime-field and extension-field arithmetic (Montgomery form, Goldilocks, BabyBear), vectorized
Reed-Solomon codes
signal-processing · low: Byte-oriented MDS code: n-k parity symbols correct (n-k)/2 unknown symbol errors or n-k known erasures; the workhorse for bursts, QR codes, discs and storage.
›
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
Segmented log
distributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.
›
Merkle tree
data-structures · low: Hash tree for verifying and diffing large datasets with logarithmic proofs
Range partitioning
distributed · low: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.
›
Token bucket
systems · low: A bucket of up to b tokens refilled at rate r; requests spend tokens, enforcing an average rate while allowing bursts of b.
›
Phi accrual failure detector
distributed · low: Instead of a fixed heartbeat timeout, output a continuous suspicion level φ from the observed heartbeat interval distribution; adapts to each link's jitter.
›
Copy-on-write fork
hardware-os · low: fork() gives the child the parent's whole address space by sharing pages read-only and copying one only when someone writes it; vfork and posix_spawn skip even that.
›
False sharing
concurrency · low: Independent variables written by different cores on one cache line force the line to bounce between cores; pad or regroup hot data to fix it.
›
Memory ordering
concurrency · low: What Relaxed, Acquire, Release and SeqCst atomics actually guarantee, what they cost on x86 and ARM, and which one each pattern needs.
›
SPSC ring buffer
data-structures · low: Bounded wait-free queue for exactly one producer and one consumer thread, built from an array and two indices with acquire/release ordering.
›
CPU caches and the memory hierarchy
hardware-os · low: Sizes, line widths and latencies of L1/L2/L3/DRAM on modern x86 and Apple M-series, and the data-layout rules that follow from them.
›
Huge pages
hardware-os · low: 2 MB/1 GB pages to cut TLB misses; THP vs explicit hugetlbfs
Numa aware allocation
hardware-os · low: Allocate memory and pin threads on the local NUMA node; first-touch and interleave policies
Nvme queues
hardware-os · low: NVMe submission and completion queue pairs, doorbells and polling from a driver
Ebpf xdp
networking · low: EBPF programs at the XDP/tc hooks for in-kernel packet processing at line rate
Receive side scaling
networking · low: NIC hashing of flows to RX queues (RSS) and symmetric hashing for per-core processing
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Copy for AI
Similar systems
Share the most building blocks with this one.
distributed-fs
41 %
object-storage
23 %
filesystem
22 %
tiered-cache
19 %
coordination-store
19 %