← Back
Home › Systems › Databases and storage engines

object-storage

object storage service

S3, MinIO

SystemTechniques · 8Primitives · 30 object-storage Quorum replicationdistributed · medium: Leaderless replication: write to W of N replicas, read from R; overlap (R + W > N) makes reads see the latest completed write.›Raftdistributed · medium: Leader-based consensus on a replicated log; one leader per term, majority commit; the default for a strongly consistent replicated state machine.›Envelope encryptionsecurity · medium: Encrypt each object with its own data key and wrap that key under a KMS-held key, so rotation re-wraps 60 bytes and deleting a key shreds data.›Erasure codingstorage · medium: Reed-Solomon k+m coding to survive m lost shards at lower overhead than replicationLSM-treestorage · medium: Buffer writes in memory, flush them as immutable sorted files and merge in the background; turns random writes into sequential I/O.›Anti-entropy repairdistributed · medium: Replicas compare hash trees of a key range, descend only into differing subtrees and stream just those ranges, healing silent divergence in the background.›Hinted handoffdistributed · medium: Park a write meant for a down replica as a tagged hint on a healthy node, replay it when the replica returns, and count it toward the write quorum only if availability matters more than consistency.›AEAD (AES-GCM, ChaCha20-Poly1305)security · low: Authenticated encryption that encrypts and detects tampering in one call; the only kind of symmetric encryption you should use.›Rendezvous hashingdistributed · low: Score every node with hash(key, node) and pick the highest: minimal key movement and exact weights without a ring, at O(n) per lookup.›CRC32C checksumshardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.›fsync, O_DSYNC and O_DIRECThardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.›HTTP/1.1 message parsingnetworking · low: Parse HTTP/1.1 requests and responses incrementally from a byte stream: RFC 9112 body-length rules, chunked decoding and the strictness that stops request smuggling.›io_uringnetworking · low: Linux async I/O through two shared-memory rings (submissions in, completions out), batching or eliminating syscalls for sockets and files alike.›Resumable uploadsnetworking · low: Open an upload session, send the body in offset-addressed chunks, and after any failure ask the server how much it committed and continue from there.›Zero-copy I/Onetworking · low: Move bytes between files, sockets and NICs by passing page references instead of copying through user space: sendfile, splice, MSG_ZEROCOPY, kTLS.›Cryptographic hashes (SHA-2, BLAKE3)security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.›Hmac request signingsecurity · low: HMAC canonical-request signatures and presigned URLs (SigV4 style) with expiryData scrubbingstorage · low: Background re-read and checksum verification of stored data to catch latent corruptionConsistent hashingdistributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.›Vector clocksdistributed · low: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›+ 1 more techniques+ 14 more primitives

Similar systems

Share the most building blocks with this one.

distributed-fs 29 %block-storage 23 %cold-archive 23 %mail-store 22 %backup-dr 19 %

In-house systems

Named systems of this kind at large companies, with their paper or write-up.

ShardStore (S3)Amazonf4Meta Platforms