← Back
Home › Systems › Communication and media

mail-store

mailbox storage and search at scale

Gmail backend, Dovecot

SystemTechniques · 8Primitives · 37 mail-store Incremental sync cursorsdistributed · medium: The client keeps one opaque token; each pull returns only the changes after it plus the next token, so sync cost scales with what changed, not with the dataset.›Quorum replicationdistributed · medium: Leaderless replication: write to W of N replicas, read from R; overlap (R + W > N) makes reads see the latest completed write.›Envelope encryptionsecurity · medium: Encrypt each object with its own data key and wrap that key under a KMS-held key, so rotation re-wraps 60 bytes and deleting a key shreds data.›Erasure codingstorage · medium: Reed-Solomon k+m coding to survive m lost shards at lower overhead than replicationInverted indexstorage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›LSM-treestorage · medium: Buffer writes in memory, flush them as immutable sorted files and merge in the background; turns random writes into sequential I/O.›Anti-entropy repairdistributed · medium: Replicas compare hash trees of a key range, descend only into differing subtrees and stream just those ranges, healing silent divergence in the background.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Consistent hashingdistributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.›AEAD (AES-GCM, ChaCha20-Poly1305)security · low: Authenticated encryption that encrypts and detects tampering in one call; the only kind of symmetric encryption you should use.›BM25 rankingsearch-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.›Dictionary compressioncompression · low: Prime the compressor with shared, pre-trained content so small messages can reference it; often 2-5x better ratio on sub-KB records.›Roaring bitmapscompression · low: Compressed set of 32-bit integers in 64K-value chunks, each a sorted array, bitmap or run list; fast intersections, the standard for bitmap indexes.›Zstandard (zstd)compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.›CRC32C checksumshardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.›Unicode text tokenization for searchsearch-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.›Content-addressed storagestorage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.›Mime parsingstorage · low: RFC 5322/2045 headers, multipart bodies and transfer encodingsMessage sequence numbersdistributed · low: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".›Cursor paginationnetworking · low: Keyset/opaque-cursor pagination for stable incremental reads from APIs and tablesVector clocksdistributed · low: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.›Aes key wrapsecurity · low: AES Key Wrap (RFC 3394/5649) and AES-GCM-SIV for deterministic, authenticated wrapping of key materialCsprngsecurity · low: OS-backed cryptographically secure random numbers (getrandom), and when a fast PRNG is not enough+ 1 more techniques+ 21 more primitives

Similar systems

Share the most building blocks with this one.

backup-dr 28 %object-storage 22 %distributed-fs 19 %filesystem 17 %logging 17 %