← Back
Home › Systems › Databases and storage engines

cold-archive

cold archival storage

Glacier, tape libraries

SystemTechniques · 2Primitives · 15 cold-archive Envelope encryptionsecurity · medium: Encrypt each object with its own data key and wrap that key under a KMS-held key, so rotation re-wraps 60 bytes and deleting a key shreds data.›Erasure codingstorage · medium: Reed-Solomon k+m coding to survive m lost shards at lower overhead than replicationAEAD (AES-GCM, ChaCha20-Poly1305)security · low: Authenticated encryption that encrypts and detects tampering in one call; the only kind of symmetric encryption you should use.›Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›Content defined chunkingcompression · low: Rolling-hash chunk boundaries (Rabin, FastCDC) for deduplication that survives insertionsZstandard (zstd)compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.›Merkle treedata-structures · low: Hash tree for verifying and diffing large datasets with logarithmic proofsIo request schedulinghardware-os · low: Reorder and batch I/O by device position (elevator/SCAN, deadline) for seek-bound mediaBin packingoptimization · low: Pack items of given sizes into the fewest fixed-capacity bins; sort descending, then first-fit, gives at most 11/9 of optimal and runs in O(n log n).›Cryptographic hashes (SHA-2, BLAKE3)security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.›Data scrubbingstorage · low: Background re-read and checksum verification of stored data to catch latent corruptionAes key wrapsecurity · low: AES Key Wrap (RFC 3394/5649) and AES-GCM-SIV for deterministic, authenticated wrapping of key materialCsprngsecurity · low: OS-backed cryptographically secure random numbers (getrandom), and when a fast PRNG is not enoughSIMD programminghardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.›Finite field arithmeticnumerics · low: Prime-field and extension-field arithmetic (Montgomery form, Goldilocks, BabyBear), vectorizedReed-Solomon codessignal-processing · low: Byte-oriented MDS code: n-k parity symbols correct (n-k)/2 unknown symbol errors or n-k known erasures; the workhorse for bursts, QR codes, discs and storage.›Constant time codesecurity · low: Writing secret-dependent code without branches or memory lookups on secrets; constant-time compare and select

Similar systems

Share the most building blocks with this one.

backup-dr 44 %e2e-cloud-backup 29 %file-sync 29 %object-storage 23 %image-snapshotter 20 %