← Back
Home › Systems › Hardware and datacenter

storage-array

erasure-coded storage array

Backblaze vaults, NetApp

SystemTechniques · 4Primitives · 14 storage-array Copy on write b treestorage · medium: Append-only/shadow-paging B+tree: copy the root-to-leaf path on write, atomic root swap (LMDB style)Declustered raidstorage · medium: Spread stripes over all disks so rebuild reads from every drive in parallelErasure codingstorage · medium: Reed-Solomon k+m coding to survive m lost shards at lower overhead than replicationLocal reconstruction codesstorage · medium: LRC: local parity groups cut repair reads versus plain Reed-Solomon (Azure-style)fsync, O_DSYNC and O_DIRECThardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.›Rendezvous hashingdistributed · low: Score every node with hash(key, node) and pick the highest: minimal key movement and exact weights without a ring, at O(n) per lookup.›SIMD programminghardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.›Finite field arithmeticnumerics · low: Prime-field and extension-field arithmetic (Montgomery form, Goldilocks, BabyBear), vectorizedContent defined chunkingcompression · low: Rolling-hash chunk boundaries (Rabin, FastCDC) for deduplication that survives insertionsLZ4compression · low: Byte-aligned LZ77 with no entropy stage; compresses at hundreds of MB/s and decompresses at several GB/s per core, for when speed beats ratio.›Phi accrual failure detectordistributed · low: Instead of a fixed heartbeat timeout, output a continuous suspicion level φ from the observed heartbeat interval distribution; adapts to each link's jitter.›CRC32C checksumshardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Token bucketsystems · low: A bucket of up to b tokens refilled at rate r; requests spend tokens, enforcing an average rate while allowing bursts of b.›B+ treestorage · low: Balanced high-fanout search tree with all values in linked leaves; the default on-disk index for point lookups and range scans.›Free space managementstorage · low: Free-space bitmaps, extent B-trees and space maps; allocation policies and fragmentationData scrubbingstorage · low: Background re-read and checksum verification of stored data to catch latent corruptionReed-Solomon codessignal-processing · low: Byte-oriented MDS code: n-k parity symbols correct (n-k)/2 unknown symbol errors or n-k known erasures; the workhorse for bursts, QR codes, discs and storage.›

Similar systems

Share the most building blocks with this one.

filesystem 24 %distributed-fs 21 %lsm-engine 19 %backup-dr 16 %btree-engine 15 %