← Back
Home › Systems › Databases and storage engines

lakehouse

lakehouse table format on object storage

Iceberg, Delta Lake

SystemTechniques · 3Primitives · 12 lakehouse Merge on read deletesstorage · medium: Row-level deletes/updates via delete files or deletion vectors applied at read timeSchema evolutionstorage · medium: Compatibility rules (backward/forward/full) for adding, removing and renaming fieldsTable snapshot manifestsstorage · medium: Table state as an immutable tree of snapshot -> manifest lists -> data files, committed by pointer swapRoaring bitmapscompression · low: Compressed set of 32-bit integers in 64K-value chunks, each a sorted array, bitmap or run list; fast intersections, the standard for bitmap indexes.›Columnar file formatstorage · low: Row groups, column chunks, pages, encodings and footer stats (Parquet/ORC style) (listed in Moved to the wiki)Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›Conditional writesdistributed · low: Put-if-absent / if-match compare-and-swap on objects or keys as a commit primitiveZone mapsstorage · low: Per-block min/max (and sparse primary) indexes to skip data during scansBloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›Bin packingoptimization · low: Pack items of given sizes into the fewest fixed-capacity bins; sort descending, then first-fit, gives at most 11/9 of optimal and runs in O(n log n).›Optimistic concurrency controlstorage · low: Run transactions without locks, record what they read, and validate at commit that nothing changed; fastest when conflicts are rare.›Partition pruningstorage · low: Skip whole partitions using partition transforms and predicate analysisUpsert mergestorage · low: MERGE/upsert by key with deduplication and late-arriving data rulesProtobuf wire formatnetworking · low: Protocol Buffers wire format: tags, varints, length-delimited fields and schema evolutionJSON Schema validationsecurity · low: Compile a JSON Schema once, then check every untrusted document against it, within hard size and depth limits, and report failures as JSON Pointers.›

Similar systems

Share the most building blocks with this one.

data-versioning 22 %