← Back
Home › Systems › Observability and operations

log-analytics

interactive log analytics at petabyte scale

Scuba, ClickHouse logs

SystemTechniques · 5Primitives · 23 log-analytics Scatter-gather querydistributed · medium: Send a query only to shards that can match, merge their small partial results, and bound the wait for stragglers.›Columnar storagestorage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›LSM-treestorage · medium: Buffer writes in memory, flush them as immutable sorted files and merge in the background; turns random writes into sequential I/O.›Vectorized executionstorage · medium: Query operators pass batches of about 1-2K values per column instead of single tuples, amortizing interpretation overhead and enabling tight SIMD loops.›Telemetry samplingsystems · medium: Head, tail and dynamic sampling of traces, events and logs to a budgetTop-k selectionsearch-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.›Bit packing and PFORcompression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›Delta encodingcompression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›Zone mapsstorage · low: Per-block min/max (and sparse primary) indexes to skip data during scansLZ4compression · low: Byte-aligned LZ77 with no entropy stage; compresses at hundreds of MB/s and decompresses at several GB/s per core, for when speed beats ratio.›Zstandard (zstd)compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.›Bloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›HyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›String interningdata-structures · low: Store each distinct string once and refer to it by idSIMD programminghardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.›Simd json parsingsearch-sort · low: Two-stage SIMD JSON parsing (structural index, then tape) at GB/sHash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskTime partitioningstorage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.›K way mergesearch-sort · low: Merge k sorted streams with a heap or loser tree (merging iterators)Deadline propagationsystems · low: Carry an absolute deadline and cancellation along a call chainHedged requestssystems · low: Sending a backup request after a latency percentile and taking the first answer to cut tail latency+ 7 more primitives

Similar systems

Share the most building blocks with this one.

columnar-warehouse 38 %realtime-olap 30 %logging 30 %continuous-profiling 25 %siem 21 %