← Back
Home › Systems › Data processing and pipelines

realtime-olap

real-time OLAP on event streams

Druid, Pinot

SystemTechniques · 6Primitives · 27 realtime-olap Scatter-gather querydistributed · medium: Send a query only to shards that can match, merge their small partial results, and bound the wait for stragglers.›Columnar storagestorage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Inverted indexstorage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›Rollup aggregationstorage · medium: Fold raw events into one mergeable partial aggregate per time bucket and dimension key, then merge fine buckets into coarser ones, so long ranges read few rows and stay exact.›Star tree indexstorage · medium: Pre-aggregated tree over dimension combinations with a split threshold for fast group-bysVectorized executionstorage · medium: Query operators pass batches of about 1-2K values per column instead of single tuples, amortizing interpretation overhead and enabling tight SIMD loops.›Top-k selectionsearch-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.›Bit packing and PFORcompression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›Dictionary encodingcompression · low: Map values to small integer codes via a dictionary for compact, comparable columnsZone mapsstorage · low: Per-block min/max (and sparse primary) indexes to skip data during scansTime partitioningstorage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.›Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›Roaring bitmapscompression · low: Compressed set of 32-bit integers in 64K-value chunks, each a sorted array, bitmap or run list; fast intersections, the standard for bitmap indexes.›HyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›Quantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsConsumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Upsert mergestorage · low: MERGE/upsert by key with deduplication and late-arriving data rulesK way mergesearch-sort · low: Merge k sorted streams with a heap or loser tree (merging iterators)Deadline propagationsystems · low: Carry an absolute deadline and cancellation along a call chainHedged requestssystems · low: Sending a backup request after a latency percentile and taking the first answer to cut tail latencyDelta encodingcompression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›Varint (LEB128)compression · low: Variable-length integers in 7-bit groups with a continuation bit; small numbers take 1 byte, the wire format of Protobuf, WebAssembly and DWARF.›+ 11 more primitives

Similar systems

Share the most building blocks with this one.

log-analytics 30 %product-analytics 29 %columnar-warehouse 26 %metrics 24 %timeseries-db 20 %