← Back
Home › Systems › Databases and storage engines

timeseries-db

time-series database

Prometheus TSDB, InfluxDB

SystemTechniques · 4Primitives · 25 timeseries-db Inverted indexstorage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›LSM compactionstorage · medium: Compaction policies (leveled, tiered, time-window, FIFO) decide which sorted files to merge and when, trading write, read and space amplification and purging tombstones.›Rollup aggregationstorage · medium: Fold raw events into one mergeable partial aggregate per time bucket and dimension key, then merge fine buckets into coarser ones, so long ranges read few rows and stay exact.›LSM-treestorage · medium: Buffer writes in memory, flush them as immutable sorted files and merge in the background; turns random writes into sequential I/O.›Delta encodingcompression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›Varint (LEB128)compression · low: Variable-length integers in 7-bit groups with a continuation bit; small numbers take 1 byte, the wire format of Protobuf, WebAssembly and DWARF.›Sorted-list intersectionsearch-sort · low: Intersect sorted id lists by merging, galloping from the short list into the long one, and skipping whole compressed blocks, shortest list first.›Time partitioningstorage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.›Gorilla XOR compressioncompression · low: Time-series codec: delta-of-delta timestamps plus XOR of consecutive float bit patterns, storing only changed bits; about 1.4 bytes per point.›Roaring bitmapscompression · low: Compressed set of 32-bit integers in 64K-value chunks, each a sorted array, bitmap or run list; fast intersections, the standard for bitmap indexes.›Quantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsString interningdata-structures · low: Store each distinct string once and refer to it by idmmap and the page cachehardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Bit packing and PFORcompression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›BM25 rankingsearch-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.›Bloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›K way mergesearch-sort · low: Merge k sorted streams with a heap or loser tree (merging iterators)Sstablestorage · low: Sorted string table: data blocks with restart points, index, filter and footerHash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to disk+ 9 more primitives

Similar systems

Share the most building blocks with this one.

metrics 52 %search-engine 23 %vector-db 21 %code-search 21 %realtime-olap 20 %