← Back
Home › Systems › Observability and operations

metrics

metrics collection and monitoring

Prometheus

SystemTechniques · 7Primitives · 33 metrics Quorum replicationdistributed · medium: Leaderless replication: write to W of N replicas, read from R; overlap (R + W > N) makes reads see the latest completed write.›Inverted indexstorage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›Rollup aggregationstorage · medium: Fold raw events into one mergeable partial aggregate per time bucket and dimension key, then merge fine buckets into coarser ones, so long ranges read few rows and stay exact.›Time-series chunk storagestorage · medium: Store each series as immutable compressed chunks of consecutive samples behind a small time index; a range query decodes only the chunks that overlap it.›Vectorized executionstorage · medium: Query operators pass batches of about 1-2K values per column instead of single tuples, amortizing interpretation overhead and enabling tight SIMD loops.›Anti-entropy repairdistributed · medium: Replicas compare hash trees of a key range, descend only into differing subtrees and stream just those ranges, healing silent divergence in the background.›Hinted handoffdistributed · medium: Park a write meant for a down replica as a tagged hint on a healthy node, replay it when the replica returns, and count it toward the write quorum only if availability matters more than consistency.›Consistent hashingdistributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.›Delta encodingcompression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›Varint (LEB128)compression · low: Variable-length integers in 7-bit groups with a continuation bit; small numbers take 1 byte, the wire format of Protobuf, WebAssembly and DWARF.›Time partitioningstorage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.›Gorilla XOR compressioncompression · low: Time-series codec: delta-of-delta timestamps plus XOR of consecutive float bit patterns, storing only changed bits; about 1.4 bytes per point.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Roaring bitmapscompression · low: Compressed set of 32-bit integers in 64K-value chunks, each a sorted array, bitmap or run list; fast intersections, the standard for bitmap indexes.›HyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›Quantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsString interningdata-structures · low: Store each distinct string once and refer to it by idSwiss tabledata-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.›mmap and the page cachehardware-os · low: How read() and mmap reach file data through the kernel page cache, what readahead and madvise do, and why databases use their own buffer pool instead.›Protobuf wire formatnetworking · low: Protocol Buffers wire format: tags, varints, length-delimited fields and schema evolutionBackpressuresystems · low: Bound every queue and make a full queue push back on the producer by blocking, dropping, rejecting or withholding credit.›Vector clocksdistributed · low: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.›Bit packing and PFORcompression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›+ 17 more primitives

Similar systems

Share the most building blocks with this one.

timeseries-db 52 %apm 31 %realtime-olap 24 %code-search 21 %logging 20 %

In-house systems

Named systems of this kind at large companies, with their paper or write-up.

MonarchAlphabetM3Uber Technologies