mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
Observability and operations
logging
log aggregation and query
Loki, ELK
System
Techniques · 9
Primitives · 39
logging
Scatter-gather query
distributed · medium: Send a query only to shards that can match, merge their small partial results, and bound the wait for stragglers.
›
Quorum replication
distributed · medium: Leaderless replication: write to W of N replicas, read from R; overlap (R + W > N) makes reads see the latest completed write.
›
Log template mining
search-sort · medium: Extract log templates and parameters by prefix-tree clustering (Drain)
Linear-time regular expression matc…
search-sort · medium: Match a regular expression in O(pattern x text) worst case by tracking all live parse positions at once, then cache the state sets as a lazily built DFA for speed.
›
Columnar storage
storage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.
›
Inverted index
storage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.
›
Anti-entropy repair
distributed · medium: Replicas compare hash trees of a key range, descend only into differing subtrees and stream just those ranges, healing silent divergence in the background.
›
Consistent hashing
distributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.
›
Unicode text tokenization for search
search-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.
›
Object storage
cloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.
›
LZ4
compression · low: Byte-aligned LZ77 with no entropy stage; compresses at hundreds of MB/s and decompresses at several GB/s per core, for when speed beats ratio.
›
Zstandard (zstd)
compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.
›
Bloom filter
data-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.
›
String interning
data-structures · low: Store each distinct string once and refer to it by id
Segmented log
distributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.
›
Simd json parsing
search-sort · low: Two-stage SIMD JSON parsing (structural index, then tape) at GB/s
Time partitioning
storage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.
›
Backpressure
systems · low: Bound every queue and make a full queue push back on the producer by blocking, dropping, rejecting or withholding credit.
›
K way merge
search-sort · low: Merge k sorted streams with a heap or loser tree (merging iterators)
Top-k selection
search-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.
›
Deadline propagation
systems · low: Carry an absolute deadline and cancellation along a call chain
Hedged requests
systems · low: Sending a backup request after a latency percentile and taking the first answer to cut tail latency
Vector clocks
distributed · low: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.
›
+ 2 more techniques
+ 23 more primitives
System
Techniques · 9
Primitives · 39
logging
Scatter-gather query
distributed · medium: Send a query only to shards that can match, merge their small partial results, and bound the wait for stragglers.
›
Quorum replication
distributed · medium: Leaderless replication: write to W of N replicas, read from R; overlap (R + W > N) makes reads see the latest completed write.
›
Log template mining
search-sort · medium: Extract log templates and parameters by prefix-tree clustering (Drain)
Linear-time regular expression matc…
search-sort · medium: Match a regular expression in O(pattern x text) worst case by tracking all live parse positions at once, then cache the state sets as a lazily built DFA for speed.
›
Columnar storage
storage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.
›
Inverted index
storage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.
›
Anti-entropy repair
distributed · medium: Replicas compare hash trees of a key range, descend only into differing subtrees and stream just those ranges, healing silent divergence in the background.
›
Hinted handoff
distributed · medium: Park a write meant for a down replica as a tagged hint on a healthy node, replay it when the replica returns, and count it toward the write quorum only if availability matters more than consistency.
›
Vectorized execution
storage · medium: Query operators pass batches of about 1-2K values per column instead of single tuples, amortizing interpretation overhead and enabling tight SIMD loops.
›
Consistent hashing
distributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.
›
Unicode text tokenization for search
search-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.
›
Object storage
cloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.
›
LZ4
compression · low: Byte-aligned LZ77 with no entropy stage; compresses at hundreds of MB/s and decompresses at several GB/s per core, for when speed beats ratio.
›
Zstandard (zstd)
compression · low: LZ77 plus Huffman and FSE entropy coding, with levels from LZ4-like speed to xz-like ratio, all decoding at about 1-2 GB/s.
›
Bloom filter
data-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.
›
String interning
data-structures · low: Store each distinct string once and refer to it by id
Segmented log
distributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.
›
Simd json parsing
search-sort · low: Two-stage SIMD JSON parsing (structural index, then tape) at GB/s
Time partitioning
storage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.
›
Backpressure
systems · low: Bound every queue and make a full queue push back on the producer by blocking, dropping, rejecting or withholding credit.
›
K way merge
search-sort · low: Merge k sorted streams with a heap or loser tree (merging iterators)
Top-k selection
search-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.
›
Deadline propagation
systems · low: Carry an absolute deadline and cancellation along a call chain
Hedged requests
systems · low: Sending a backup request after a latency percentile and taking the first answer to cut tail latency
Vector clocks
distributed · low: One counter per node; comparing two vectors tells whether events are ordered or concurrent, which is exactly what conflict detection needs.
›
Radix trie
data-structures · low: Compressed prefix tree for path routing, domain matching and prefix lookups
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
Recursive-descent parsing
compilers · low: Hand-written top-down parser with one function per grammar rule; simple, fast, with full control over error messages and recovery.
›
Aho-Corasick multi-pattern matching
search-sort · low: Find all occurrences of many fixed patterns in one pass over the text, in time independent of the pattern count, using a trie with failure links.
›
SIMD substring search
search-sort · low: Find bytes or substrings by comparing 16-64 haystack positions per instruction against the needle's first and last byte, then verifying the rare candidates.
›
LRU and CLOCK
systems · low: LRU evicts the least recently used entry via hash map plus linked list; CLOCK approximates it with a reference bit, so hits take no lock.
›
Bit packing and PFOR
compression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.
›
Delta encoding
compression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.
›
Dictionary encoding
compression · low: Map values to small integer codes via a dictionary for compact, comparable columns
Zone maps
storage · low: Per-block min/max (and sparse primary) indexes to skip data during scans
Varint (LEB128)
compression · low: Variable-length integers in 7-bit groups with a continuation bit; small numbers take 1 byte, the wire format of Protobuf, WebAssembly and DWARF.
›
BM25 ranking
search-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.
›
Sorted-list intersection
search-sort · low: Intersect sorted id lists by merging, galloping from the short list into the long one, and skipping whole compressed blocks, shortest list first.
›
Merkle tree
data-structures · low: Hash tree for verifying and diffing large datasets with logarithmic proofs
Range partitioning
distributed · low: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.
›
Token bucket
systems · low: A bucket of up to b tokens refilled at rate r; requests spend tokens, enforcing an average rate while allowing bursts of b.
›
Phi accrual failure detector
distributed · low: Instead of a fixed heartbeat timeout, output a continuous suspicion level φ from the observed heartbeat interval distribution; adapts to each link's jitter.
›
Write-ahead log
storage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.
›
d-ary heap
data-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.
›
CRC32C checksums
hardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.
›
fsync, O_DSYNC and O_DIRECT
hardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
Text normalization
machine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.
›
Copy for AI
Similar systems
Share the most building blocks with this one.
log-analytics
30 %
tracing
23 %
siem
21 %
metrics
20 %
apm
18 %