← Back
Home › Systems › Databases and storage engines

htap-db

one database for transactions and analytics

TiDB, SingleStore

SystemTechniques · 17Primitives · 35 htap-db Data shuffledistributed · medium: Hash/range repartitioning of intermediate data between workers (exchange operator)Multi raftdistributed · medium: Thousands of Raft groups per node: batched heartbeats, shared log engine, range split/mergePercolator transactionsdistributed · medium: Snapshot-isolation transactions over a KV store with a timestamp oracle and primary locksPrimary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Raftdistributed · medium: Leader-based consensus on a replicated log; one leader per term, majority commit; the default for a strongly consistent replicated state machine.›Cardinality estimationstorage · medium: Row-count estimates from histograms, MCVs, sketches and sampling for the optimizerColumnar storagestorage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Range partitioningdistributed · low: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.›Dictionary encodingcompression · low: Map values to small integer codes via a dictionary for compact, comparable columnsZone mapsstorage · low: Per-block min/max (and sparse primary) indexes to skip data during scansHash joinstorage · low: Build a hash table on the smaller input's join key and probe it with every row of the larger one; the linear-time workhorse for equi-joins.›Hash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskFast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Columnar memory formatstorage · low: In-memory column buffers with validity bitmaps and offsets (Arrow style) for zero-copy exchangeGroup commitstorage · low: Let concurrent committers share one log write and fsync; throughput grows with concurrency while each commit still waits at most about two fsyncs.›Distributed ID generationdistributed · low: Mint unique, roughly time-sorted IDs on many nodes with no per-ID coordination: Snowflake (time, worker, sequence) or UUIDv7 (time, random).›Leasesdistributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.›Write-ahead logstorage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.›Health checkingsystems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Exponential backoff with jittersystems · low: Retry after delays that grow geometrically and are randomized, so failed clients spread out instead of hammering a recovering service in synchronized waves.›HyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›Quantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsReservoir samplingdata-structures · low: Uniform k-sample from a stream of unknown length (Algorithm R/L)+ 10 more techniques+ 19 more primitives

Similar systems

Share the most building blocks with this one.

columnar-warehouse 30 %query-engine 26 %distributed-sql 26 %embedded-sql 24 %batch-engine 21 %