← Back
Home › Systems › Biotech and life sciences

variant-database

genomic variant database

gnomAD

SystemTechniques · 6Primitives · 31 variant-database External merge sortsearch-sort · medium: Sort data larger than RAM: write memory-sized sorted runs, then k-way merge them with large sequential buffers; terabytes in one merge pass.›Differential privacysecurity · medium: Calibrated noise (Laplace/Gaussian) and privacy budgets for aggregate releasesColumnar storagestorage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Inverted indexstorage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.›Vectorized executionstorage · medium: Query operators pass batches of about 1-2K values per column instead of single tuples, amortizing interpretation overhead and enabling tight SIMD loops.›Privacy budget accountingsecurity · medium: Tracking cumulative epsilon/delta under composition (basic, advanced, Renyi/moments accountant)Bit packing and PFORcompression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›Object storagecloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.›Huffman codingcompression · low: Optimal prefix code with whole-bit codeword lengths; table-driven decoding at about one symbol per few cycles, the entropy stage of DEFLATE, JPEG and zstd literals.›LZ77compression · low: Replace repeated byte strings with (distance, length) back-references into a sliding window; the parsing stage of gzip, LZ4, zstd, Brotli and xz.›Interval treedata-structures · low: Index of ranges that answers "which intervals overlap this point or range?" in O(log n + k); static sorted-array form is the compact, cache-friendly choice.›Sparse matrices (CSR)numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.›Genomic binning indexsearch-sort · low: Hierarchical bin index (UCSC/tabix/BAI) for region queries over sorted genomic recordsVcf variant normalizationsearch-sort · low: Parse VCF and left-align/trim indels so equal variants share one representationB+ treestorage · low: Balanced high-fanout search tree with all values in linked leaves; the default on-disk index for point lookups and range scans.›Hash joinstorage · low: Build a hash table on the smaller input's join key and probe it with every row of the larger one; the linear-time workhorse for equi-joins.›Sort-merge joinstorage · low: Sort both inputs on the key, then advance two cursors in lockstep and emit matching runs; cheap on sorted inputs and good for range joins.›Buffer pools for zero-allocation I/Ohardware-os · low: Preallocate a fixed set of aligned I/O buffers and recycle them through a free list, so the hot path never mallocs, faults or unmaps. The bounded count doubles as back pressure.›K way mergesearch-sort · low: Merge k sorted streams with a heap or loser tree (merging iterators)Pattern-defeating quicksort (pdqsor…search-sort · low: Introsort successor: quicksort with adaptive pivots, linear on sorted and equal-heavy input, heapsort fallback; the unstable sort of Go and (until 1.81) Rust.›Floating-point errornumerics · low: How IEEE 754 rounding error accumulates and how to contain it: compensated (Kahan/Neumaier) and pairwise summation, error-free transformations with FMA, ULP comparisons.›Csprngsecurity · low: OS-backed cryptographically secure random numbers (getrandom), and when a fast PRNG is not enough+ 15 more primitives

Similar systems

Share the most building blocks with this one.

query-optimizer 24 %columnar-warehouse 19 %batch-engine 18 %realtime-olap 18 %embedded-sql 17 %