mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
AI and ML infrastructure
model-serving
model registry and serving
Triton, KServe
System
Techniques · 19
Primitives · 75
model-serving
Deterministic simulation testing
distributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.
›
Primary-backup replication
distributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.
›
Dynamic batching
ml-training · medium: Queue independent requests and dispatch them as one batch when it is full or the oldest has waited max_delay; bounded extra latency buys throughput.
›
Model registry
ml-training · medium: Versioned model artifacts with metadata, stages, lineage and immutable content hashes
Coverage-guided fuzzing
security · medium: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.
›
A/B testing (online controlled expe…
statistics · medium: Randomly split units into arms by a salted hash, log exposure, compare metrics with a confidence interval, and guard the result with sample-ratio, guardrail and peeking checks.
›
Point-in-time recovery
storage · medium: Periodic base backups plus a continuously archived change log let you rebuild the state as of any instant, then branch from there.
›
Containers and Kubernetes
cloud · low: Containers package a service with its dependencies; Kubernetes schedules them across machines. Use the smallest orchestrator that does the job.
›
Hdr histogram
data-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentiles
Process supervision
hardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.
›
Cuda graphs
ml-training · low: Capturing a sequence of GPU kernel launches once and replaying it to remove CPU launch overhead
Kernel fusion
ml-training · low: Merge chains of memory-bound GPU ops into one kernel, so intermediates stay in registers instead of round-tripping through HBM.
›
Lora adapters
ml-training · low: Low-rank adapter matrices for fine-tuning; QLoRA; serving many adapters on one base model
Weight quantization
ml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.
›
HTTP/2
networking · low: Binary multiplexed framing for HTTP over one TCP connection: streams, per-stream and per-connection credit windows, HPACK header compression and the limits that contain abuse.
›
Power of two choices
networking · low: Pick two random backends and send to the less loaded one; exponentially better balance than random, with no global state and no herding.
›
Protobuf wire format
networking · low: Protocol Buffers wire format: tags, varints, length-delimited fields and schema evolution
Zero-copy serialization
networking · low: Wire formats laid out like in-memory structs (FlatBuffers, Cap'n Proto, SBE, rkyv) so readers access fields in place with no decode pass or allocation.
›
Mutual TLS
security · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.
›
Circuit breaker
systems · low: Stop calling a failing dependency for a while, probe it with trial calls, and resume on recovery: fail fast instead of piling up timeouts.
›
Graceful draining
systems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.
›
Cooperative scheduling
concurrency · low: Green threads/coroutines yielding at await points on a user-space scheduler
Record replay
distributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuild
+ 12 more techniques
+ 59 more primitives
System
Techniques · 19
Primitives · 75
model-serving
Deterministic simulation testing
distributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.
›
Primary-backup replication
distributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.
›
Dynamic batching
ml-training · medium: Queue independent requests and dispatch them as one batch when it is full or the oldest has waited max_delay; bounded extra latency buys throughput.
›
Model registry
ml-training · medium: Versioned model artifacts with metadata, stages, lineage and immutable content hashes
Coverage-guided fuzzing
security · medium: Mutate inputs, run an instrumented target, and keep any input that reaches a new code edge; feedback turns blind guessing into incremental search.
›
A/B testing (online controlled expe…
statistics · medium: Randomly split units into arms by a salted hash, log exposure, compare metrics with a confidence interval, and guard the result with sample-ratio, guardrail and peeking checks.
›
Point-in-time recovery
storage · medium: Periodic base backups plus a continuously archived change log let you rebuild the state as of any instant, then branch from there.
›
Autoscaling
systems · medium: Target-tracking scaler with stabilization windows, cooldowns and hysteresis
Staged rollout
systems · medium: Rolling, blue-green and canary rollout with health gates and automatic rollback
Delta debugging
compilers · medium: Ddmin minimization of failing inputs or change sets
Fault injection
distributed · medium: Inject latency, errors, kills and partitions via proxies, netem or hooks
Hypothesis testing
statistics · medium: Pick a test statistic, compare it to its distribution under "no effect", read off a p-value, and control error rates across one test or many.
›
Sequential hypothesis testing
statistics · medium: Group-sequential and alpha-spending designs for interim analyses without inflating error
ARIES recovery
storage · medium: Three-pass crash recovery (analysis, redo, undo) for steal/no-force buffer pools; repeats history, then rolls back losers with compensation log records.
›
Burn rate alerting
systems · medium: Multi-window multi-burn-rate alerts on SLO error-budget consumption
Syscall interception
security · medium: Intercept syscalls via ptrace, seccomp user notify or a user-space kernel
Sequential probability ratio test
statistics · medium: Add up the log-likelihood ratio per observation and stop when it crosses a fixed bound; the least data of any test with the same error rates (for two point hypotheses).
›
Buffer pool
storage · medium: The database's own page cache (frames, page table, pin counts, dirty bits, replacement) so it, not the OS, controls eviction, write order and I/O.
›
Rollup aggregation
storage · medium: Fold raw events into one mergeable partial aggregate per time bucket and dimension key, then merge fine buckets into coarser ones, so long ranges read few rows and stay exact.
›
Containers and Kubernetes
cloud · low: Containers package a service with its dependencies; Kubernetes schedules them across machines. Use the smallest orchestrator that does the job.
›
Hdr histogram
data-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentiles
Process supervision
hardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.
›
Cuda graphs
ml-training · low: Capturing a sequence of GPU kernel launches once and replaying it to remove CPU launch overhead
Kernel fusion
ml-training · low: Merge chains of memory-bound GPU ops into one kernel, so intermediates stay in registers instead of round-tripping through HBM.
›
Lora adapters
ml-training · low: Low-rank adapter matrices for fine-tuning; QLoRA; serving many adapters on one base model
Weight quantization
ml-training · low: Store weights as 8/4-bit integers with per-group scales; decode is memory-bound, so 4-bit weights give about 3-4x tokens/s and fit 4x larger models.
›
HTTP/2
networking · low: Binary multiplexed framing for HTTP over one TCP connection: streams, per-stream and per-connection credit windows, HPACK header compression and the limits that contain abuse.
›
Power of two choices
networking · low: Pick two random backends and send to the less loaded one; exponentially better balance than random, with no global state and no herding.
›
Protobuf wire format
networking · low: Protocol Buffers wire format: tags, varints, length-delimited fields and schema evolution
Zero-copy serialization
networking · low: Wire formats laid out like in-memory structs (FlatBuffers, Cap'n Proto, SBE, rkyv) so readers access fields in place with no decode pass or allocation.
›
Mutual TLS
security · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.
›
Circuit breaker
systems · low: Stop calling a failing dependency for a while, probe it with trial calls, and resume on recovery: fail fast instead of piling up timeouts.
›
Graceful draining
systems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.
›
Cooperative scheduling
concurrency · low: Green threads/coroutines yielding at await points on a user-space scheduler
Record replay
distributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuild
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Leases
distributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.
›
Write-ahead log
storage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.
›
Health checking
systems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.
›
MPMC bounded queue
concurrency · low: Vyukov's array queue for many producers and consumers: each slot carries a sequence number, so one CAS per operation claims a slot without locks.
›
Backpressure
systems · low: Bound every queue and make a full queue push back on the producer by blocking, dropping, rejecting or withholding credit.
›
Hierarchical timing wheel
systems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.
›
Object storage
cloud · low: S3-compatible blob storage at cents per GB-month with 11 nines durability; pick by egress price, request price and latency, not by storage price alone.
›
Conditional writes
distributed · low: Put-if-absent / if-match compare-and-swap on objects or keys as a commit primitive
Model file formats
ml-training · low: Weight file layouts (safetensors, GGUF, ONNX): headers, alignment for mmap, tensor metadata, quantized blocks
Cryptographic hashes (SHA-2, BLAKE3)
security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.
›
Content-addressed storage
storage · low: Store each blob under the hash of its bytes: immutable, deduplicated by construction, self-verifying on every read, and composable into Merkle trees.
›
Code coverage instrumentation
compilers · low: Insert counters into compiled code to record which blocks, edges or lines ran; a spanning tree of the control-flow graph cuts the counters needed.
›
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Sanitizers
security · low: ASan/UBSan/TSan shadow memory and instrumentation to catch memory bugs
Bootstrap confidence intervals
statistics · low: Resampling-based confidence intervals for arbitrary statistics
CUPED (variance reduction with pre-…
statistics · low: Subtract the part of each unit's outcome that a pre-experiment covariate predicts; the effect estimate stays unbiased and its variance shrinks by 1 - rho^2.
›
Experiment layering (overlapping ex…
statistics · low: Partition the tunable parameters into layers, hash each unit with a per-layer salt so layers are independent, and give experiments inside a layer disjoint bucket ranges.
›
Deterministic bucketing
systems · low: Hash a stable unit id with a per-purpose salt into a bucket; "in" means bucket below the percentage, so assignment is stateless, repeatable and only grows on ramp-up.
›
Copy on write snapshots
storage · low: Point-in-time snapshots by sharing pages/extents and copying only on write
PID controller
robotics · low: Drive a measured value to a setpoint with three terms on the error: proportional, integral, derivative. The default feedback loop for motors, heaters, drones and gimbals.
›
Exponential smoothing (ETS, Holt-Wi…
statistics · low: Forecast by keeping a few running states (level, trend, seasonal) updated by a fraction of each forecast error; O(1) memory per series and a strong baseline.
›
Sliding-window counters
systems · low: Count events in the last W seconds with a ring of sub-bucket counters; the two-bucket weighted estimate gives rate limits in 16 bytes per key.
›
Traffic splitting
systems · low: Weighted routing between versions for canaries and A/B, sticky by hash
Failure bisection
search-sort · low: Binary search over commits, batches or inputs to isolate a failure
Transparent interception
networking · low: Redirect traffic to a local proxy (iptables TPROXY/REDIRECT, eBPF) without app changes
Wilson score interval
statistics · low: Wilson lower bound for ranking by up/down votes and proportions with few samples
Bayesian rate estimation
statistics · low: Estimate a conversion or event rate from few observations with a Beta-Binomial or Gamma-Poisson update: two counters, an honest interval, and P(B beats A) directly.
›
Ebpf
hardware-os · low: Verified in-kernel programs on hooks (kprobes, XDP, tc) with maps
Ptrace process control
hardware-os · low: Ptrace: attach, read and write registers and memory, single-step, intercept signals
Seccomp bpf
security · low: Syscall filtering with seccomp BPF programs
Loglikelihood scoring
machine-learning · low: Scoring choices by summed token log-probabilities, normalization by length, perplexity computation
fsync, O_DSYNC and O_DIRECT
hardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.
›
LRU and CLOCK
systems · low: LRU evicts the least recently used entry via hash map plus linked list; CLOCK approximates it with a reference bit, so hits take no lock.
›
S3-FIFO
systems · low: Three FIFO queues (small, main, ghost) that evict one-hit wonders quickly and keep reused objects; low miss ratios with no list moves on hits.
›
Hash aggregation
storage · low: GROUP BY via hash tables with partitioning and spill to disk
Time partitioning
storage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.
›
Control-flow graph
compilers · low: Split a linear instruction list into basic blocks and connect them by jumps; the graph every optimizer, SSA builder and dataflow pass walks.
›
False sharing
concurrency · low: Independent variables written by different cores on one cache line force the line to bounce between cores; pad or regroup hot data to fix it.
›
Memory ordering
concurrency · low: What Relaxed, Acquire, Release and SeqCst atomics actually guarantee, what they cost on x86 and ARM, and which one each pattern needs.
›
d-ary heap
data-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.
›
Merkle dag
data-structures · low: Content-addressed DAG of hashed nodes (git trees/commits) for dedupe and verifiable history
Fencing tokens
distributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.
›
BFS and DFS
graphs · low: The two basic graph traversals; BFS gives hop distances and levels, DFS gives discovery/finish order for cycles, components and topological order.
›
Minimum spanning tree
graphs · low: Cheapest set of edges connecting all vertices; Kruskal sorts edges and joins components with union-find, Prim grows one tree, Boruvka merges in parallel rounds.
›
Union-find
graphs · low: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.
›
Atomic file replacement
hardware-os · low: Write temp file, fsync, rename over target, fsync directory: crash-safe whole-file updates
Cgroups and namespaces
hardware-os · low: Linux namespaces limit what a process can see; cgroup v2 limits and accounts what it may use. Containers, sandboxes and job schedulers are built from the two.
›
CPU caches and the memory hierarchy
hardware-os · low: Sizes, line widths and latencies of L1/L2/L3/DRAM on modern x86 and Apple M-series, and the data-layout rules that follow from them.
›
CRC32C checksums
hardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.
›
Gpu memory coalescing banking
hardware-os · low: Coalesced global loads, shared memory bank conflicts, L2 and HBM bandwidth on GPUs
Overlay filesystem
hardware-os · low: Union mounts (overlayfs) stacking read-only layers under a writable layer
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
CUDA and GPU languages
languages · low: Programming GPUs: thousands of lockstep threads, bound by memory bandwidth; libraries first, Triton for fused kernels, CUDA C++ for peak speed.
›
Tiled matrix multiplication
ml-training · low: Block GEMM into tiles held in shared memory and registers, so each byte loaded from DRAM feeds many FMAs; the core of fast GEMM.
›
Low precision number formats
numerics · low: Bf16, fp16, fp8 (E4M3, E5M2), MX block formats and int4: ranges, rounding, scaling
LU and Cholesky factorization
numerics · low: Factor A = PLU (any square matrix, partial pivoting) or A = LLᵀ (SPD) once in O(n³), then solve each right-hand side in O(n²).
›
Sparse matrices (CSR)
numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.
›
Slab allocator
systems · low: Pre-carve pages into equal-size slots per size class with an in-place free list: O(1) alloc and free, no external fragmentation, cache-warm reuse.
›
Copy for AI
Similar systems
Share the most building blocks with this one.
rpc-framework
39 %
chaos-engineering
33 %
service-mesh
31 %
global-rate-limiter
29 %
l7-proxy
29 %