mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
Consumer applications
moderation
content moderation pipeline
YouTube, Meta
System
Techniques · 6
Primitives · 40
moderation
Partitioned commit log
distributed · medium: A topic split into replicated append-only partitions: key hash keeps per-key order, offsets give position, and a high watermark limits readers to fully replicated records.
›
Graph community detection
graphs · medium: Louvain/label propagation community detection for rings, fraud clusters and campaigns
Active learning
machine-learning · medium: Choosing which unlabeled examples to label next: uncertainty, margin, diversity and committee sampling
HNSW
search-sort · medium: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.
›
Unicode confusables
security · medium: UTS #39 maps every string to a "skeleton" so look-alikes collide, and restricts script mixing so most spoofs never get registered.
›
Primary-backup replication
distributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.
›
Calibration
machine-learning · low: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.
›
Embeddings
machine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.
›
d-ary heap
data-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.
›
Evaluation metrics
machine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.
›
Continuous batching
ml-training · low: Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.
›
Convolutional networks
ml-training · low: Layers that slide a small shared kernel over a grid, giving translation equivariance and few parameters; still the most compute-efficient choice for many vision tasks.
›
Transformer block
ml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.
›
Aho-Corasick multi-pattern matching
search-sort · low: Find all occurrences of many fixed patterns in one pass over the text, in time independent of the pattern count, using a trie with failure links.
›
Perceptual hashing
search-sort · low: Compact fingerprints (64 to 256 bits) of an image that stay close under resizing and recompression, compared by Hamming distance; pHash, dHash and Meta's PDQ.
›
SimHash
search-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.
›
Sliding-window counters
systems · low: Count events in the last W seconds with a ring of sub-bucket counters; the two-bucket weighted estimate gives rate limits in 16 bytes per key.
›
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Consumer offsets
distributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.
›
Segmented log
distributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.
›
Zero-copy I/O
networking · low: Move bytes between files, sockets and NICs by passing page references instead of copying through user space: sendfile, splice, MSG_ZEROCOPY, kTLS.
›
Union-find
graphs · low: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.
›
+ 24 more primitives
System
Techniques · 6
Primitives · 40
moderation
Partitioned commit log
distributed · medium: A topic split into replicated append-only partitions: key hash keeps per-key order, offsets give position, and a high watermark limits readers to fully replicated records.
›
Graph community detection
graphs · medium: Louvain/label propagation community detection for rings, fraud clusters and campaigns
Active learning
machine-learning · medium: Choosing which unlabeled examples to label next: uncertainty, margin, diversity and committee sampling
HNSW
search-sort · medium: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.
›
Unicode confusables
security · medium: UTS #39 maps every string to a "skeleton" so look-alikes collide, and restricts script mixing so most spoofs never get registered.
›
Primary-backup replication
distributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.
›
Calibration
machine-learning · low: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.
›
Embeddings
machine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.
›
d-ary heap
data-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.
›
Evaluation metrics
machine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.
›
Continuous batching
ml-training · low: Schedule LLM inference per iteration, not per request; finished sequences leave and new ones join the running batch every step, which keeps the GPU full.
›
Convolutional networks
ml-training · low: Layers that slide a small shared kernel over a grid, giving translation equivariance and few parameters; still the most compute-efficient choice for many vision tasks.
›
Transformer block
ml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.
›
Aho-Corasick multi-pattern matching
search-sort · low: Find all occurrences of many fixed patterns in one pass over the text, in time independent of the pattern count, using a trie with failure links.
›
Perceptual hashing
search-sort · low: Compact fingerprints (64 to 256 bits) of an image that stay close under resizing and recompression, compared by Hamming distance; pHash, dHash and Meta's PDQ.
›
SimHash
search-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.
›
Sliding-window counters
systems · low: Count events in the last W seconds with a ring of sub-bucket counters; the two-bucket weighted estimate gives rate limits in 16 bytes per key.
›
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Consumer offsets
distributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.
›
Segmented log
distributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.
›
Zero-copy I/O
networking · low: Move bytes between files, sockets and NICs by passing page references instead of copying through user space: sendfile, splice, MSG_ZEROCOPY, kTLS.
›
Union-find
graphs · low: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.
›
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Sparse matrices (CSR)
numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.
›
k-means clustering
machine-learning · low: Partition points into k clusters by alternating nearest-centroid assignment and centroid averaging; k-means++ seeding makes it dependable.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
Scalar quantization
search-sort · low: Per-dimension int8/binary quantization of vectors with rescoring
Text normalization
machine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.
›
Unicode text tokenization for search
search-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.
›
Leases
distributed · low: A lock that expires on its own: bounded-time ownership that survives crashes and partitions, given bounded clock drift and fencing tokens on writes.
›
Write-ahead log
storage · low: Append each change to a checksummed sequential log and fsync it before touching data pages; the basis of crash durability in nearly every database.
›
Health checking
systems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.
›
Reference counting
compilers · low: Keep a count of owners in each object, free it when the count hits zero; deterministic and local, but pays on every copy and leaks cycles.
›
Fencing tokens
distributed · low: A number that grows with every lock grant and is checked by the protected resource, so a paused ex-holder's late write is rejected.
›
Message sequence numbers
distributed · low: A per-stream counter on every message lets the receiver detect loss, duplicates and reordering, ack cheaply and resume after reconnect with "send from N".
›
CRC32C checksums
hardware-os · low: The Castagnoli CRC-32 with dedicated instructions on x86 (SSE4.2) and ARMv8. It is the standard fast integrity check for storage blocks, logs and network frames.
›
fsync, O_DSYNC and O_DIRECT
hardware-os · low: What fsync, fdatasync, O_DSYNC, O_DIRECT and F_FULLFSYNC really guarantee on Linux and macOS, and the exact call sequences for crash-safe files.
›
Linear and logistic regression
machine-learning · low: Fit a weighted sum of features to a number (least squares) or a probability (logistic); the baseline every model must beat.
›
Loss functions
machine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.
›
AdamW
ml-training · low: Adam's per-parameter adaptive steps with weight decay applied directly to the weights; the default optimizer for transformers and LLM pretraining.
›
Paged KV cache
ml-training · low: Store each sequence's attention keys/values in fixed-size blocks mapped through a block table, like virtual memory pages; near-zero fragmentation and copy-on-write sharing.
›
Reverse-mode automatic differentiat…
ml-training · low: Record the forward computation on a tape, then sweep it backwards once to get the gradient of a scalar loss for all inputs.
›
Token sampling
ml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.
›
L-BFGS
numerics · low: Quasi-Newton minimizer that approximates the inverse Hessian from the last m gradient differences via the two-loop recursion; O(m·n) memory and work per step.
›
LU and Cholesky factorization
numerics · low: Factor A = PLU (any square matrix, partial pivoting) or A = LLᵀ (SPD) once in O(n³), then solve each right-hand side in O(n²).
›
Bootstrap confidence intervals
statistics · low: Resampling-based confidence intervals for arbitrary statistics
Copy for AI
Similar systems
Share the most building blocks with this one.
abuse-classifier
33 %
prompt-injection-detection
20 %
aml-monitoring
19 %
machine-translation
19 %
recommender
17 %