mhtirogla
Home
Systems
API
Pricing
About
Log in
← Back
Home
›
Systems
›
Consumer applications
news-ranking
news aggregation and ranking
Google News
System
Techniques · 9
Primitives · 35
news-ranking
Pagerank
graphs · medium: Power-iteration PageRank on a sparse graph with damping and dangling nodes
Learning to rank
machine-learning · medium: Train a scorer on query-grouped data so that the order it induces maximizes NDCG; LambdaMART (boosted trees with NDCG-weighted pair gradients) is the production default.
›
Near-duplicate detection with MinHa…
machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.
›
Two-tower retrieval
machine-learning · medium: Two separate encoders map users/queries and items into one vector space; item vectors are precomputed and searched by nearest neighbour, so recall over millions costs one small forward pass.
›
HNSW
search-sort · medium: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.
›
Url frontier
search-sort · medium: Prioritised per-host queues with politeness delays for crawling
Inverted index
storage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.
›
Embeddings
machine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.
›
Bloom filter
data-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.
›
BM25 ranking
search-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.
›
Union-find
graphs · low: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.
›
Maximal marginal relevance
search-sort · low: MMR and other diversification re-ranking of result lists
SimHash
search-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.
›
Time decay scoring
search-sort · low: Scoring items by recency with exponential decay or half-life, combined with relevance
Top-k selection
search-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.
›
Unicode text tokenization for search
search-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.
›
Reproducible reduction
numerics · low: Order-independent floating-point sums (fixed order, Kahan, binned) for bit-for-bit reproducibility
Sparse matrices (CSR)
numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.
›
Evaluation metrics
machine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.
›
Loss functions
machine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.
›
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Rolling hash
data-structures · low: Rabin-Karp polynomial rolling hashes and winnowing for substring fingerprints and document matching
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
+ 2 more techniques
+ 19 more primitives
System
Techniques · 9
Primitives · 35
news-ranking
Pagerank
graphs · medium: Power-iteration PageRank on a sparse graph with damping and dangling nodes
Learning to rank
machine-learning · medium: Train a scorer on query-grouped data so that the order it induces maximizes NDCG; LambdaMART (boosted trees with NDCG-weighted pair gradients) is the production default.
›
Near-duplicate detection with MinHa…
machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.
›
Two-tower retrieval
machine-learning · medium: Two separate encoders map users/queries and items into one vector space; item vectors are precomputed and searched by nearest neighbour, so recall over millions costs one small forward pass.
›
HNSW
search-sort · medium: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.
›
Url frontier
search-sort · medium: Prioritised per-host queues with politeness delays for crawling
Inverted index
storage · medium: Map each term to a sorted, compressed list of the documents containing it; the core of full-text search and other fast set queries.
›
Contrastive learning
machine-learning · medium: Train an encoder so two views of the same thing score higher than every other pair in the batch; InfoNCE with a temperature, no labels needed.
›
Gradient-boosted decision trees (GB…
machine-learning · medium: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.
›
Embeddings
machine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.
›
Bloom filter
data-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.
›
BM25 ranking
search-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.
›
Union-find
graphs · low: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.
›
Maximal marginal relevance
search-sort · low: MMR and other diversification re-ranking of result lists
SimHash
search-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.
›
Time decay scoring
search-sort · low: Scoring items by recency with exponential decay or half-life, combined with relevance
Top-k selection
search-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.
›
Unicode text tokenization for search
search-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.
›
Reproducible reduction
numerics · low: Order-independent floating-point sums (fixed order, Kahan, binned) for bit-for-bit reproducibility
Sparse matrices (CSR)
numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.
›
Evaluation metrics
machine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.
›
Loss functions
machine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.
›
Fast hash functions
data-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.
›
Rolling hash
data-structures · low: Rabin-Karp polynomial rolling hashes and winnowing for substring fingerprints and document matching
Swiss table
data-structures · low: Open-addressing hash table that filters 8-16 slots at once with 1-byte hash tags and SIMD, giving fast lookups at 87.5 % load.
›
Negative sampling
machine-learning · low: Train by contrasting each observed positive with a few sampled negatives instead of normalising over every candidate; the choice of sampler decides bias and quality.
›
d-ary heap
data-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.
›
SIMD programming
hardware-os · low: Process 4-16 lanes per instruction with AVX2, AVX-512 or NEON via Rust std::arch, with runtime dispatch and a portable fallback.
›
PRNGs (PCG, xoshiro)
numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.
›
Scalar quantization
search-sort · low: Per-dimension int8/binary quantization of vectors with rescoring
Robots exclusion protocol
search-sort · low: Decide per URL whether a crawler may fetch it from a host's robots.txt (RFC 9309): status-code rules, group choice, longest-match Allow/Disallow, 24 h cache.
›
URL normalization
search-sort · low: Rewrite a URL to one canonical key, in safe RFC 3986 steps plus lossy, site-dependent ones (tracking parameters, query order, fragments), and keep the original beside the key.
›
Token bucket
systems · low: A bucket of up to b tokens refilled at rate r; requests spend tokens, enforcing an average rate while allowing bursts of b.
›
Bit packing and PFOR
compression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.
›
Delta encoding
compression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.
›
Varint (LEB128)
compression · low: Variable-length integers in 7-bit groups with a continuation bit; small numbers take 1 byte, the wire format of Protobuf, WebAssembly and DWARF.
›
Sorted-list intersection
search-sort · low: Intersect sorted id lists by merging, galloping from the short list into the long one, and skipping whole compressed blocks, shortest list first.
›
Data augmentation
machine-learning · low: Generate label-preserving variants of each training example on the fly to enlarge the effective dataset and bake in invariances.
›
Quantile sketches
data-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histograms
Regularization
machine-learning · low: Techniques that trade training fit for generalization: weight decay, dropout, early stopping, label smoothing and data mixing, with the exact update rules and defaults.
›
Text normalization
machine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.
›
AdamW
ml-training · low: Adam's per-parameter adaptive steps with weight decay applied directly to the weights; the default optimizer for transformers and LLM pretraining.
›
Reverse-mode automatic differentiat…
ml-training · low: Record the forward computation on a tape, then sweep it backwards once to get the gradient of a scalar loss for all inputs.
›
Bootstrap confidence intervals
statistics · low: Resampling-based confidence intervals for arbitrary statistics
Copy for AI
Similar systems
Share the most building blocks with this one.
web-search
34 %
product-search
30 %
threat-intel-platform
26 %
professional-network
23 %
agent-memory
22 %