← Back
Home › Systems › AI applications

web-search

web search engine with crawler and ranking

Google, Bing

SystemTechniques · 13Primitives · 44 web-search Pagerankgraphs · medium: Power-iteration PageRank on a sparse graph with damping and dangling nodesHTML main-text extractionmachine-learning · medium: Decode bytes, tolerate broken markup, drop non-content nodes and score text blocks by length and link density to keep the main run, with headings, metadata and offsets for citation.›Learning to rankmachine-learning · medium: Train a scorer on query-grouped data so that the order it induces maximizes NDCG; LambdaMART (boosted trees with NDCG-weighted pair gradients) is the production default.›Near-duplicate detection with MinHa…machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Top-k pruning with score upper boun…search-sort · medium: Skip documents that provably cannot enter the top-k, using per-list and per-block maximum scores against the current k-th best score (WAND, Block-Max WAND, MaxScore).›Cross-encoder rerankingsearch-sort · medium: Re-score the top-k candidates of a cheap first stage with a model that reads query and candidate together; more accurate than separate embeddings, but one model call per pair.›External merge sortsearch-sort · medium: Sort data larger than RAM: write memory-sized sorted runs, then k-way merge them with large sequential buffers; terabytes in one merge pass.›Unicode text tokenization for searchsearch-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.›Bit packing and PFORcompression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›BM25 rankingsearch-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.›Top-k selectionsearch-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.›Varint (LEB128)compression · low: Variable-length integers in 7-bit groups with a continuation bit; small numbers take 1 byte, the wire format of Protobuf, WebAssembly and DWARF.›Edit distancesearch-sort · low: Levenshtein/Damerau distance by DP, bit-parallel Myers variantStream VBytecompression · low: Integer codec that keeps 2-bit lengths in a separate control stream, so one table-driven SIMD shuffle decodes 4 integers without branches.›Language identificationmachine-learning · low: Score a text's character n-grams against per-language statistics, after a cheap script check, and return an ISO 639 code with a confidence or "und".›DNS message format and client looku…networking · low: Binary query/response format of DNS (RFC 1035) and how a stub client uses it: record types, name compression, UDP/TCP/EDNS sizes, TTL caching.›Finite state transducersearch-sort · low: Minimal acyclic FST mapping sorted keys to values; compact term dictionaries with prefix/fuzzy searchSimHashsearch-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.›Hedged requestssystems · low: Sending a backup request after a latency percentile and taking the first answer to cut tail latencyLRU and CLOCKsystems · low: LRU evicts the least recently used entry via hash map plus linked list; CLOCK approximates it with a reference bit, so hits take no lock.›Reproducible reductionnumerics · low: Order-independent floating-point sums (fixed order, Kahan, binned) for bit-for-bit reproducibilitySparse matrices (CSR)numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.›Text normalizationmachine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›+ 6 more techniques+ 28 more primitives

Similar systems

Share the most building blocks with this one.

news-ranking 34 %product-search 27 %search-engine 24 %pretraining-data 18 %rag 16 %