← Back
Home › Systems › Consumer applications

professional-network

professional network and people search

LinkedIn

SystemTechniques · 16Primitives · 52 professional-network Fan-out on writedistributed · medium: Append each new item id to every follower's precomputed, capped list so a read is one fetch; pull the few huge authors at read time instead.›Partitioned commit logdistributed · medium: A topic split into replicated append-only partitions: key hash keeps per-key order, offsets give position, and a high watermark limits readers to fully replicated records.›Learning to rankmachine-learning · medium: Train a scorer on query-grouped data so that the order it induces maximizes NDCG; LambdaMART (boosted trees with NDCG-weighted pair gradients) is the production default.›Two-tower retrievalmachine-learning · medium: Two separate encoders map users/queries and items into one vector space; item vectors are precomputed and searched by nearest neighbour, so recall over millions costs one small forward pass.›HNSWsearch-sort · medium: Hierarchical Navigable Small World: a layered proximity graph searched greedily from a sparse top layer; the default in-memory approximate nearest-neighbour index.›Change data capturestorage · medium: Decode the database's own commit log into row-change events in commit order, with a resume position, so any consumer can mirror the data exactly.›Entity resolutionstorage · medium: Decide which records describe the same real-world thing: block to cut the pair space, score pairs with a probabilistic model, cluster the matches transitively.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›BM25 rankingsearch-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.›Fuzzy string matchingsearch-sort · low: Edit distance, Jaro-Winkler and trigram indexes for approximate name and address matchingRoaring bitmapscompression · low: Compressed set of 32-bit integers in 64K-value chunks, each a sorted array, bitmap or run list; fast intersections, the standard for bitmap indexes.›HyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›Consistent hashingdistributed · low: Hash nodes and keys onto a ring; each key belongs to the next node clockwise, so membership changes move only about 1/n of keys.›BFS and DFSgraphs · low: The two basic graph traversals; BFS gives hop distances and levels, DFS gives discovery/finish order for cycles, components and topological order.›Finite state transducersearch-sort · low: Minimal acyclic FST mapping sorted keys to values; compact term dictionaries with prefix/fuzzy searchRange partitioningdistributed · low: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.›Fast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Consumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Segmented logdistributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›Zero-copy I/Onetworking · low: Move bytes between files, sockets and NICs by passing page references instead of copying through user space: sendfile, splice, MSG_ZEROCOPY, kTLS.›Evaluation metricsmachine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.›Loss functionsmachine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›Negative samplingmachine-learning · low: Train by contrasting each observed positive with a few sampled negatives instead of normalising over every candidate; the choice of sampler decides bias and quality.›+ 9 more techniques+ 36 more primitives

Similar systems

Share the most building blocks with this one.

product-search 30 %news-ranking 23 %workplace-search 21 %aml-monitoring 19 %agent-memory 18 %