← Back
Home › Systems › Consumer applications

encyclopedia

collaborative encyclopedia at scale

Wikipedia, MediaWiki

SystemTechniques · 17Primitives · 65 encyclopedia CDN and edgecloud · medium: CDNs cache content in hundreds of PoPs near users, cutting latency and egress bills; edge compute runs small code there. Bandwidth price spreads 10x between providers.›Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Partitioned commit logdistributed · medium: A topic split into replicated append-only partitions: key hash keeps per-key order, offsets give position, and a high watermark limits readers to fully replicated records.›Primary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Three-way mergedistributed · medium: Merge two descendants against their common ancestor: take whichever side changed, report a conflict only where both changed the same spot differently.›Gradient-boosted decision trees (GB…machine-learning · medium: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.›Myers diffsearch-sort · medium: Greedy O(ND) shortest edit script between two sequences, the default of git and GNU diff; fast when the inputs are similar.›Delta encodingcompression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›BM25 rankingsearch-sort · low: The standard lexical relevance score: IDF-weighted term frequency with saturation (k1) and length normalization (b); default in Lucene, Elasticsearch and SQLite FTS5.›Dictionary compressioncompression · low: Prime the compressor with shared, pre-trained content so small messages can reference it; often 2-5x better ratio on sub-KB records.›Hdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesProcess supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Unicode text tokenization for searchsearch-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.›Html sanitizationsecurity · low: Allowlist-based HTML sanitizing of untrusted markupMutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Optimistic concurrency controlstorage · low: Run transactions without locks, record what they read, and validate at commit that nothing changed; fastest when conflicts are rare.›Graceful drainingsystems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›Stale while revalidatesystems · low: Serving cached data past TTL while refreshing in the background, with request coalescingW-TinyLFUsystems · low: An LRU window plus segmented LRU, guarded by a 4-bit Count-Min sketch that admits a newcomer only if it beats the victim's frequency.›HTTP caching semanticsnetworking · low: The rules (RFC 9111) that decide whether a stored response is still fresh, and how a cheap conditional request revalidates it when it is not.›Cooperative schedulingconcurrency · low: Green threads/coroutines yielding at await points on a user-space schedulerRecord replaydistributed · low: Record every nondeterministic input (time, randomness, I/O, model and tool outputs, per-tick player input) and re-execute deterministically for debugging, replays and state rebuildPRNGs (PCG, xoshiro)numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›+ 10 more techniques+ 49 more primitives

Similar systems

Share the most building blocks with this one.

code-review 43 %translation-management 40 %mail-sync 38 %forum-qa 38 %merge-queue 38 %