← Back
Home › Systems › Sales, marketing and customer service

seo-platform

site crawling, backlink index and rank tracking

Ahrefs, Semrush

SystemTechniques · 28Primitives · 90 seo-platform Async io runtimeconcurrency · medium: Futures/tasks executor over a reactor (epoll/io_uring) with wakers and timers (listed in Moved to the wiki)Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Mapreducedistributed · medium: Map, partition, sort/shuffle, reduce with combiners and re-execution of failed tasksPrimary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Html parsinggraphics · medium: WHATWG HTML tokenizer state machine and tree construction with error recoveryPagerankgraphs · medium: Power-iteration PageRank on a sparse graph with damping and dangling nodesHTML main-text extractionmachine-learning · medium: Decode bytes, tolerate broken markup, drop non-content nodes and score text blocks by length and link density to keep the main run, with headings, metadata and offsets for citation.›Bloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›Robots exclusion protocolsearch-sort · low: Decide per URL whether a crawler may fetch it from a host's robots.txt (RFC 9309): status-code rules, group choice, longest-match Allow/Disallow, 24 h cache.›URL normalizationsearch-sort · low: Rewrite a URL to one canonical key, in safe RFC 3986 steps plus lossy, site-dependent ones (tracking parameters, query order, fragments), and keep the original beside the key.›Public Suffix Listnetworking · low: Find the registrable domain (eTLD+1) of a host name by matching the longest rule in a community-maintained list of shared suffixes.›Range partitioningdistributed · low: Cut the sorted key space into contiguous half-open ranges kept in a boundary map; split hot or big ranges, merge small ones, move ranges between nodes.›Browser automation protocolagentic-ai · low: CDP / WebDriver BiDi commands and events over websockets to drive browsersHdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesHyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›Process supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›SimHashsearch-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.›Mutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Connection poolingsystems · low: Reuse and multiplex backend connections with bounded pools, checkout queues and health evictionGraceful drainingsystems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›epoll event loopnetworking · low: One thread waits on thousands of non-blocking sockets with epoll_wait and handles only ready ones; the core of nginx, Redis and tokio.›io_uringnetworking · low: Linux async I/O through two shared-memory rings (submissions in, completions out), batching or eliminating syscalls for sockets and files alike.›Hierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›+ 21 more techniques+ 74 more primitives

Similar systems

Share the most building blocks with this one.

web-crawler 38 %sales-intelligence 30 %job-board 28 %url-reputation 27 %forward-proxy 26 %