← Back
Home › Systems › Security and identity

url-reputation

URL and domain reputation at web scale

Google Safe Browsing, Microsoft SmartScreen, Netcraft

SystemTechniques · 22Primitives · 79 url-reputation CDN and edgecloud · medium: CDNs cache content in hundreds of PoPs near users, cutting latency and egress bills; edge compute runs small code there. Bandwidth price spreads 10x between providers.›Deterministic simulation testingdistributed · medium: Run the whole system in one thread under a seeded scheduler with simulated network, disk and clock, so every failure replays from its seed.›Primary-backup replicationdistributed · medium: One node orders all writes and ships them to standbys; the acknowledgement rule (none, one, all) sets what a crash loses, and an external arbiter decides who takes over.›Html parsinggraphics · medium: WHATWG HTML tokenizer state machine and tree construction with error recoveryGradient-boosted decision trees (GB…machine-learning · medium: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.›Text classificationmachine-learning · medium: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›Url frontiersearch-sort · medium: Prioritised per-host queues with politeness delays for crawlingURL normalizationsearch-sort · low: Rewrite a URL to one canonical key, in safe RFC 3986 steps plus lossy, site-dependent ones (tracking parameters, query order, fragments), and keep the original beside the key.›Cryptographic hashes (SHA-2, BLAKE3)security · low: Collision-resistant fingerprints of data; SHA-256 for interoperability, BLAKE3 for speed, HMAC/HKDF for keyed uses.›Browser automation protocolagentic-ai · low: CDP / WebDriver BiDi commands and events over websockets to drive browsersDelta encodingcompression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›Hdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesProcess supervisionhardware-os · low: A parent process starts services, reaps their exits and restarts them under a delay and burst-limit policy, then stops them with SIGTERM before SIGKILL.›Dns blocklistsnetworking · low: DNSBL/URIBL reputation lookups via reversed-IP DNS queriesRdap whoisnetworking · low: Look up domain and IP registration data via RDAP (JSON over HTTP, IANA bootstrap) or WHOIS text protocol on port 43Fuzzy string matchingsearch-sort · low: Edit distance, Jaro-Winkler and trigram indexes for approximate name and address matchingPerceptual hashingsearch-sort · low: Compact fingerprints (64 to 256 bits) of an image that stay close under resizing and recompression, compared by Hamming distance; pHash, dHash and Meta's PDQ.›K anonymity hash prefixsecurity · low: Querying a breach/blocklist by sending only a short hash prefix and matching suffixes locallyMutual TLSsecurity · low: TLS where the client also proves a certificate key, so every connection carries a verified, rotatable machine identity.›Annotator agreementstatistics · low: Inter-annotator agreement measures: Cohen and Fleiss kappa, Krippendorff alpha, gold-question accuracyGraceful drainingsystems · low: On a termination signal, stop taking new work, finish in-flight work until a deadline, tell clients to reconnect elsewhere, then exit; clients resume from a token.›HTTP caching semanticsnetworking · low: The rules (RFC 9111) that decide whether a stored response is still fresh, and how a cheap conditional request revalidates it when it is not.›Cooperative schedulingconcurrency · low: Green threads/coroutines yielding at await points on a user-space scheduler+ 15 more techniques+ 63 more primitives

Similar systems

Share the most building blocks with this one.

email-auth 33 %cert-transparency 31 %secure-updates 30 %mcp-supply-chain 30 %package-distro 29 %