← Back
Home › Systems › Communication and media

spam-filter

spam filtering

Rspamd, Gmail

SystemTechniques · 5Primitives · 31 spam-filter Gradient-boosted decision trees (GB…machine-learning · medium: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.›Near-duplicate detection with MinHa…machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Linear-time regular expression matc…search-sort · medium: Match a regular expression in O(pattern x text) worst case by tracking all live parse positions at once, then cache the state sets as a lazily built DFA for speed.›Reputation scoringsecurity · medium: Turn per-entity good/bad event counts into a trust score that forgets old evidence, borrows strength from parent groups for new keys, and maps to allow, challenge or block.›Unicode confusablessecurity · medium: UTS #39 maps every string to a "skeleton" so look-alikes collide, and restricts script mixing so most spoofs never get registered.›Aho-Corasick multi-pattern matchingsearch-sort · low: Find all occurrences of many fixed patterns in one pass over the text, in time independent of the pattern count, using a trie with failure links.›Sliding-window counterssystems · low: Count events in the last W seconds with a ring of sub-bucket counters; the two-bucket weighted estimate gives rate limits in 16 bytes per key.›Unicode text tokenization for searchsearch-sort · low: Turn multilingual text into index terms: segment words (UAX 29 style, CJK bigrams), normalize, case-fold and accent-fold, identically at index and query time.›Bloom filterdata-structures · low: Bit array answering "definitely not present" or "probably present" in about 10 bits per key for 1 % false positives; inserts are cheap, deletes impossible.›Count-min sketchdata-structures · low: d rows of w counters estimate any item's frequency in a stream, overestimating by at most ε·N with probability 1-δ, in fixed tiny memory.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Feature hashing (the hashing trick)machine-learning · low: Hash each feature name to an index in a fixed-size vector, with a second hash for the sign, so no vocabulary is needed and memory is fixed.›Linear and logistic regressionmachine-learning · low: Fit a weighted sum of features to a number (least squares) or a probability (logistic); the baseline every model must beat.›Naive Bayes classifiermachine-learning · low: Count how often each feature occurs per class and score a new item by summed log-likelihoods; trains in one pass, updates and merges by adding counts.›SPF, DKIM and DMARC sender authenti…networking · low: SPF authorizes sending IPs, DKIM signs the message, DMARC requires either to match the visible From domain and says what to do on failure.›Dns blocklistsnetworking · low: DNSBL/URIBL reputation lookups via reversed-IP DNS queriesDNS message format and client looku…networking · low: Binary query/response format of DNS (RFC 1035) and how a stub client uses it: record types, name compression, UDP/TCP/EDNS sizes, TTL caching.›Mime parsingstorage · low: RFC 5322/2045 headers, multipart bodies and transfer encodingsQuantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsLoss functionsmachine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›Regularizationmachine-learning · low: Techniques that trade training fit for generalization: weight decay, dropout, early stopping, label smoothing and data mixing, with the exact update rules and defaults.›+ 15 more primitives

Similar systems

Share the most building blocks with this one.

abuse-classifier 23 %email-auth 22 %