← Back
Home › Systems › AI applications

abuse-classifier

abuse and spam classifier

YouTube comments

SystemTechniques · 11Primitives · 44 abuse-classifier Windowed stream aggregationdistributed · medium: Group events by event-time window per key, keep one partial aggregate per slice, and emit a window when the watermark passes its end.›Active learningmachine-learning · medium: Choosing which unlabeled examples to label next: uncertainty, margin, diversity and committee samplingGradient-boosted decision trees (GB…machine-learning · medium: Add small regression trees one at a time, each fitted to the loss gradient; histogram splits over 256 bins make it the strongest tabular default.›Near-duplicate detection with MinHa…machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Text classificationmachine-learning · medium: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›Dynamic batchingml-training · medium: Queue independent requests and dispatch them as one batch when it is full or the oldest has waited max_delay; bounded extra latency buys throughput.›Knowledge distillationml-training · medium: Train a small student to match a large teacher's softened outputs (plus the hard labels) so it keeps most of the teacher's accuracy at a fraction of the cost.›Calibrationmachine-learning · low: Make predicted probabilities match observed frequencies: measure with ECE, fix post hoc with temperature, Platt or isotonic scaling on held-out data.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Byte-pair encoding (BPE) tokenizati…machine-learning · low: Learn a subword vocabulary by repeatedly merging the most frequent adjacent pair; byte-level BPE encodes any input losslessly with 32k-200k tokens.›Transformer blockml-training · low: The decoder-only pre-norm block (attention plus MLP on a residual stream) behind every modern LLM, with exact shapes, parameter, FLOP and KV-cache counts.›Text normalizationmachine-learning · low: Rewrite messy text into one canonical form for matchers and models, or into spoken words for speech output (and back), keeping a map to the original offsets.›Count-min sketchdata-structures · low: d rows of w counters estimate any item's frequency in a stream, overestimating by at most ε·N with probability 1-δ, in fixed tiny memory.›Evaluation metricsmachine-learning · low: Precision, recall, ROC AUC, average precision, log loss, NDCG and pass@k computed correctly, plus paired confidence intervals for comparing models.›Aho-Corasick multi-pattern matchingsearch-sort · low: Find all occurrences of many fixed patterns in one pass over the text, in time independent of the pattern count, using a trie with failure links.›Perceptual hashingsearch-sort · low: Compact fingerprints (64 to 256 bits) of an image that stay close under resizing and recompression, compared by Hamming distance; pHash, dHash and Meta's PDQ.›SimHashsearch-sort · low: Charikar's random-hyperplane sketch: a 64-bit fingerprint per document whose Hamming distance tracks cosine similarity, searchable in sub-linear time by exact block matches.›Label aggregationstatistics · low: Combining noisy labels from several annotators: majority vote, Dawid-Skene EM, annotator weightingHash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskHierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›k-means clusteringmachine-learning · low: Partition points into k clusters by alternating nearest-centroid assignment and centroid averaging; k-means++ seeding makes it dependable.›Quantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsLoss functionsmachine-learning · low: The scalar a model minimizes: cross-entropy for classes and tokens, MSE or Huber for values, each with a stable form and simple gradient.›+ 4 more techniques+ 28 more primitives

Similar systems

Share the most building blocks with this one.

prompt-injection-detection 48 %moderation 33 %safety-classifiers 27 %fraud-detection 24 %machine-translation 24 %