← Back
Home › Systems › Frontier AI research

misuse-monitoring

account-level misuse detection at API scale

usage-policy enforcement

SystemTechniques · 27Primitives · 70 misuse-monitoring Partitioned commit logdistributed · medium: A topic split into replicated append-only partitions: key hash keeps per-key order, offsets give position, and a high watermark limits readers to fully replicated records.›Windowed stream aggregationdistributed · medium: Group events by event-time window per key, keep one partial aggregate per slice, and emit a window when the watermark passes its end.›Graph community detectiongraphs · medium: Louvain/label propagation community detection for rings, fraud clusters and campaignsDensity-based clustering (DBSCAN)machine-learning · medium: Group points that are packed densely, label the rest as noise: no k, arbitrary shapes, two parameters (eps, minPts), cost set by the neighbour index.›LLM-as-judgemachine-learning · medium: Grade open-ended outputs with a language model under a rubric, then correct its known biases by order swapping, voting and calibration against human labels.›Near-duplicate detection with MinHa…machine-learning · medium: Estimate set Jaccard similarity from k min-hashes and find similar pairs with banded LSH buckets; the standard dedup for web crawls, training corpora and sequence collections.›Text classificationmachine-learning · medium: Assign one or more labels to a text; climb a ladder from regexes to linear models on n-grams, embeddings, fine-tuned encoders and LLM prompts, only as far as the data and budget demand.›K anonymitysecurity · low: Minimum group-size thresholds and suppression so aggregates cannot single out a personSliding-window counterssystems · low: Count events in the last W seconds with a ring of sub-bucket counters; the two-bucket weighted estimate gives rate limits in 16 bytes per key.›Embeddingsmachine-learning · low: A learned table mapping discrete ids to dense vectors: gather forward, sparse scatter-add backward, row-wise optimizers to keep huge tables cheap.›Isolation forestmachine-learning · low: Score each record by how few random axis-aligned splits it takes to isolate it in a small forest of random trees; anomalies are few and different, so they isolate fast.›Fast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›Consumer offsetsdistributed · low: Each consumer group stores one integer per partition, the next position to read, so a restarted consumer resumes where it left off with at-least-once delivery.›Segmented logdistributed · low: Split an append-only log into files named by their first offset, write only the newest, and find any record by binary search plus a sparse index.›Zero-copy I/Onetworking · low: Move bytes between files, sockets and NICs by passing page references instead of copying through user space: sendfile, splice, MSG_ZEROCOPY, kTLS.›Hash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskHierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Union-findgraphs · low: Disjoint-set forest with union by size and path compression; merges sets and answers same-set queries in effectively constant time.›PRNGs (PCG, xoshiro)numerics · low: Small, fast, statistically strong non-cryptographic generators: PCG32 and xoshiro256**, with seeding, parallel streams and unbiased ranges.›Sparse matrices (CSR)numerics · low: Compressed sparse row stores only nonzeros as row pointers, column indices and values; SpMV streams them once, so it is memory-bandwidth bound.›k-d treegeometry · low: A binary tree that splits k-dimensional points by one coordinate per level, giving fast nearest-neighbour and box queries on static low-dimensional data.›Token samplingml-training · low: Turn a vector of scores into one chosen item with temperature, top-k, top-p, min-p, repetition penalties and a seed.›Annotator agreementstatistics · low: Inter-annotator agreement measures: Cohen and Fleiss kappa, Krippendorff alpha, gold-question accuracy+ 20 more techniques+ 54 more primitives

Similar systems

Share the most building blocks with this one.

aml-monitoring 21 %reviews 17 %fraud-detection 16 %moderation 16 %