← Back
Home › Systems › AI applications

metric-anomaly-detection

anomaly detection on metrics

Datadog Watchdog

SystemTechniques · 8Primitives · 31 metric-anomaly-detection Windowed stream aggregationdistributed · medium: Group events by event-time window per key, keep one partial aggregate per slice, and emit a window when the watermark passes its end.›Arimastatistics · medium: ARIMA/SARIMA forecasting: differencing, AR and MA terms, order selection, maximum-likelihood fitChange point detectionstatistics · medium: CUSUM/E-divisive/PELT to find shifts in a time series mean or distributionHypothesis testingstatistics · medium: Pick a test statistic, compare it to its distribution under "no effect", read off a p-value, and control error rates across one test or many.›Columnar storagestorage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Watermarksdistributed · medium: Event-time progress markers that bound lateness and trigger window resultsLSM-treestorage · medium: Buffer writes in memory, flush them as immutable sorted files and merge in the background; turns random writes into sequential I/O.›Robust statisticsstatistics · low: Location and scale estimates (median, MAD, trimmed mean, Huber M-estimator) that stay correct when a minority of the data is garbage.›Gorilla XOR compressioncompression · low: Time-series codec: delta-of-delta timestamps plus XOR of consecutive float bit patterns, storing only changed bits; about 1.4 bytes per point.›Count-min sketchdata-structures · low: d rows of w counters estimate any item's frequency in a stream, overestimating by at most ε·N with probability 1-δ, in fixed tiny memory.›Quantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsIsolation forestmachine-learning · low: Score each record by how few random axis-aligned splits it takes to isolate it in a small forest of random trees; anomalies are few and different, so they isolate fast.›Exponential smoothing (ETS, Holt-Wi…statistics · low: Forecast by keeping a few running states (level, trend, seasonal) updated by a fraction of each forecast error; O(1) memory per series and a strong baseline.›STL decomposition (seasonal-trend w…statistics · low: Split a series into trend, seasonal and remainder with iterated loess smoothing; the seasonal shape may drift and robust weights keep outliers out of it.›Hash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskHierarchical timing wheelsystems · low: Timers bucketed by expiry into wheels of slots, coarser wheels for far deadlines: O(1) insert and cancel for millions of timeouts.›Finite differencesnumerics · low: Replace derivatives with weighted differences of grid values; simple stencils plus a stability condition (CFL) turn PDEs like heat, advection and waves into array loops.›L-BFGSnumerics · low: Quasi-Newton minimizer that approximates the inverse Hessian from the last m gradient differences via the two-loop recursion; O(m·n) memory and work per step.›Kalman filterrobotics · low: Optimal recursive estimator for a linear system with Gaussian noise: fuses a motion model with noisy measurements into a state estimate plus its covariance.›Bayesian rate estimationstatistics · low: Estimate a conversion or event rate from few observations with a Beta-Binomial or Gamma-Poisson update: two counters, an honest interval, and P(B beats A) directly.›Bootstrap confidence intervalsstatistics · low: Resampling-based confidence intervals for arbitrary statisticsWilson score intervalstatistics · low: Wilson lower bound for ranking by up/down votes and proportions with few samplesBit packing and PFORcompression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›+ 1 more techniques+ 15 more primitives

Similar systems

Share the most building blocks with this one.

model-monitoring 33 %demand-forecasting 19 %flaky-test-detection 18 %canary-analysis 18 %continuous-benchmarking 18 %