← Back
Home › Systems › Networking and edge

flow-monitoring

network flow monitoring

sFlow/IPFIX collectors

SystemTechniques · 4Primitives · 21 flow-monitoring Time-series anomaly detectionstatistics · medium: Predict the expected value from a seasonal baseline, scale the residual by a robust spread (MAD), threshold at a chosen false-alarm rate, then merge flags into incidents.›Columnar storagestorage · medium: Store each column contiguously so analytic scans read only the columns they need, compress well and run as tight SIMD loops.›Rollup aggregationstorage · medium: Fold raw events into one mergeable partial aggregate per time bucket and dimension key, then merge fine buckets into coarser ones, so long ranges read few rows and stay exact.›Vectorized executionstorage · medium: Query operators pass batches of about 1-2K values per column instead of single tuples, amortizing interpretation overhead and enabling tight SIMD loops.›Count-min sketchdata-structures · low: d rows of w counters estimate any item's frequency in a stream, overestimating by at most ε·N with probability 1-δ, in fixed tiny memory.›HyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›Ebpf xdpnetworking · low: EBPF programs at the XDP/tc hooks for in-kernel packet processing at line rateIpfix flow exportnetworking · low: IPFIX/NetFlow template-based flow record export and collectionLongest prefix matchnetworking · low: IP prefix lookup structures (DIR-24-8, Poptrie, LC-trie) for routes, ACLs and geo-IPPacket samplingnetworking · low: 1-in-N and random packet sampling with unbiased traffic estimationTop-k selectionsearch-sort · low: Keep the k best of n items without sorting all n - bounded min-heap for streams, quickselect for arrays, threshold pruning to skip most items.›Isolation forestmachine-learning · low: Score each record by how few random axis-aligned splits it takes to isolate it in a small forest of random trees; anomalies are few and different, so they isolate fast.›Exponential smoothing (ETS, Holt-Wi…statistics · low: Forecast by keeping a few running states (level, trend, seasonal) updated by a fraction of each forecast error; O(1) memory per series and a strong baseline.›Robust statisticsstatistics · low: Location and scale estimates (median, MAD, trimmed mean, Huber M-estimator) that stay correct when a minority of the data is garbage.›STL decomposition (seasonal-trend w…statistics · low: Split a series into trend, seasonal and remainder with iterated loess smoothing; the seasonal shape may drift and robust weights keep outliers out of it.›Bit packing and PFORcompression · low: Store blocks of integers with exactly b bits each and patch rare outliers separately (PFOR); the fastest integer codec family, billions of values per second.›Delta encodingcompression · low: Store differences between consecutive values instead of the values, turning sorted IDs, timestamps and counters into small numbers that pack into few bits.›Dictionary encodingcompression · low: Map values to small integer codes via a dictionary for compact, comparable columnsZone mapsstorage · low: Per-block min/max (and sparse primary) indexes to skip data during scansHash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to disk+ 5 more primitives

Similar systems

Share the most building blocks with this one.

web-analytics 27 %ddos-waf 24 %rum 22 %data-quality 20 %realtime-olap 17 %