← Back
Home › Systems › CI/CD and developer tooling

continuous-benchmarking

benchmarks with regression detection

Bencher, codspeed

SystemTechniques · 4Primitives · 13 continuous-benchmarking Benchmarking methodologyhardware-os · medium: Warmup, CPU pinning, frequency control, iteration counts and outlier handling for stable benchmarksChange point detectionstatistics · medium: CUSUM/E-divisive/PELT to find shifts in a time series mean or distributionHypothesis testingstatistics · medium: Pick a test statistic, compare it to its distribution under "no effect", read off a p-value, and control error rates across one test or many.›Time-series anomaly detectionstatistics · medium: Predict the expected value from a seasonal baseline, scale the residual by a robust spread (MAD), threshold at a chosen false-alarm rate, then merge flags into incidents.›Hardware performance countershardware-os · low: PMU counters via perf_event: instructions, cycles, cache missesBootstrap confidence intervalsstatistics · low: Resampling-based confidence intervals for arbitrary statisticsQuantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsFailure bisectionsearch-sort · low: Binary search over commits, batches or inputs to isolate a failureHdr histogramdata-structures · low: Log-linear bucketed histograms with bounded relative error for latency percentilesCpu isolation and pinninghardware-os · low: Isolcpus/nohz_full, thread affinity and IRQ steering for jitter-free hot threadsPower management dvfshardware-os · low: DVFS governors, P- and C-states, race-to-idle, thermal throttlingRobust statisticsstatistics · low: Location and scale estimates (median, MAD, trimmed mean, Huber M-estimator) that stay correct when a minority of the data is garbage.›Bayesian rate estimationstatistics · low: Estimate a conversion or event rate from few observations with a Beta-Binomial or Gamma-Poisson update: two counters, an honest interval, and P(B beats A) directly.›Wilson score intervalstatistics · low: Wilson lower bound for ranking by up/down votes and proportions with few samplesIsolation forestmachine-learning · low: Score each record by how few random axis-aligned splits it takes to isolate it in a small forest of random trees; anomalies are few and different, so they isolate fast.›Exponential smoothing (ETS, Holt-Wi…statistics · low: Forecast by keeping a few running states (level, trend, seasonal) updated by a fraction of each forecast error; O(1) memory per series and a strong baseline.›STL decomposition (seasonal-trend w…statistics · low: Split a series into trend, seasonal and remainder with iterated loess smoothing; the seasonal shape may drift and robust weights keep outliers out of it.›

Similar systems

Share the most building blocks with this one.

canary-analysis 33 %model-monitoring 18 %metric-anomaly-detection 18 %