← Back
Home › Systems › CI/CD and developer tooling

canary-analysis

automated canary analysis

Kayenta

SystemTechniques · 6Primitives · 15 canary-analysis Hypothesis testingstatistics · medium: Pick a test statistic, compare it to its distribution under "no effect", read off a p-value, and control error rates across one test or many.›Sequential probability ratio teststatistics · medium: Add up the log-likelihood ratio per observation and stop when it crosses a fixed bound; the least data of any test with the same error rates (for two point hypotheses).›Time-series anomaly detectionstatistics · medium: Predict the expected value from a seasonal baseline, scale the residual by a robust spread (MAD), threshold at a chosen false-alarm rate, then merge flags into incidents.›Staged rolloutsystems · medium: Rolling, blue-green and canary rollout with health gates and automatic rollbackBurn rate alertingsystems · medium: Multi-window multi-burn-rate alerts on SLO error-budget consumptionRollup aggregationstorage · medium: Fold raw events into one mergeable partial aggregate per time bucket and dimension key, then merge fine buckets into coarser ones, so long ranges read few rows and stay exact.›Bootstrap confidence intervalsstatistics · low: Resampling-based confidence intervals for arbitrary statisticsExponential smoothing (ETS, Holt-Wi…statistics · low: Forecast by keeping a few running states (level, trend, seasonal) updated by a fraction of each forecast error; O(1) memory per series and a strong baseline.›Traffic splittingsystems · low: Weighted routing between versions for canaries and A/B, sticky by hashQuantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsWilson score intervalstatistics · low: Wilson lower bound for ranking by up/down votes and proportions with few samplesLoglikelihood scoringmachine-learning · low: Scoring choices by summed token log-probabilities, normalization by length, perplexity computationBayesian rate estimationstatistics · low: Estimate a conversion or event rate from few observations with a Beta-Binomial or Gamma-Poisson update: two counters, an honest interval, and P(B beats A) directly.›Isolation forestmachine-learning · low: Score each record by how few random axis-aligned splits it takes to isolate it in a small forest of random trees; anomalies are few and different, so they isolate fast.›Robust statisticsstatistics · low: Location and scale estimates (median, MAD, trimmed mean, Huber M-estimator) that stay correct when a minority of the data is garbage.›STL decomposition (seasonal-trend w…statistics · low: Split a series into trend, seasonal and remainder with iterated loess smoothing; the seasonal shape may drift and robust weights keep outliers out of it.›Deterministic bucketingsystems · low: Hash a stable unit id with a per-purpose salt into a bucket; "in" means bucket below the percentage, so assignment is stateless, repeatable and only grows on ramp-up.›Health checkingsystems · low: Decide per backend whether to route to it using probes and real-traffic errors, with consecutive-count hysteresis, capped ejection and a panic fallback.›Sliding-window counterssystems · low: Count events in the last W seconds with a ring of sub-bucket counters; the two-bucket weighted estimate gives rate limits in 16 bytes per key.›Hash aggregationstorage · low: GROUP BY via hash tables with partitioning and spill to diskTime partitioningstorage · low: Cut a table into fixed-interval time blocks so queries read only overlapping blocks and retention drops a whole block as one metadata change.›

Similar systems

Share the most building blocks with this one.

continuous-benchmarking 33 %continuous-delivery 27 %metric-anomaly-detection 18 %experimentation 17 %