← Back
Home › Systems › Data processing and pipelines

data-quality

data validation and quality checks

Great Expectations

SystemTechniques · 3Primitives · 14 data-quality Distribution drift detectionmachine-learning · medium: PSI, KS and other tests for feature and score drift between training and productionTime-series anomaly detectionstatistics · medium: Predict the expected value from a seasonal baseline, scale the residual by a robust spread (MAD), threshold at a chosen false-alarm rate, then merge flags into incidents.›Hypothesis testingstatistics · medium: Pick a test statistic, compare it to its distribution under "no effect", read off a p-value, and control error rates across one test or many.›Quantile sketchesdata-structures · low: Mergeable quantile sketches: t-digest, DDSketch, HDR histogramsCount-min sketchdata-structures · low: d rows of w counters estimate any item's frequency in a stream, overestimating by at most ε·N with probability 1-δ, in fixed tiny memory.›HyperLogLogdata-structures · low: Counts distinct elements in kilobytes with about 1 % error by tracking the longest run of leading zero bits in hashed values per bucket.›Merkle treedata-structures · low: Hash tree for verifying and diffing large datasets with logarithmic proofsReservoir samplingdata-structures · low: Uniform k-sample from a stream of unknown length (Algorithm R/L)JSON Schema validationsecurity · low: Compile a JSON Schema once, then check every untrusted document against it, within hard size and depth limits, and report failures as JSON Pointers.›Isolation forestmachine-learning · low: Score each record by how few random axis-aligned splits it takes to isolate it in a small forest of random trees; anomalies are few and different, so they isolate fast.›Exponential smoothing (ETS, Holt-Wi…statistics · low: Forecast by keeping a few running states (level, trend, seasonal) updated by a fraction of each forecast error; O(1) memory per series and a strong baseline.›Robust statisticsstatistics · low: Location and scale estimates (median, MAD, trimmed mean, Huber M-estimator) that stay correct when a minority of the data is garbage.›STL decomposition (seasonal-trend w…statistics · low: Split a series into trend, seasonal and remainder with iterated loess smoothing; the seasonal shape may drift and robust weights keep outliers out of it.›Bootstrap confidence intervalsstatistics · low: Resampling-based confidence intervals for arbitrary statisticsWilson score intervalstatistics · low: Wilson lower bound for ranking by up/down votes and proportions with few samplesd-ary heapdata-structures · low: Array heap with d children per node (d = 4 typical): half the binary heap's depth, siblings in one cache line, faster priority queues.›Fast hash functionsdata-structures · low: Non-cryptographic hashes built from 64×64→128-bit multiplies and xor-shifts; pick by key size, trust level and whether the hash is persisted.›

Similar systems

Share the most building blocks with this one.

model-monitoring 25 %flow-monitoring 20 %