A/B testing (online controlled experiments)

Deep dive

How it works

Parts:

• unit — what is randomized: user, device, session, or a cluster (city, team, graph community) when units interfere
• assigner — hash(salt, unit) mod buckets; one salt per experiment keeps experiments independent (layering); bucket ranges set the split
• exposure log — one record when the unit first sees the change (not at assignment)
• metrics — primary (the decision), guardrail (must not regress: latency, errors, churn), sanity (arm sizes, trigger rate)
• checks — sample ratio mismatch (SRM), A/A runs, pre-period balance
• analyzer — difference of means, Welch SE, 95 % CI; delta method for ratio metrics

Steps:

1. Fix primary metric, guardrails, unit, minimum detectable effect mde and duration (whole weeks) before launch.
2. Size: n = 2 (z_a + z_b)^2 sigma^2 / mde^2 per arm, z_a = 1.96, z_b = 0.84, sigma from history.
3. Assign by hash; log exposures; keep salt and bucket ranges fixed during the test.
4. Daily: chi-square test of exposed counts per arm against the designed split; p < 0.001 invalidates the results (Kohavi et al.; Fabijan et al. 2019 diagnose causes).
5. At the planned end compute d = mean_T - mean_C, se = sqrt(var_C/n_C + var_T/n_T), CI d +- 1.96 se.
6. Ratio metrics (clicks per session): per-user sums, then the delta method Var(R) ~ (s_X^2 - 2 R s_XY + R^2 s_Y^2) / (n mu_Y^2), R = mean(X)/mean(Y).
7. Ship if the primary CI excludes 0 in the good direction and no guardrail CI crosses its tolerance.

 unit id ──hash(salt,id) mod 1000──► bucket ─► arm A [0,500) | arm B [500,1000)
                                                  │              │
                          exposure log ◄──────────┴──────────────┘
                                │
          SRM chi2 ◄── counts   │──► per-unit metrics ──► diff, Welch SE, CI
          (stop if p<0.001)                      guardrails ─┘ (+ CUPED, delta method)

Variance reduction. A pre-experiment covariate shrinks variance by 1 - rho^2: see CUPED (Deng et al. 2013), not re-derived here. Stratified assignment does the same on the design side.

Peeking. The CI is valid at one planned look. Stopping at the first p < 0.05 inflates false positives: in a null simulation (4000 runs, 10,000 units) with 1, 5 and 10 equally spaced looks, "significant" results occurred in 5.5 %, 14.5 % and 18.5 %. Fixes: one look (fixed horizon), group-sequential boundaries at planned looks, or always-valid tests such as mSPRT (SPRT, sequential testing).

Other traps.

• Multiple testing: one declared primary metric; correct the rest (Bonferroni or Benjamini-Hochberg).
• Novelty and primacy: run one to two weeks at least and plot the effect by day.
• Interference: a treated seller changes what control buyers see; randomize by cluster (city, time slice, graph community) and analyze at cluster level, accepting fewer effective units.
• Twyman's law: a surprisingly large effect is usually a bug (logging, filter, SRM); investigate before celebrating.

Code

Salted SplitMix64 assignment, fixed-seed simulation with a known lift of 0.1, Welch CI, SRM check, timing loop; compiled and run in release mode.

use std::time::Instant;

fn splitmix64(mut x: u64) -> u64 {
    x = x.wrapping_add(0x9E37_79B9_7F4A_7C15);
    x = (x ^ (x >> 30)).wrapping_mul(0xBF58_476D_1CE4_E5B9);
    x = (x ^ (x >> 27)).wrapping_mul(0x94D0_49BB_1331_11EB);
    x ^ (x >> 31)
}
const BUCKETS: u64 = 1000;
/// Arm 0 = control owns buckets [0, control_buckets); the rest is treatment.
fn assign(salt: u64, unit: u64, control_buckets: u64) -> usize {
    let bucket = splitmix64(salt ^ splitmix64(unit)) % BUCKETS;
    (bucket >= control_buckets) as usize
}
struct Rng(u64); // xorshift64
impl Rng {
    fn next(&mut self) -> u64 { let mut x = self.0; x ^= x << 13; x ^= x >> 7; x ^= x << 17; self.0 = x; x }
    fn unif(&mut self) -> f64 { ((self.next() >> 11) as f64 + 0.5) / (1u64 << 53) as f64 }
    fn normal(&mut self) -> f64 { // Box-Muller
        (-2.0 * self.unif().ln()).sqrt() * (6.283_185_307_179_586 * self.unif()).cos()
    }
}
#[derive(Default, Clone, Copy)]
struct Arm { n: f64, sum: f64, sumsq: f64 }
impl Arm {
    fn add(&mut self, y: f64) { self.n += 1.0; self.sum += y; self.sumsq += y * y; }
    fn mean(&self) -> f64 { self.sum / self.n }
    fn var(&self) -> f64 { (self.sumsq - self.sum * self.sum / self.n) / (self.n - 1.0) }
}
/// Difference of means and 95 % CI with the Welch standard error.
fn welch_ci(c: &Arm, t: &Arm) -> (f64, f64, f64) {
    let d = t.mean() - c.mean();
    let se = (c.var() / c.n + t.var() / t.n).sqrt();
    (d, d - 1.959_964 * se, d + 1.959_964 * se)
}
/// Sample ratio mismatch: chi-square with 1 df against the designed split.
fn srm_chi2(obs: [f64; 2], p_control: f64) -> f64 {
    let n = obs[0] + obs[1];
    let e = [n * p_control, n * (1.0 - p_control)];
    (0..2).map(|i| (obs[i] - e[i]).powi(2) / e[i]).sum()
}
const CHI2_P001: f64 = 10.828; // 1 df, p = 0.001 (the SRM alarm level in Kohavi et al.)
/// Users per arm for 80 % power, two-sided alpha 0.05: 2 (z_a + z_b)^2 sigma^2 / mde^2.
fn n_per_arm(sigma: f64, mde: f64) -> f64 { 2.0 * (1.959_964 + 0.841_621f64).powi(2) * sigma * sigma / (mde * mde) }

fn main() {
    let (sigma, lift) = (3.0, 0.1);
    let n_arm = n_per_arm(sigma, lift).ceil() as u64; // 14,130
    let mut rng = Rng(0x2545_F491_4F6C_DD1D);
    let (mut covered, mut significant) = (0, 0);
    for exp in 0..1000u64 {
        let (mut arms, salt) = ([Arm::default(); 2], 0xC0FFEE + exp);
        for unit in 0..2 * n_arm {
            let a = assign(salt, unit, 500);
            assert_eq!(a, assign(salt, unit, 500)); // stable across calls
            arms[a].add(10.0 + sigma * rng.normal() + lift * a as f64);
        }
        let (_, lo, hi) = welch_ci(&arms[0], &arms[1]);
        covered += (lo <= lift && lift <= hi) as u32;
        significant += (lo > 0.0) as u32;
        if exp == 0 { assert!(srm_chi2([arms[0].n, arms[1].n], 0.5) < CHI2_P001); }
    }
    println!("coverage {:.3}, power {:.3}", covered as f64 / 1000.0, significant as f64 / 1000.0);
    assert!(covered >= 930);
    assert!((740..=860).contains(&significant)); // designed power 0.80
    // SRM: a clean 50/50 hash passes, a 52/48 split at n = 100,000 is flagged.
    let count = |cb: u64| { let mut o = [0.0; 2]; for u in 0..100_000u64 { o[assign(7, u, cb)] += 1.0; } o };
    let (ok, bad) = (srm_chi2(count(500), 0.5), srm_chi2(count(520), 0.5));
    println!("chi2 clean {ok:.2}, biased {bad:.1}");
    assert!(ok < CHI2_P001 && bad > CHI2_P001);
    let t = Instant::now();
    let mut acc = 0usize;
    for u in 0..20_000_000u64 { acc += assign(7, std::hint::black_box(u), 500); }
    println!("{:.2} ns per assignment ({acc})", t.elapsed().as_nanos() as f64 / 2e7);
}

Measured: CI coverage 0.945 over 1000 experiments of 14,130 units per arm, power 0.795 (designed 0.80), chi-square 0.35 for the hash split and 154.3 for 52/48 at n = 100,000, 1.70 ns per assignment (Apple silicon, release).

Complexity and constants

• Assignment: O(1), one 64-bit mix, no state, no network; analysis: one pass, 24 bytes of accumulators per arm and metric.
• Units scale with sigma^2 / mde^2: halving the detectable effect quadruples them; variance reduction that halves sigma cuts them by four.
• Cost drivers: exposure-log bytes per unit, the daily join of exposures to metric events, metric recompute per experiment.

Efficiency

• Cost: assignment 1.7 ns/unit (measured, one core); the experiment's real cost is the units spent: 14,130 per arm for sigma = 3, mde = 0.1; analysis is O(n) streaming.
• Levers:
  • variance reduction with a pre-period covariate — variance times 1 - rho^2 (Deng et al. 2013) — a correlated pre-period metric exists
  • analyze exposed (triggered) units only — removes dilution from units that never saw the change (Kohavi et al., from memory) — the feature shows to a small share
  • per-unit pre-aggregated sums, not events — one row per unit — always
  • sequential or group-sequential stopping — fewer expected units for large effects (SPRT) — early stopping has value
• Naive → standard → state of the art:
  • before/after comparison, daily p-value peeking · confounded, 18.5 % false positives at 10 looks (measured)
  • fixed-horizon two-sample test sized by power, SRM check · units = 16 sigma^2 / mde^2
  • CUPED or stratified design plus sequential monitoring · fewer units at a valid error rate

Implementation notes

• Hash unit id plus salt, never a counter or clock; reusing a salt across experiments correlates arms.
• Use bucket < cutoff so ramps (1 %, 10 %, 50 %) only add buckets and never reshuffle assigned units.
• Log exposure, not assignment, and analyze exposed units under the same trigger in both arms.
• The sum-of-squares variance above is fine at 1e5 units; use Welford or pairwise sums at 1e9.
• Store seed, salt and bucket ranges with the experiment record for audit.

Pitfalls

• SRM ignored (bots, redirects, crashes on one arm): every metric is then untrustworthy.
• Randomize by user but analyze events with a naive standard error: variance is understated.
• Changing the split mid-test without analyzing by period: Simpson's paradox.
• Extrapolating a two-week lift to the long run; keep a holdback for strategic changes.

Variants and alternatives

In production

• Large web companies run thousands of controlled experiments a year (Kohavi, Tang, Xu 2020, from memory).
• Open-source: GrowthBook, Spotify's confidence library.
• Pricing, ads and email platforms use the same loop with revenue and unsubscribe guardrails.
• Randomized clinical trials: registered primary endpoint, interim looks with spending boundaries.

Implementations

Sources

• R. Kohavi, D. Tang, Y. Xu, Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, Cambridge University Press, 2020. book
• A. Deng, Y. Xu, R. Kohavi, T. Walker, Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data, WSDM 2013. paper
• A. Deng, U. Knoblich, J. Lu, Applying the Delta Method in Metric Analytics, KDD 2018. paper
• A. Fabijan, J. Gupchup, S. Gupta, J. Omhover, W. Qin, L. Vermeer, Diagnosing Sample Ratio Mismatch in Online Controlled Experiments, KDD 2019. paper
• R. Johari, L. Pekelis, D. J. Walsh, Always Valid Inference: Bringing Sequential Analysis to A/B Testing, arXiv 2015. paper