Calibration

Deep dive

How it works

In: held-out scores or logits with true labels · Out: a monotone map from score to calibrated probability, plus ECE · State: the fitted map (T; a, b; or isotonic block boundaries) · Cost: O(n) per fit step; isotonic O(n log n)

Measure. Bin predictions by confidence (top-class probability for multi-class). Per bin, compare the mean confidence with the empirical accuracy. Plot both (reliability diagram) and report ECE. Also report a proper scoring rule (log loss or Brier), because ECE alone can be gamed: giving every example the overall accuracy as its confidence has ECE 0. Murphy's decomposition splits the Brier score into reliability (calibration error) − resolution + uncertainty.

Temperature scaling. p = softmax(z / T). Fit T > 0 by minimizing NLL on the calibration set. Optimize t = ln T (unconstrained) with Newton's method:

NLL(t) = mean( logsumexp(z·e^{−t}) − z_y·e^{−t} )
dNLL/dt  = mean( z_y·u − E_p[z]·u )                 with u = e^{−t}
d²NLL/dt² = mean( u²·Var_p[z] − (z_y·u − E_p[z]·u) )

Convex in 1/T, so it converges in a few iterations. T > 1 means the model was over-confident.

Platt scaling. Logistic regression on the 1-D score: p = σ(a·s + b). Platt suggested smoothed targets (N₊+1)/(N₊+2) and 1/(N₋+2) instead of 1 and 0 to avoid over-fitting the extremes.

Isotonic regression (PAV). Sort by score. Walk left to right, keep blocks with their mean label; whenever a block's mean is below the previous block's mean, merge them (weighted mean) and recheck backward. The result is the least-squares monotone fit. Predict by step lookup (or linear interpolation) on the block boundaries.

reliability diagram (over-confident model)
accuracy 1.0 |                 .    ideal: diagonal
             |            .  x
             |       .  x            x = model: below the diagonal at high confidence
             |   .  x
         0.0 +x--------------- confidence 1.0

Code

/// ECE with equal-width bins over the top-class confidence. conf in [0,1], correct = top class right.
pub fn ece(conf: &[f64], correct: &[bool], bins: usize) -> f64 {
    let (mut c, mut a, mut n) = (vec![0.0; bins], vec![0.0; bins], vec![0usize; bins]);
    for (&p, &ok) in conf.iter().zip(correct) {
        let b = ((p * bins as f64) as usize).min(bins - 1);     // p = 1.0 goes in the last bin
        c[b] += p; a[b] += ok as u8 as f64; n[b] += 1;
    }
    (0..bins).filter(|&b| n[b] > 0).map(|b| (n[b] as f64 / conf.len() as f64) * (a[b] / n[b] as f64 - c[b] / n[b] as f64).abs()).sum()
}

fn softmax_stats(z: &[f64], u: f64) -> (f64, f64, f64) {     // returns (logsumexp(u·z), E_p[z], Var_p[z])
    let m = z.iter().cloned().fold(f64::NEG_INFINITY, f64::max) * u;
    let w: Vec<f64> = z.iter().map(|&v| (v * u - m).exp()).collect();
    let s: f64 = w.iter().sum();
    let ez: f64 = z.iter().zip(&w).map(|(v, wi)| v * wi).sum::<f64>() / s;
    let ez2: f64 = z.iter().zip(&w).map(|(v, wi)| v * v * wi).sum::<f64>() / s;
    (m + s.ln(), ez, ez2 - ez * ez)
}

/// Fit temperature by Newton on t = ln T. logits: n×k row-major.
pub fn fit_temperature(logits: &[f64], y: &[usize], k: usize) -> f64 {
    let mut t = 0.0f64;
    for _ in 0..50 {
        let u = (-t).exp();
        let (mut g, mut h) = (0.0, 0.0);
        for (i, &yi) in y.iter().enumerate() {
            let z = &logits[i * k..(i + 1) * k];
            let (_, ez, var) = softmax_stats(z, u);
            let gi = (z[yi] - ez) * u;                            // dNLL_i/dt
            g += gi; h += u * u * var - gi;
        }
        let step = g / h.max(1e-12);
        t -= step.clamp(-1.0, 1.0);                               // damped Newton
        if step.abs() < 1e-10 { break; }
    }
    t.exp()
}

/// Platt scaling: fit p = sigmoid(a*s + b) by Newton with Platt's smoothed targets. Returns (a, b).
pub fn fit_platt(s: &[f64], y: &[bool]) -> (f64, f64) {
    let np = y.iter().filter(|&&v| v).count() as f64;
    let nn = y.len() as f64 - np;
    let (hi, lo) = ((np + 1.0) / (np + 2.0), 1.0 / (nn + 2.0));
    let (mut a, mut b) = (1.0, 0.0);
    for _ in 0..100 {
        let (mut ga, mut gb, mut haa, mut hab, mut hbb) = (0.0, 0.0, 1e-12, 0.0, 1e-12);
        for (&si, &yi) in s.iter().zip(y) {
            let p = 1.0 / (1.0 + (-(a * si + b)).exp());
            let (t, w) = (if yi { hi } else { lo }, p * (1.0 - p));
            ga += (p - t) * si; gb += p - t;
            haa += w * si * si; hab += w * si; hbb += w;
        }
        let det = haa * hbb - hab * hab;
        let (da, db) = ((hbb * ga - hab * gb) / det, (haa * gb - hab * ga) / det);
        a -= da; b -= db;
        if da.abs() + db.abs() < 1e-12 { break; }
    }
    (a, b)
}

/// Pool-adjacent-violators: returns block (upper score bound, value) pairs, non-decreasing values.
pub fn isotonic(s: &[f64], y: &[bool]) -> Vec<(f64, f64)> {
    let mut idx: Vec<usize> = (0..s.len()).collect();
    idx.sort_by(|&i, &j| s[i].partial_cmp(&s[j]).unwrap());
    let mut blocks: Vec<(f64, f64, f64)> = Vec::new();            // (sum of labels, weight, max score)
    for &i in &idx {
        blocks.push((y[i] as u8 as f64, 1.0, s[i]));
        while blocks.len() > 1 {
            let (l, r) = (blocks[blocks.len() - 2], blocks[blocks.len() - 1]);
            if l.0 / l.1 <= r.0 / r.1 { break; }
            blocks.pop(); blocks.pop();
            blocks.push((l.0 + r.0, l.1 + r.1, r.2));              // merge violators
        }
    }
    blocks.iter().map(|b| (b.2, b.0 / b.1)).collect()
}

pub fn isotonic_predict(model: &[(f64, f64)], s: f64) -> f64 {
    let i = model.partition_point(|b| b.0 < s).min(model.len() - 1);
    model[i].1
}

fn main() {
    let (n, k) = (5000, 4);
    let mut st = 3u64;
    let mut u = || { st ^= st << 13; st ^= st >> 7; st ^= st << 17; (st >> 11) as f64 / (1u64 << 53) as f64 };
    let (mut logits, mut y) = (vec![0.0; n * k], vec![0usize; n]);
    for i in 0..n {                                              // true model has logits z; ours reports 3z
        let z: Vec<f64> = (0..k).map(|_| 2.0 * (u() - 0.5)).collect();
        let (lse, _, _) = softmax_stats(&z, 1.0);
        let mut r = u();
        y[i] = (0..k).find(|&c| { r -= (z[c] - lse).exp(); r <= 0.0 }).unwrap_or(k - 1);
        for c in 0..k { logits[i * k + c] = 3.0 * z[c]; }
    }
    let conf_at = |t: f64| -> (Vec<f64>, Vec<bool>) { (0..n).map(|i| {
        let z = &logits[i * k..(i + 1) * k];
        let (lse, _, _) = softmax_stats(z, 1.0 / t);
        let top = (0..k).max_by(|&a, &b| z[a].partial_cmp(&z[b]).unwrap()).unwrap();
        ((z[top] / t - lse).exp(), top == y[i])
    }).unzip() };
    let (c1, ok1) = conf_at(1.0);
    let t = fit_temperature(&logits, &y, k);
    let (ct, okt) = conf_at(t);
    println!("T = {t:.2} (truth 3); ECE before {:.3}, after {:.3}", ece(&c1, &ok1, 15), ece(&ct, &okt, 15));
    let (a, b) = fit_platt(&c1, &ok1);
    let iso = isotonic(&c1, &ok1);
    println!("Platt a = {a:.2}, b = {b:.2}; isotonic: {} blocks, p(conf 0.9) = {:.3}", iso.len(), isotonic_predict(&iso, 0.9));
}

Complexity and constants

• ECE, temperature and Platt fitting: a few passes over the calibration set, milliseconds for 10⁵ examples.
• Isotonic: one sort plus a linear PAV pass (each merge removes a block, so O(n) amortized).
• Calibration set size: a few thousand examples give stable temperature estimates. ECE itself has bias and variance that depend on the bin count; with 15 bins and 10⁴ examples, differences below order of 0.01 are within noise.
• Inference cost: one division of the logits by T, or one sigmoid, or a binary search over isotonic blocks.

Efficiency

• Cost: compute is negligible: fitting temperature, Platt or isotonic takes milliseconds for 10⁵ examples; inference is one division, one sigmoid or a binary search over isotonic blocks (nanoseconds per example, order of). Memory is 1-2 floats for temperature or Platt, one pair per block for isotonic. The real bound is held-out data, not compute: a few thousand examples give stable temperatures.
• Levers:
  • Temperature scaling — one parameter, argmax and accuracy unchanged (Guo 2017) — neural classifiers.
  • Fold 1/T into the final linear layer (W/T, b/T) — exactly zero inference cost (algebra) — deployment.
  • Cross-fitting (CalibratedClassifierCV with cv) — no data wasted on a separate calibration split — small datasets.
  • Platt over isotonic — 2 parameters overfit less than a step function (Niculescu-Mizil 2005) — under about 1,000 calibration points (order of).
  • Debiased ECE estimator — lower bias than binned ECE (Kumar 2019) — checking small shifts.
• Naive → standard → state of the art:
  • Retrain with a different loss hoping for calibrated outputs · hours of training
  • Temperature or Platt fit on a held-out set · milliseconds, 1-2 parameters
  • Temperature folded into the weights, isotonic when data is plentiful, debiased ECE to verify · zero inference cost

Implementation notes

• Hold out a dedicated calibration split (or use cross-validated out-of-fold predictions, as scikit-learn's CalibratedClassifierCV does, default 5-fold with method='sigmoid').
• Fit temperature on logits, never on probabilities. In mixed precision, upcast the logits to fp32 or fp64 first.
• Multi-class with isotonic or Platt: calibrate one-vs-rest per class and renormalize, or use matrix or vector scaling (per-class temperature and bias). Temperature scaling is the safe default.
• Report ECE together with log loss or Brier, plus the number of samples per bin. Adaptive (equal-mass) bins reduce the bias for skewed confidence distributions.
• LLMs: the GPT-4 report found the pretrained model well calibrated on multiple-choice benchmarks and the post-trained (RLHF) model much less so. Calibrate on the deployed checkpoint.
• Re-calibrate after any change of class priors: for a known prior shift, adjust the logits by ln(π_new/π_old) per class.

Pitfalls

• Calibrating on the training set or on the early-stopping validation set used for model selection: optimistic and over-confident results.
• Isotonic regression on a few hundred points: over-fits into a staircase with probabilities 0 and 1. Use Platt or temperature.
• Claiming improvement from lower ECE with a different bin count, or from ECE alone while log loss got worse.
• Calibrating each ensemble member and then averaging: the average of calibrated members tends to be under-confident. Calibrate the averaged prediction.
• Label smoothing during training, then temperature scaling: the two interact. Check the reliability diagram after both.

Variants and alternatives

In production

scikit-learn CalibratedClassifierCV (sigmoid, isotonic) and calibration_curve, IsotonicRegression, torchmetrics CalibrationError, temperature scaling in most production classifiers, click-through-rate models (calibrated probabilities feed ad auctions); clinical prediction-model validation reports a calibration plot and slope; forecast verification in meteorology uses reliability diagrams.

Implementations

Sources

• Van Calster et al., Calibration: the Achilles heel of predictive analytics, BMC Medicine 2019. paper
• Guo, Pleiss, Sun, Weinberger, On Calibration of Modern Neural Networks, ICML 2017. paper
• Platt, Probabilistic Outputs for Support Vector Machines and Comparisons to Regularized Likelihood Methods, Advances in Large Margin Classifiers, 1999.
• Zadrozny and Elkan, Transforming Classifier Scores into Accurate Multiclass Probability Estimates, KDD 2002. paper
• Niculescu-Mizil and Caruana, Predicting Good Probabilities with Supervised Learning, ICML 2005. paper
• Naeini, Cooper, Hauskrecht, Obtaining Well Calibrated Probabilities Using Bayesian Binning, AAAI 2015.
• Murphy, A New Vector Partition of the Probability Score, Journal of Applied Meteorology 1973.
• Kumar, Liang, Ma, Verified Uncertainty Calibration, NeurIPS 2019. paper · code
• OpenAI, GPT-4 Technical Report, 2023. paper