Perceptual quality metrics (PSNR, SSIM, VMAF)

Deep dive

How it works

In: a reference frame (or video) and a distorted one, aligned frame for frame, plus a model for VMAF · Out: a score per frame and a pooled score (dB for PSNR, 0-1 for SSIM, 0-100 for VMAF) · State: none (VMAF's motion term looks at the previous frame) · Cost: PSNR about 0.1 ms, SSIM about 3 ms per 1080p luma frame on one core (measured below); VMAF far heavier

1. PSNR = 10 log10(255^2 / MSE) for 8-bit. +6 dB is half the error amplitude. Compute per plane (Y, U, V) and report separately.
2. SSIM, per window: l * cs, with l = (2 mx my + C1) / (mx^2 + my^2 + C1) and cs = (2 cov + C2) / (vx + vy + C2); C1 = (0.01*255)^2, C2 = (0.03*255)^2. The score is the mean over windows (Wang et al. 2004: 11x11 Gaussian, sigma 1.5, stride 1).
3. MS-SSIM: repeat on 2x downsampled copies, 5 scales, and combine cs of scales 1-4 and l*cs of scale 5 as a weighted product (exponents 0.0448, 0.2856, 0.3001, 0.2363, 0.1333).
4. VMAF: per frame, extract VIF (4 scales), ADM (detail-loss measure) and a motion term (mean absolute luma difference between adjacent frames), regress to a score, then average per-frame scores. Quality is calibrated so "bad" is about 20 and "excellent" 100.
5. Pool over time: mean hides short disasters; also report the minimum or a low percentile (libvmaf also gives the harmonic mean).

Code

/// PSNR in dB between two 8-bit planes (infinity if identical).
pub fn psnr(a: &[u8], b: &[u8]) -> f64 {
    assert_eq!(a.len(), b.len());
    let sse: u64 = a.iter().zip(b).map(|(&x, &y)| { let d = x as i32 - y as i32; (d * d) as u64 }).sum();
    if sse == 0 { return f64::INFINITY; }
    10.0 * (255.0f64 * 255.0 * a.len() as f64 / sse as f64).log10()
}

/// Mean luminance term l, mean contrast-structure term cs, and mean SSIM (= mean l*cs)
/// over uniform 8x8 windows at stride 4. Wang et al. used an 11x11 Gaussian window
/// (sigma 1.5, stride 1); the 8x8 box is the fast variant common in video tools.
pub fn ssim_terms(a: &[u8], b: &[u8], w: usize, h: usize) -> (f64, f64, f64) {
    assert!(a.len() == w * h && b.len() == w * h && w >= 8 && h >= 8);
    const C1: f64 = (0.01 * 255.0) * (0.01 * 255.0);
    const C2: f64 = (0.03 * 255.0) * (0.03 * 255.0);
    let (mut sl, mut scs, mut ss, mut n) = (0.0, 0.0, 0.0, 0u32);
    for y in (0..=h - 8).step_by(4) {
        for x in (0..=w - 8).step_by(4) {
            let (mut sa, mut sb, mut saa, mut sbb, mut sab) = (0u32, 0u32, 0u32, 0u32, 0u32);
            for j in 0..8 { // integer sums are exact; floats only at the end
                let o = (y + j) * w + x;
                for i in 0..8 {
                    let (p, q) = (a[o + i] as u32, b[o + i] as u32);
                    sa += p; sb += q; saa += p * p; sbb += q * q; sab += p * q;
                }
            }
            let (ma, mb) = (sa as f64 / 64.0, sb as f64 / 64.0);
            let (va, vb) = (saa as f64 / 64.0 - ma * ma, sbb as f64 / 64.0 - mb * mb);
            let cov = sab as f64 / 64.0 - ma * mb;
            let l = (2.0 * ma * mb + C1) / (ma * ma + mb * mb + C1);
            let cs = (2.0 * cov + C2) / (va + vb + C2);
            sl += l; scs += cs; ss += l * cs; n += 1;
        }
    }
    let n = n as f64;
    (sl / n, scs / n, ss / n)
}

pub fn ssim(a: &[u8], b: &[u8], w: usize, h: usize) -> f64 { ssim_terms(a, b, w, h).2 }

/// 2x2 box average: the low-pass + decimate step between scales.
fn down2(a: &[u8], w: usize, h: usize) -> Vec<u8> {
    let mut o = Vec::with_capacity(w / 2 * (h / 2));
    for y in 0..h / 2 { for x in 0..w / 2 {
        let i = 2 * y * w + 2 * x;
        o.push(((a[i] as u32 + a[i + 1] as u32 + a[i + w] as u32 + a[i + w + 1] as u32 + 2) / 4) as u8);
    }}
    o
}

/// MS-SSIM (Wang, Simoncelli, Bovik 2003): cs^beta_j at scales 1..4, l*cs at scale 5.
pub fn ms_ssim(a: &[u8], b: &[u8], w: usize, h: usize) -> f64 {
    const BETA: [f64; 5] = [0.0448, 0.2856, 0.3001, 0.2363, 0.1333];
    let (mut a, mut b, mut w, mut h) = (a.to_vec(), b.to_vec(), w, h);
    let mut r = 1.0;
    for (i, beta) in BETA.iter().enumerate() {
        let (l, cs, _) = ssim_terms(&a, &b, w, h);
        if i == BETA.len() - 1 { r *= (l * cs).max(1e-6).powf(*beta); }
        else { r *= cs.max(1e-6).powf(*beta); a = down2(&a, w, h); b = down2(&b, w, h); w /= 2; h /= 2; }
    }
    r
}

#[test]
fn metrics() {
    let (w, h) = (128, 128);
    let img: Vec<u8> = (0..w * h).map(|i| (((i % w) * 2 + i / w) as u8 ^ (i * 7 % 13) as u8).clamp(20, 235)).collect();
    assert!(psnr(&img, &img).is_infinite() && (ms_ssim(&img, &img, w, h) - 1.0).abs() < 1e-9);
    let shifted: Vec<u8> = img.iter().map(|&p| p + 10).collect();
    let alt: Vec<u8> = img.iter().enumerate().map(|(i, &p)| if i % 2 == 0 { p + 10 } else { p - 10 }).collect();
    // identical MSE (100) and PSNR (28.13 dB), very different SSIM
    assert!((psnr(&img, &shifted) - psnr(&img, &alt)).abs() < 1e-9);
    assert!(ssim(&img, &shifted, w, h) > 0.98 && ssim(&img, &alt, w, h) < 0.7);
}

Complexity and constants

• PSNR: one pass, 2 reads per pixel; the squared error is SIMD-friendly (16 pixels per 128-bit register).
• SSIM: five running sums per window; with an integral image or separable filters it is O(1) per window, independent of window size.
• MS-SSIM adds the downsample passes and about 1/3 more window work (each scale has 1/4 the pixels).
• Measured here (Apple M4 Pro, one core, 1080p luma, code above, synthetic data in cache): PSNR about 0.1 ms, SSIM (8x8, stride 4) about 3 ms, MS-SSIM about 5 ms (order of).
• VMAF's features are far heavier (multi-scale filtering); libvmaf 2.0's fixed-point AVX2/AVX-512 path is stated to be 2x faster than the float one.

Efficiency

• Cost: measured here (Apple M4 Pro, one core, 1080p luma, synthetic data in cache): PSNR about 0.1 ms, SSIM (8x8, stride 4) about 3 ms, MS-SSIM about 5 ms (order of). VMAF's features are far heavier; libvmaf 2.0's fixed-point AVX2/AVX-512 path is stated to be 2x faster than the float one. Bound: memory bandwidth for PSNR, arithmetic for VMAF.
• Levers:
  • vmaf --subsample N — scores every Nth frame, cost drops about N-fold, with sampling error on non-stationary content — long videos
  • threads and GPU — libvmaf takes --threads; a CUDA build exists (-Denable_cuda=true) — batch scoring
  • luma only — each 4:2:0 chroma plane is 1/4 of the pixels — ranking encoder settings
  • no YUV round trip — raw 1080p 8-bit 4:2:0 is about 3.1 MB per frame, 93 MB/s at 30 fps; let the encoder report PSNR/SSIM (x264 --psnr --ssim) or pipe decoded frames — pipelines
• Naive → standard → state of the art:
  • PSNR on every frame · about 0.1 ms per 1080p frame, ranks reliably only within one codec and content
  • SSIM or MS-SSIM for the inner search · 3-5 ms per frame
  • PSNR or SSIM for inner-loop search, VMAF on every Nth frame to confirm the final ladder · VMAF cost divided by N

Implementation notes

• Frames must match in size and timing: scale the distorted video up to the reference size (ffmpeg scale before libvmaf, as in the libvmaf FAQ) and match frame rates first.
• Compute at the reference bit depth. libvmaf caps PSNR at 60 dB for 8-bit and 72 dB for 10-bit, so identical frames do not give infinity. VMAF v1 (June 2026) should be run at 10-bit for SDR to catch banding.
• Integer window sums (as in the code) are exact; convert to float once per window.
• Gaussian-weighted SSIM, 11x11, stride 1, is the reference definition; box-8x8 numbers are not comparable to it. Say which one you ran.

Pitfalls

• Comparing PSNR across codecs or content: a codec that blurs wins on SSIM/VMAF and loses on PSNR, and vice versa.
• Averaging over a whole title: a short blocky scene disappears in the mean.
• Reporting VMAF without the model (1080p/3H, phone, 4K) or version (v0.6.1 vs v1).
• Using plain VMAF to judge a codec that adds sharpening or contrast: use the NEG model files for codec comparison.
• Misaligned frames (one off, wrong colour range or matrix) silently wreck every metric.

Variants and alternatives

In production

• Video streaming: Netflix built its Dynamic Optimizer (shot-based encoding) on VMAF (docs); quality targets for a transcoding ladder and adaptive bitrate streaming are often set in VMAF.
• Image codecs: ssimulacra2, butteraugli and dssim score still-image codec output against the source (see Variants and alternatives).
• Machine learning: LPIPS compares deep-network features as a learned perceptual metric, built after deep features proved useful as training losses for image synthesis (Sources).
• Rate-distortion optimization inside encoders mostly minimises SSE (PSNR-style); perceptual tuning such as --tune ssim and adaptive quantization sits on top.

Implementations

Sources

• Z. Wang, A. Bovik, H. Sheikh, E. Simoncelli, Image quality assessment: from error visibility to structural similarity, IEEE TIP 13(4), 2004. paper
• Z. Wang, E. Simoncelli, A. Bovik, Multiscale structural similarity for image quality assessment, Asilomar 2003. paper
• H. Sheikh, A. Bovik, Image information and visual quality, IEEE TIP 15(2), 2006. paper
• S. Li, F. Zhang, L. Ma, K. Ngan, Image quality assessment by separately evaluating detail losses and additive impairments, IEEE Trans. Multimedia 13(5), 2011.
• Q. Huynh-Thu, M. Ghanbari, Scope of validity of PSNR in image/video quality assessment, Electronics Letters 44(13), 2008. paper
• C. Helmrich et al., XPSNR: a low-complexity extension of the perceptually weighted PSNR for high-resolution video quality assessment, ICASSP 2020. paper
• R. Zhang, P. Isola, A. Efros, E. Shechtman, O. Wang, The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (LPIPS), CVPR 2018. paper · arXiv
• G. Bjontegaard, Calculation of average PSNR differences between RD-curves, ITU-T VCEG-M33, 2001.
• Netflix VMAF documentation: models (v0), models (v1), FAQ, features.