Adaptive quantization

Deep dive

How it works

In: source luma of a frame, a base QP from rate control, a strength · Out: a QP offset per block (or quantization group, or segment) · State: none; the map depends only on the source frame (some encoders add temporal smoothing) · Cost: O(pixels), one read of luma; about 480 µs per 1080p frame scalar (measured below)

1. Activity. For each 16x16 luma block (x264 macroblock; HEVC/AV1 use quantization groups or superblocks) compute the AC energy: the sum of squared deviations from the block mean. A flat block gives about 0, noise or edges give thousands.
2. Map to a QP offset. Human sensitivity falls roughly with the log of activity. x264 mode 1: off = strength * 1.0397 * (log2(max(E,1)) - 14.427), zero at E of about 21.6k. Mode 2 uses E^(1/8) centred on the frame mean (which keeps the frame average near the base QP).
3. Combine. qp = clamp(round(base_qp + off), qp_min, qp_max); base_qp comes from rate control (CRF, ABR, CBR). The RDO lambda follows the block QP.
4. Signal. H.264 sends mb_qp_delta against the previous macroblock in decode order (range -26..+25 for 8-bit); HEVC sends cu_qp_delta once per quantization group; VP9 and AV1 can use up to 8 segments with a QP delta each, AV1 also per-superblock delta q.
5. Decode. The decoder dequantises each block with its own QP and the loop filter uses the average QP of the two sides of an edge.

frame: [flat sky: E~0   ][ textured grass: E~20k+ ]
offset:   -15 (clamped)      +4..+5     (strength 1.0, mode 1/2 as in the test below)
QP:        11                30..31     (base QP 26)

Code

Luma-only reference of x264's modes 1 and 2 (x264 also adds chroma energy). The test below drives a frame with a flat half and a noisy half.

pub const MB: usize = 16;

#[derive(Clone, Copy, PartialEq, Debug)]
pub enum Mode { Variance, AutoVariance } // x264 --aq-mode 1 and 2

/// AC energy of one 16x16 luma block: sum of squared deviations from the block mean.
pub fn ac_energy(luma: &[u8], stride: usize, bx: usize, by: usize) -> u32 {
    let (mut sum, mut ssd) = (0u32, 0u32);
    for y in 0..MB {
        let row = &luma[(by * MB + y) * stride + bx * MB..][..MB];
        for &p in row {
            sum += p as u32;
            ssd += p as u32 * p as u32;
        }
    }
    ssd - (sum * sum >> 8) // sum of squares minus N*mean^2, N = 256
}

/// Per-block QP offsets (in QP steps; negative = finer) for a whole frame.
/// Constants follow x264 encoder/ratecontrol.c (luma only here, x264 adds chroma energy).
pub fn qp_offsets(luma: &[u8], width: usize, height: usize, mode: Mode, strength: f32) -> Vec<f32> {
    let (mbw, mbh) = (width / MB, height / MB);
    let energy: Vec<u32> = (0..mbw * mbh)
        .map(|i| ac_energy(luma, width, i % mbw, i / mbw))
        .collect();
    match mode {
        // log mask: offset = s * (log2(E) - 14.427), 0 at E ~ 21.6k (stddev ~ 9.2)
        Mode::Variance => energy
            .iter()
            .map(|&e| strength * 1.0397 * ((e.max(1) as f32).log2() - 14.427))
            .collect(),
        // E^(1/8) mask, centred on the frame mean (so the frame average stays near the base QP)
        Mode::AutoVariance => {
            let a: Vec<f32> = energy.iter().map(|&e| ((e + 1) as f32).powf(0.125)).collect();
            let n = a.len() as f32;
            let avg = a.iter().sum::<f32>() / n;
            let avg2 = a.iter().map(|x| x * x).sum::<f32>() / n;
            let centre = avg - 0.5 * (avg2 - 14.0) / avg; // x264's variance correction
            a.iter().map(|&x| strength * avg * (x - centre)).collect()
        }
    }
}

/// Final QP the block is coded with. H.264/HEVC code the difference to the previous
/// block's QP (mb_qp_delta / cu_qp_delta), so smooth offsets are cheaper to signal.
pub fn block_qp(base: u8, off: f32, min: u8, max: u8) -> u8 {
    (base as f32 + off).round().clamp(min as f32, max as f32) as u8
}

/// H.264 quantiser step, approx.: 0.625 at QP 0, doubling every 6 QP (spec table exact from QP 6 up).
pub fn qstep(qp: u8) -> f32 {
    0.625 * (2f32).powf(qp as f32 / 6.0)
}

#[cfg(test)]
mod tests {
    use super::*;

    fn frame() -> Vec<u8> {
        // 64x32: left 32 columns flat 128, right 32 columns pseudo-random texture
        let (w, h) = (64, 32);
        let mut s = 12345u32;
        let mut f = vec![128u8; w * h];
        for y in 0..h {
            for x in 32..w {
                s = s.wrapping_mul(1664525).wrapping_add(1013904223);
                f[y * w + x] = 64 + (s >> 24) as u8 / 2; // 64..191
            }
        }
        f
    }

    #[test]
    fn flat_gets_finer_texture_gets_coarser() {
        let f = frame();
        for mode in [Mode::Variance, Mode::AutoVariance] {
            let o = qp_offsets(&f, 64, 32, mode, 1.0); // 4x2 blocks, row-major
            assert!(o[0] < -5.0 && o[1] < -5.0, "{mode:?} flat {o:?}");
            assert!(o[2] > 0.0 && o[3] > 0.0, "{mode:?} texture {o:?}");
            assert!(o[0] == o[1] && o[2] > o[0]);
            println!("{mode:?}: {o:?}");
        }
        let o = qp_offsets(&f, 64, 32, Mode::Variance, 0.0);
        assert!(o.iter().all(|&x| x == 0.0)); // strength 0 = AQ off
    }

    #[test]
    fn qstep_doubles_every_6() {
        assert!((qstep(4) - 1.0).abs() < 0.02);
        assert!((qstep(28) / qstep(22) - 2.0).abs() < 1e-5);
        assert_eq!(block_qp(26, -15.0, 0, 51), 11);
        assert_eq!(block_qp(50, 9.0, 0, 51), 51);
    }
}

Run as cargo test: the flat half gets about -15 (mode 1) or -6.5 (mode 2), the noisy half about +4 to +5 at strength 1.0; strength 0 returns all zeros.

Complexity and constants

• Analysis: O(pixels), one read of source luma, about 1 byte per pixel; the 16x16 sums fit in u32 for 8-bit input (sum*sum up to 4.26e9). 10-bit needs u64 and x264's 1/(1<<(2*(bitdepth-8))) scaling.
• Measured here: the scalar Rust above takes about 480 µs per 1080p frame on one core of this Mac (aarch64, release, noise frame, mode 2), about 4 GB/s; x264 uses SIMD variance kernels.
• Signalling: one delta per block or quantization group, order of a few bits each when neighbours are similar; smooth offsets are cheaper than jumpy ones.
• Cost drivers: offset-map block size (smaller = more deltas), QP jitter between neighbours, and the extra bits spent on flat blocks.

Efficiency

• Cost: about 53 µs per 640x360 frame (scaling of the 480 µs 1080p figure above), against about 8 ms per frame for a whole x264 medium encode on one thread here (120 frames in about 1 s): under 1%. The bound is bits, not CPU: AQ is a rate redistribution. Measured on one synthetic clip (x264 0.164 medium, 2-pass ABR 120 kbps, 640x360, 4 s, smooth top half, noisy bottom half): no AQ 53.1 kB, flat half 41.3 dB; mode 2 53.8 kB (+1.3%), flat half 49.3 dB, noisy half unchanged at 35.7 dB. Real footage will differ.
• Levers:
  • strength — 1.0 default; x264 tunes use 0.6 (animation), 0.5 (grain), 1.2 (stillimage) — content-dependent
  • compute the map once — it depends on the source, not the bitrate, so reuse across ladder renditions and passes (suggestion, not measured) — multi-bitrate encodes
  • fixed low QP in known flat regions (letterbox bars, UI), or VP9/AV1 segments (at most 8 QP levels per frame) — a coarse map instead of per-block — hardware encoders that expose only ROI or segment maps
  • mode 2 over mode 1 — keeps the frame average QP near the base (+1.3% vs +3.3% bytes in the clip above) — rate-constrained encodes
• Naive → standard → state of the art:
  • one QP per frame · lowest PSNR error, blocking in flat areas
  • per-block variance AQ (x264 mode 1/2) · +1.3% to +3.3% bytes in the clip above for a 8-10 dB gain on the flat half
  • AQ with RDO whose lambda follows the block QP (step 3) · same analysis pass

Implementation notes

• Compute energy on the source frame in the lookahead stage, so the map is ready when the block is coded.
• Keep the frame mean offset near zero (mode 2 does) so rate control does not fight AQ; mode 1 is not centred and shifts the bitrate.
• Clamp the final QP and the delta to the legal range; with flat blocks mode 1 hits about -15 and qp_min takes over.
• Match the offset map to the coding unit: x265 --qg-size defaults to 32 (choices 64, 32, 16, 8) and --aq-strength runs 0 to 3.0; x265 --aq-mode 4 adds edge information.
• x264 --aq-mode 3 adds a bias for dark scenes; MB-tree (temporal QP offsets) stacks with AQ.
• ROI or saliency maps can be added to the offsets (x264 takes per-block quant_offsets).

Pitfalls

• Judging AQ by PSNR: it reads worse. x264 --tune psnr sets --aq-mode 0, --tune ssim sets --aq-mode 2.
• Perfectly flat regions (black bars, fades) get the maximum negative offset and can waste bits or fight qp_min.
• Abrupt QP steps between neighbours show as block edges and cost delta bits.
• Noise-free gradients can still band at 8 bits; dithering or 10-bit is the fix, AQ only helps.

Variants and alternatives

In production

• Software encoders: default on in x264 (aq-mode 1), x265 (aq-mode 2) and SVT-AV1 (aq-mode 2); FFmpeg passes -aq-mode and -aq-strength to libx264.
• Hardware encoders: the NVIDIA NVENC programming guide describes spatial AQ, which gives extra bits to low-complexity flat regions because quality differences are more visible there (docs).
• Standards: H.264 mb_qp_delta, HEVC cu_qp_delta and AV1 segmentation with delta q carry the per-block QP (Sources).

Implementations

Sources

• x264 encoder/ratecontrol.c (constants 1.0397, 14.427, exponent 0.125) and x264 --fullhelp (modes, defaults, tunes). code
• x265 CLI documentation, --aq-mode, --aq-strength, --qg-size. docs
• Thomas Wiegand, Gary J. Sullivan, Gisle Bjontegaard, Ajay Luthra, Overview of the H.264/AVC Video Coding Standard, IEEE TCSVT 2003 (mb_qp_delta, QP and step size). paper
• Gary J. Sullivan, Jens-Rainer Ohm, Woo-Jin Han, Thomas Wiegand, Overview of the High Efficiency Video Coding (HEVC) Standard, IEEE TCSVT 2012 (quantization groups, cu_qp_delta). paper
• Gary J. Sullivan, Thomas Wiegand, Rate-distortion optimization for video compression, IEEE Signal Processing Magazine 1998. paper
• Chun-Hsien Chou, Yun-Chin Li, A perceptually tuned subband image coder based on the measure of just-noticeable-distortion profile, IEEE TCSVT 1995 (spatial masking model behind activity-based quantization). paper
• FFmpeg wiki, H.264 Video Encoding Guide (+6 CRF roughly halves bitrate). guide
• AOMedia, AV1 Bitstream & Decoding Process Specification (segmentation, delta q). spec