A/V sync and media clocks

Deep dive

How it works

In: timestamped frames and audio blocks from several streams, each with its own timebase; transport clock samples (PCR, RTCP SR) · Out: per frame a present, repeat or drop decision at the right display time · State: per-stream timebase and offset, the master clock, queued audio · Cost: one rescale and one clock read per frame; negligible next to decode

1. Timebases. Each stream counts ticks of a rational unit: 1/90000 (MPEG-TS, RTP video), 1/sample_rate (audio), 1001/30000 or a container timescale. Convert with exact integer math: ts * from.num * to.den / (from.den * to.num), rounded. Never accumulate float seconds.
2. PTS vs DTS. B-frames are decoded out of display order, so DTS (decode) differs from PTS (present). PTS - DTS is the reorder delay; see GOP structure.
3. Wrap and offset. Unwrap 33-bit values, subtract each stream's first timestamp or the container's start time. Streams rarely start at 0 or at the same value.
4. Transport clock. MPEG-TS: PCR (base*300 + extension, 27 MHz) samples the encoder's system clock; the receiver locks to it by arrival time (PLL). RTP: timestamps have a random start; RTCP sender reports pair an RTP timestamp with a 64-bit NTP wall time per stream, which aligns audio and video (RTP/RTCP, RFC 3550).
5. Master clock. Audio clock = PTS of the last sample handed to the device minus the audio still queued (and minus device latency), interpolated by the system clock between callbacks.
6. Slaves. Each display tick, show the newest video frame that is due; frames already past are dropped; frames in the future wait. ffplay's rule: below 40 ms difference do nothing, above 100 ms always correct, with the frame duration in between; beyond 10 s assume a timestamp jump and stop correcting.
7. Drift. Gradual error is absorbed by resampling audio slightly (ffplay caps the change at 10 %; keep it far smaller) or by occasional frame repeats and drops.

audio device ─► written PTS 5.200 − queued 0.080 = master 5.120
video frames:   5.083  5.125  5.167        display tick at master 5.120 (tick 16.7 ms)
                 drop   show*   wait       * due: pts <= master + tick/2

Code

use std::collections::VecDeque;

/// Timebase: one tick = num/den seconds (1/90000, 1001/30000, ...).
#[derive(Clone, Copy)]
pub struct Tb { pub num: u32, pub den: u32 }

/// Rescale between timebases, round half away from zero. i128 because
/// ts * 90000 * 1_000_000 overflows i64 after a few hours.
pub fn rescale(ts: i64, from: Tb, to: Tb) -> i64 {
    let n = ts as i128 * from.num as i128 * to.den as i128;
    let d = from.den as i128 * to.num as i128;
    let q = (2 * n.abs() + d) / (2 * d);
    (if n < 0 { -q } else { q }) as i64
}

/// Unwrap 33-bit MPEG PTS/DTS; tolerates small backward steps across the wrap.
#[derive(Default)]
pub struct Unwrap33 { last: Option<i64> }
impl Unwrap33 {
    pub fn push(&mut self, raw: u64) -> i64 {
        const M: i64 = 1 << 33;
        let raw = raw as i64 & (M - 1);
        let v = match self.last {
            None => raw,
            Some(l) => {
                let mut v = l - l.rem_euclid(M) + raw;
                if v - l > M / 2 { v -= M } else if l - v > M / 2 { v += M }
                v
            }
        };
        self.last = Some(v);
        v
    }
}

/// PCR = base (33 bit, 90 kHz) * 300 + extension (9 bit): 27 MHz ticks.
pub fn pcr_27mhz(base: u64, ext: u16) -> u64 { base * 300 + ext as u64 }

/// RTCP SR anchor for one stream: an RTP timestamp <-> NTP seconds.
pub struct SrMap { ntp_s: f64, rtp: u32, hz: f64 }
impl SrMap {
    pub fn new(ntp_s: f64, rtp: u32, hz: u32) -> Self { SrMap { ntp_s, rtp, hz: hz as f64 } }
    pub fn to_wall(&self, ts: u32) -> f64 {
        self.ntp_s + ts.wrapping_sub(self.rtp) as i32 as f64 / self.hz // signed delta survives u32 wrap
    }
}

/// Clock slaved to the audio device: set from the audio callback, interpolated between calls.
pub struct MasterClock { pts: f64, at: f64, pub speed: f64 }
impl MasterClock {
    pub fn new() -> Self { MasterClock { pts: 0.0, at: 0.0, speed: 1.0 } }
    pub fn set(&mut self, pts: f64, now: f64) { self.pts = pts; self.at = now; }
    pub fn get(&self, now: f64) -> f64 { self.pts + (now - self.at) * self.speed }
}

/// Per display tick: show the newest due frame, drop the older due ones.
pub struct Presenter { q: VecDeque<f64>, pub dropped: u32 }
impl Presenter {
    pub fn new() -> Self { Presenter { q: VecDeque::new(), dropped: 0 } }
    pub fn push(&mut self, pts: f64) { self.q.push_back(pts) }
    pub fn pick(&mut self, master: f64, tick: f64) -> Option<f64> {
        let mut shown = None;
        while self.q.front().map_or(false, |&p| p <= master + tick / 2.0) {
            if shown.is_some() { self.dropped += 1; }
            shown = self.q.pop_front();
        }
        shown
    }
}

/// Audio resample ratio closing `diff_s` (video - audio) over `window_s`, capped at 2 %.
pub fn drift_ratio(diff_s: f64, window_s: f64) -> f64 { 1.0 + (diff_s / window_s).clamp(-0.02, 0.02) }

#[test]
fn checks() {
    let (t90, us) = (Tb { num: 1, den: 90_000 }, Tb { num: 1, den: 1_000_000 });
    assert_eq!(rescale(3003, t90, us), 33_367);                          // 29.97 fps frame
    assert_eq!(rescale(1920, t90, Tb { num: 1, den: 48_000 }), 1024);    // AAC frame
    let mut u = Unwrap33::default();
    let m = 1u64 << 33;
    assert_eq!(u.push(m - 10), (m - 10) as i64);
    assert_eq!(u.push(5), (m + 5) as i64);                               // wrap
    assert_eq!(u.push(m - 1), (m - 1) as i64);                           // late B-frame
    assert_eq!(pcr_27mhz(90_000, 299), 27_000_299);
    let a = SrMap::new(1000.0, 0xFFFF_FF00, 48_000);
    assert!((a.to_wall(0xFFFF_FF00u32.wrapping_add(48_000)) - 1001.0).abs() < 1e-9);
    // 24 fps video, 60 Hz display, renderer stalls for 0.3 s: frames are dropped, none shown twice.
    let (mut clk, mut p) = (MasterClock::new(), Presenter::new());
    (0..48).for_each(|i| p.push(i as f64 / 24.0));
    let (mut now, mut shown, mut last) = (0.0, 0, -1.0);
    while now < 2.0 {
        clk.set(now, now);
        if !(0.5..0.8).contains(&now) {
            if let Some(f) = p.pick(clk.get(now), 1.0 / 60.0) { assert!(f > last); last = f; shown += 1; }
        }
        now += 1.0 / 60.0;
    }
    assert_eq!(shown + p.dropped as i32, 48);
}

Compiled with stable Rust and run as a test; the stall run shows 42 frames and 6 dropped.

Complexity and constants

• Per frame: one rescale (an i128 multiply and divide, tens of ns order of), one clock read, one queue pop. Negligible next to decode.
• Memory: a few dozen bytes of state per stream; the queues (decoded frames, audio ring) dominate: 1080p NV12 is 3.1 MB per frame, so an 8-frame queue is about 25 MB.
• Exactness: 29.97 fps is exactly 3003 ticks at 90 kHz and AAC-LC 1024 samples at 48 kHz is exactly 1920 ticks, but 23.976 fps is 3753.75 ticks: frames alternate 3754/3754/3754/3753. MP4 avoids this with timescale 24000 and duration 1001.
• Wrap: 33 bits at 90 kHz is 26.5 h; a 64-bit microsecond count lasts hundreds of thousands of years.

Efficiency

• Cost: per frame one rescale (an i128 multiply and divide, order of tens of ns), one clock read and one queue pop. Memory is a few dozen bytes of state per stream; the queues dominate (1080p NV12 is 3.1 MB per frame, so 8 frames are about 25 MB). Bound: latency and correctness, not compute; the costs are wake-ups, queued bytes and wasted decoding.
• Levers:
  • present on vsync only — wake once per display refresh, sleep to the next deadline with one call, never busy-wait — any player
  • drop early, not late — a non-reference frame dropped before decode saves its decode and colour conversion — when the video is behind
  • correct by resampling, not seeking — a speed change of order 0.1 % needs no flush or re-buffering; repeating one frame per few seconds is cheaper than re-syncing — small drift
  • exact integer timebases — no float accumulation, no rounding drift (23.976 fps alternates 3754/3754/3754/3753 ticks) — long streams
• Naive → standard → state of the art:
  • system clock for both streams · 180 ms drift per hour at 50 ppm
  • audio master with drop and repeat · video follows the audio clock, one repeated or dropped frame per correction
  • audio master plus resampling for small drift, and transport-clock mapping (PCR, RTCP SR) · small drift is absorbed without repeating or dropping frames (design property)

Implementation notes

• Query the real output latency (AudioTimeStamp on CoreAudio, IAudioClock on WASAPI, snd_pcm_delay on ALSA, AudioTrack.getTimestamp on Android) and subtract it; Bluetooth output adds order of 100+ ms that a naive clock ignores.
• Update the audio clock in the audio callback (O(1), no locks beyond an atomic pair), read it from the render thread.
• Pause stops the clock (speed = 0); seek flushes queues and re-seeds the clock with the first audible PTS; a PTS discontinuity (ad splice, TS discontinuity flag, new RTP SSRC) resets the offset instead of reaching a 10 s correction.
• Codec delay is not sync error: AAC and Opus encoder priming (Opus pre-skip, MP4 edit lists) must be removed so that the first sample matches the first PTS.
• RTP: wait for the first RTCP SR on every stream before mapping to wall time; until then play unsynchronized. Sync Opus on its fixed 48 kHz RTP clock regardless of the encoder rate.

Pitfalls

• Using wall time as the clock while audio runs on its own crystal: audio underruns or overflows within minutes.
• Taking the video PTS as master: audio resampling noise, and a dropped frame becomes a dropped audio chunk.
• Forgetting the device queue: audio clock set to "last written PTS" leads by the entire queue (tens to hundreds of ms).
• Floats: pts += 1/29.97 accumulates error; keep integer ticks, convert on output.
• Treating a timestamp jump as drift: always cap, then resync.

Variants and alternatives

In production

• Players: FFmpeg ffplay (audio master by default), mpv, VLC; GStreamer keeps a pipeline clock and base time with clock slaving per sink (docs).
• Real-time calls: RTP's RTCP sender reports let a receiver synchronize a source's audio and video (RFC 3550); libwebrtc has a stream synchronization module (code).
• Broadcast: set-top box demuxers lock a PLL to the PCR; BT.1359-1 sets the sound/vision timing limits (Sources).
• Robotics: ROS message_filters ApproximateTime matches messages from several topics even when their time stamps differ (docs).

Implementations

Sources

• ITU-R, Recommendation BT.1359-1: Relative timing of sound and vision for broadcasting, 1998. PDF
• H. Schulzrinne et al., RFC 3550: RTP: A Transport Protocol for Real-Time Applications, 2003. RFC
• ISO/IEC 13818-1 (ITU-T H.222.0), MPEG-2 Systems (PTS, DTS, PCR).
• FFmpeg, fftools/ffplay.c: compute_target_delay, AV_SYNC_THRESHOLD_MIN/MAX, SAMPLE_CORRECTION_PERCENT_MAX.
• mpv manual, --video-sync.