Voice activity detection

Deep dive

How it works

In: a stream of audio frames (e.g. 160 samples = 10 ms at 16 kHz) · Out: speech segments [start, end) in frames, or a per-frame flag · State: noise-floor window (a sliding minimum), in-speech flag, two run counters, segment start · Cost: O(frame length) for the feature, O(1) amortized for floor and decision

1. Feature: log frame energy 10·log10(mean x²) in dB. Richer: per-band energies (WebRTC: six bands, 80 Hz-4 kHz), zero-crossing rate, spectral flatness, or log-mel input for a network.
2. Noise floor: minimum of the feature over a window longer than any speech burst (here 2 s). It needs no speech label; a slow average updated only on non-speech frames freezes when the noise steps up.
3. Score: e − floor in dB (energy), a speech-vs-noise likelihood ratio (statistical) or a network output in [0, 1] (neural).
4. Enter: min_on consecutive frames above enter_db; the start is back-dated to the first of them.
5. Stay: between exit_db and enter_db the state does not change (hysteresis).
6. Exit: hang consecutive frames below exit_db; the segment ends at the first of them. The hangover keeps word endings and short pauses inside one segment.
7. Optional padding of a few frames restores weak onsets such as fricatives.

Code

use std::collections::VecDeque;

/// Energy VAD: log energy, sliding-minimum noise floor, hysteresis, onset confirmation, hangover.
pub struct Vad {
    win: usize, enter_db: f32, exit_db: f32, min_on: usize, hang: usize,
    idx: usize,                   // frame counter
    mins: VecDeque<(usize, f32)>, // monotonic deque (frame, dB), increasing dB
    in_speech: bool, run_on: usize, run_off: usize, start: usize,
    pub segments: Vec<(usize, usize)>, // [start, end) in frames
}

impl Vad {
    pub fn new(win: usize, enter_db: f32, exit_db: f32, min_on: usize, hang: usize) -> Self {
        Vad { win, enter_db, exit_db, min_on, hang, idx: 0, mins: VecDeque::new(),
              in_speech: false, run_on: 0, run_off: 0, start: 0, segments: vec![] }
    }

    pub fn push(&mut self, frame: &[f32]) -> bool {
        let e = 10.0 * (frame.iter().map(|x| x * x).sum::<f32>() / frame.len() as f32 + 1e-9).log10();
        // sliding minimum, O(1) amortized
        while self.mins.back().map_or(false, |&(_, v)| v >= e) { self.mins.pop_back(); }
        self.mins.push_back((self.idx, e));
        while self.mins.front().unwrap().0 + self.win <= self.idx { self.mins.pop_front(); }
        let floor = self.mins.front().unwrap().1;

        if !self.in_speech {
            self.run_on = if e > floor + self.enter_db { self.run_on + 1 } else { 0 };
            if self.run_on >= self.min_on {
                self.in_speech = true;
                self.start = self.idx + 1 - self.run_on; // back-dated
                self.run_off = 0;
            }
        } else {
            self.run_off = if e < floor + self.exit_db { self.run_off + 1 } else { 0 };
            if self.run_off >= self.hang {
                self.in_speech = false;
                self.run_on = 0;
                self.segments.push((self.start, self.idx + 1 - self.run_off)); // trim the hangover
            }
        }
        self.idx += 1;
        self.in_speech
    }
    pub fn finish(&mut self) { if self.in_speech { self.segments.push((self.start, self.idx)); } }
}

/// xorshift64* noise, roughly Gaussian: sum of 4 uniforms, unit variance.
pub struct Noise(u64);
impl Noise {
    pub fn next(&mut self) -> f32 {
        let mut s = 0.0;
        for _ in 0..4 {
            self.0 ^= self.0 >> 12; self.0 ^= self.0 << 25; self.0 ^= self.0 >> 27;
            s += (self.0.wrapping_mul(0x2545F4914F6CDD1D) >> 40) as f32 / (1u64 << 24) as f32;
        }
        (s - 2.0) * 3.0f32.sqrt() // var of sum of 4 U(0,1) = 1/3
    }
}

const FS: f32 = 16000.0;
const N: usize = 160; // 10 ms frames

/// Noise (sigma per frame) plus a 5-harmonic 120 Hz "voiced" burst in frames [b0, b1).
pub fn signal(frames: usize, sigma: impl Fn(usize) -> f32, burst: Option<(usize, usize, f32)>) -> Vec<f32> {
    let mut rng = Noise(0x9E3779B97F4A7C15);
    (0..frames * N).map(|i| {
        let f = i / N;
        let mut x = sigma(f) * rng.next();
        if let Some((b0, b1, amp)) = burst {
            if (b0..b1).contains(&f) {
                let t = i as f32 / FS;
                x += (1..=5).map(|h| amp * (2.0 * std::f32::consts::PI * 120.0 * h as f32 * t).sin() / h as f32).sum::<f32>();
            }
        }
        x
    }).collect()
}

pub fn run(x: &[f32]) -> Vec<(usize, usize)> {
    let mut v = Vad::new(200, 6.0, 3.0, 3, 8);
    for fr in x.chunks_exact(N) { v.push(fr); }
    v.finish();
    v.segments
}

#[cfg(test)]
mod tests {
    use super::*;
    #[test] fn noise_only() { assert!(run(&signal(1000, |_| 100.0, None)).is_empty()); }
    #[test] fn burst_found() { // burst power about 2.9x noise power (5 dB SNR)
        let s = run(&signal(400, |_| 100.0, Some((100, 160, 200.0))));
        assert_eq!(s.len(), 1);
        assert!((s[0].0 as i64 - 100).abs() <= 2 && (s[0].1 as i64 - 160).abs() <= 2, "{s:?}");
    }
    #[test] fn floor_adapts() {
        let s = run(&signal(1400, |f| if f < 400 { 100.0 } else { 1000.0 }, None));
        // a false segment is allowed until the window forgets the quiet frames
        assert!(s.iter().all(|&(a, b)| a < 410 && b <= 400 + 200 + 8 + 2), "{s:?}");
    }
}

Run result: noise only gives no segment; the burst in frames 100-160 is found as [100, 160); after a 20 dB noise step at frame 400 one false segment [400, 599) appears and ends once the window forgets the quiet noise.

Complexity and constants

• Per frame: N multiply-adds, one log10, O(1) amortized deque and counters. Memory: at most win entries of 16 bytes (about 3 KB).
• Latency: min_on frames to confirm an onset, hang frames to confirm an end.
• Measured on this Mac (Apple M4 Pro, one core, release build, 10 min of 16 kHz audio in memory): about 11.5 M frames/s, 87 ns per 160-sample frame (about 0.5 ns/sample), real-time factor about 9·10⁻⁶.
• WebRTC VAD: six sub-band log energies, a 2-Gaussian mixture per band for speech and for noise (vad_core.h), fixed-point.

Efficiency

• Cost: the energy VAD above spends about 87 ns per 10 ms frame on one M4 Pro core (measured here), bound by one pass over the samples. Silero's README states under 1 ms per chunk of 30+ ms on one CPU thread (model about 2 MB); Earshot's README claims a real-time factor of 0.0003 (vendor figure, unchecked). Both are small next to the model they gate.
• Levers:
  • gate the expensive model — saves the silence share of the audio — assistants, batch transcription, always-on devices
  • 30 ms frames instead of 10 ms — a third as many decisions, coarser edges — offline segmentation
  • band-limit to 4 kHz — WebRTC's bands stop there (source) — any speech VAD
  • cascade: energy test first, network only near the threshold — most frames skip it — battery-powered capture
  • fixed-point int16 features, as in WebRTC — no float unit — microcontrollers
• Naive → standard → state of the art:
  • transcribe everything, silence included · full model cost for all audio
  • energy VAD with adaptive floor and hysteresis · about 87 ns per 10 ms frame (measured here)
  • small neural VAD · under 1 ms per 30 ms chunk on a CPU thread (Silero README)

Implementation notes

• Align frames with the codec frame (an Opus frame is 2.5 to 60 ms, RFC 6716) so the flag maps to packets.
• Remove DC offset first (high-pass at tens of Hz).
• Resample to the model's rate before framing (WebRTC: 8/16/32/48 kHz; Silero: 8 or 16 kHz); keep int16 vs float scaling consistent.
• State is per stream.
• Keep a pre-roll buffer so a recorder includes the back-dated start.
• After noise suppression, onsets may be distorted; test both orders.

Pitfalls

• Floor window shorter than a long burst: the minimum rises into the speech and the segment collapses.
• The minimum of a noisy feature is biased low; leave margin in enter_db.
• Noise step up: false segment until the window forgets (third test); a step down is harmless.
• Gain control pumping changes the apparent floor within a word.
• Energy cannot tell speech from music; min_on removes clicks, not sustained sounds.
• Too little hangover splits words; too much merges turns and delays end-of-turn decisions.
• A frame length outside the allowed set is rejected by WebRTC.

Variants and alternatives

In production

Discontinuous transmission in telephony codecs (G.729 Annex B), WebRTC audio pipelines, recognition front ends and voice assistants, diarization pipelines (pyannote), transcription services.

Implementations

Sources

• Jongseo Sohn, Nam Soo Kim, Wonyong Sung, A statistical model-based voice activity detection, IEEE Signal Processing Letters 6(1), 1999, pp. 1-3. paper
• Adil Benyassine, Eyal Shlomot, Huan-Yu Su, Dominique Massaloux, Claude Lamblin, Jean-Pierre Petit, ITU-T Recommendation G.729 Annex B: a silence compression scheme for use with G.729 optimized for V.70 digital simultaneous voice and data applications, IEEE Communications Magazine 35(9), 1997, pp. 64-73. paper
• Fei Jia, Somshubra Majumdar, Boris Ginsburg, MarbleNet: Deep 1D Time-Channel Separable Convolutional Neural Network for Voice Activity Detection, 2020. paper · code
• Hervé Bredin, Antoine Laurent, End-to-end speaker segmentation for overlap-aware resegmentation, 2021. paper · code
• WebRTC project, common_audio/vad. code
• Silero Team, Silero VAD. code