Voice Activity Detection (VAD)

Overview

Voice Activity Detection identifies which parts of an audio signal contain speech versus silence or background noise. This is a critical first step in speaker diarization pipelines.

When to Use

• Preprocessing audio before speaker diarization
• Filtering out silence and noise
• Segmenting audio into speech chunks
• Improving diarization accuracy by focusing on speech regions

Available VAD Tools

1. Silero VAD (Recommended for Short Segments)

Best for: Short audio segments, real-time applications, better detection of brief speech

import torch

# Load Silero VAD model
model, utils = torch.hub.load(
    repo_or_dir='snakers4/silero-vad',
    model='silero_vad',
    force_reload=False,
    onnx=False
)
get_speech_timestamps = utils[0]

# Run VAD
speech_timestamps = get_speech_timestamps(
    waveform[0],  # mono audio waveform
    model,
    threshold=0.6,  # speech probability threshold
    min_speech_duration_ms=350,  # minimum speech segment length
    min_silence_duration_ms=400,  # minimum silence between segments
    sampling_rate=sample_rate
)

# Convert to boundaries format
boundaries = [[ts['start'] / sample_rate, ts['end'] / sample_rate]
              for ts in speech_timestamps]

Advantages:

• Better at detecting short speech segments
• Lower false alarm rate
• Optimized for real-time processing

2. SpeechBrain VAD

Best for: General-purpose VAD, longer audio files

from speechbrain.inference.VAD import VAD

VAD_model = VAD.from_hparams(
    source="speechbrain/vad-crdnn-libriparty",
    savedir="/tmp/speechbrain_vad"
)

# Get speech segments
boundaries = VAD_model.get_speech_segments(audio_path)

Advantages:

• Well-tested and reliable
• Good for longer audio files
• Part of comprehensive SpeechBrain toolkit

3. WebRTC VAD

Best for: Lightweight applications, real-time processing

import webrtcvad

vad = webrtcvad.Vad(2)  # Aggressiveness: 0-3 (higher = more aggressive)

# Process audio frames (must be 10ms, 20ms, or 30ms)
is_speech = vad.is_speech(frame_bytes, sample_rate)

Advantages:

• Very lightweight
• Fast processing
• Good for real-time applications

Postprocessing VAD Boundaries

After VAD, you should postprocess boundaries to:

• Merge close segments
• Remove very short segments
• Smooth boundaries

def postprocess_boundaries(boundaries, min_dur=0.30, merge_gap=0.25):
    """
    boundaries: list of [start_sec, end_sec]
    min_dur: drop segments shorter than this (sec)
    merge_gap: merge segments if silence gap <= this (sec)
    """
    # Sort by start time
    boundaries = sorted(boundaries, key=lambda x: x[0])

    # Remove short segments
    boundaries = [(s, e) for s, e in boundaries if (e - s) >= min_dur]

    # Merge close segments
    merged = [list(boundaries[0])]
    for s, e in boundaries[1:]:
        prev_s, prev_e = merged[-1]
        if s - prev_e <= merge_gap:
            merged[-1][1] = max(prev_e, e)
        else:
            merged.append([s, e])

    return merged

Choosing the Right VAD

| Tool | Best For | Pros | Cons | |------|----------|------|------| | Silero VAD | Short segments, real-time | Better short-segment detection | Requires PyTorch | | SpeechBrain VAD | General purpose | Reliable, well-tested | May miss short segments | | WebRTC VAD | Lightweight apps | Fast, lightweight | Less accurate, requires specific frame sizes |

Common Issues and Solutions

1. Too many false alarms: Increase threshold or min_speech_duration_ms
2. Missing short segments: Use Silero VAD or decrease threshold
3. Over-segmentation: Increase merge_gap in postprocessing
4. Missing speech at boundaries: Decrease min_silence_duration_ms

Integration with Speaker Diarization

VAD boundaries are used to:

1. Extract speech segments for speaker embedding extraction
2. Filter out non-speech regions
3. Improve clustering by focusing on actual speech

# After VAD, extract embeddings only for speech segments
for start, end in vad_boundaries:
    segment_audio = waveform[:, int(start*sr):int(end*sr)]
    embedding = speaker_model.encode_batch(segment_audio)
    # ... continue with clustering