DEV Community

Cover image for Solving Silent Label Drift in Speech AI & ASR Datasets: Announcing driftcheck-audio on PyPI
Boobalan Arjunan
Boobalan Arjunan

Posted on

Solving Silent Label Drift in Speech AI & ASR Datasets: Announcing driftcheck-audio on PyPI

๐Ÿš€ Excited to announce the official release of driftcheck-audio (v0.1.0) on PyPI!

If you train Speech-to-Text (ASR) models (Whisper, Conformer, Kaldi) or curate speech datasets, you've hit this silent evaluation killer:

๐Ÿ‘‰ Label drift and misaligned timestamps.

Speech datasets pair audio recordings with timed text (SRT, WebVTT, Lhotse manifests, Audacity labels). When these timestamps slipโ€”due to frame-rate mismatches (25 vs 23.976 fps), edits, or sync errorsโ€”models silently train on pure noise or the wrong words. No exception is raised; Word Error Rate (WER) degrades silently.

driftcheck-audio solves this: an open-source Python library and CLI that audits timed labels against speech audio with mathematical precisionโ€”without needing a heavy speech model.

๐Ÿ” What Does driftcheck-audio Check?
โฑ๏ธ Constant Offsets: Finds whether labels are early or late by milliseconds or seconds and provides the exact time-shift value to correct them.
๐Ÿ“ˆ Steady Drift: Detects offsets that grow over time (e.g. video frame-rate mix-ups), reports the drift rate (ms/min), and calculates the stretch multiplier.
โšก Sudden Jumps: Identifies mid-stream timing discontinuities caused by inserted ads, cut audio, or spliced takes.
๐ŸŽ™๏ธ Unlabelled Speech & Phantom Labels: Quantifies speech omitted from labels and flags labels mapped over dead silence.
๐ŸŽฒ Null Permutation Testing: Compares every measurement against shuffled null labels to eliminate false positives and coincidental alignments.
๐Ÿ“Š Real-Data Benchmarks (LibriSpeech dev-clean)

Validated across multi-speaker recordings from LibriSpeech dev-clean:

โœ… Undamaged Labels: 0 failures, median measured offset of 0.001 s (1 ms). โœ… Planted Offsets: 6/6 (100%) recovered across ยฑ0.2s to +2.0s shifts (median error 10โ€“39 ms). โœ… Drift Detection: 100% caught on 1.004 stretch with slope accuracy of 0.014 s/min. โœ… False Positive Rejection: 0/6 false alignments when evaluated against unrelated text. โœ… Test Coverage: 110 / 110 unit tests passing.

โšก Lightweight & Fast (Zero Heavy ML Bloat)

Requires no PyTorch, no Whisper weights, and zero GPU compute. Built purely on speech energy envelopes, cross-correlation, and NumPy/SciPy/SoundFile for instant, sub-second execution on any CPU or CI/CD runner.

๐Ÿ“ฆ Quick Start & Links

Install directly from PyPI: pip install driftcheck-audio

Run an automated audit on your dataset in 1 command: driftcheck check talks/ --out report/

Supports SRT, WebVTT, CSV, Audacity .tsv, and Lhotse .jsonl.gz supervisions!

๐Ÿ”— PyPI: https://pypi.org/project/driftcheck-audio/
๐Ÿ’ป GitHub: https://github.com/BoobalanArjunan/Driftcheck

If you curate ASR datasets or build speech pipelines, give it a spin! โญ

SpeechAI #ASR #AudioML #MachineLearning #DeepLearning #Whisper #SpeechRecognition #Python #OpenSource #DataQuality #AudioEngineering #AI

Top comments (0)