This post walks the full audio-AI pipeline —
denoising → vocal/instrumental separation → voice activity detection → automatic speech recognition (ASR) → speaker diarization → subtitles/summaries/translation
— from an engineering point of view. Using DeVoice as a working example, you'll get a production-minded workflow plus runnable Python and ffmpeg code for meeting notes, short-form captions, podcast repurposing, and call-center QA.
1. Why audio has to become "structured"
In most systems, audio and video are unstructured data. A one-hour meeting recording, a 40-minute podcast, or a product walkthrough video is, to a program, just a stream of samples. You can't run a SELECT on it, you can't search for a keyword, you can't locate "who said what at which minute," and you can't feed it directly into retrieval, analytics, or an LLM Q&A pipeline.
Turning audio into structured data is a kind of modality reduction:
waveform/video → clean vocals → timestamped text → speaker-labeled document → summary/translation/knowledge graph
Once that conversion exists, you can:
Search hundreds of hours of recordings and jump to the exact moment a decision was made;
Repurpose one video into captions, a blog post, podcast show notes, and social clips;
Analyze support calls for compliance, review sales talk tracks, and extract action items from meetings;
Improve accessibility and reach with automatic captions and cross-language translation for global audiences.
The full pipeline looks like this:
flowchart LR
A[Raw audio/video<br/>MP3/WAV/MP4/M4A...] --> B[Preprocessing<br/>resample/loudness/split]
B --> C[AI denoising<br/>speech vs non-speech]
C --> D[Vocal/instrumental<br/>separation]
D --> E[VAD voice activity]
E --> F[ASR acoustic model<br/>+ word timestamps]
F --> G[Speaker diarization]
G --> H[Punctuation / ITN / language ID]
H --> I[Structured output<br/>SRT/VTT/TXT/DOCX/CSV]
I --> J[Post-processing<br/>translation/summary/AI chat/mind map]Let's go through it stage by stage.
2. Denoising first: why "clean before you recognize" is non-negotiable
2.1 Noise is the #1 killer of ASR accuracy
Speech recognition quality is typically measured with WER (Word Error Rate). A model that performs well on clean audio degrades sharply when it meets keyboard clicks, air-conditioner hum, room reverb, outdoor wind, or background chatter. The reason is simple: the model has to distinguish phonemes in a spectrogram where speech and non-speech overlap. Noise masks consonants, shifts formants, and produces plenty of "hallucinated" words.
That's why the standard engineering practice is speech enhancement before recognition.
2.2 Classic methods: the limits of spectral subtraction and Wiener filtering
Early denoising worked in the frequency domain by "subtraction." Take spectral subtraction:
Grab a snippet of pure noise and estimate its average power spectrum
|N(ω)|²;Run STFT on the noisy signal to get
|Y(ω)|²;Estimate clean speech with
|Ŝ(ω)|² = |Y(ω)|² − α|N(ω)|², then reconstruct using the original phase.
The drawbacks are obvious:
It assumes stationary noise. Air-conditioner hum is fine, but non-stationary noise — keyboards, traffic, passers-by — can't be represented by a single noise sample;
Subtract too much and you damage the voice, producing a muffled, watery "flanging" artifact;
Subtract too little and you get "musical noise" — isolated frame-level spectral blobs that sound like electronic chirping.
Wiener filters and statistical-model methods are smoother but still rely on estimating a fixed noise floor and subtracting it, which fails whenever speech and noise overlap heavily.
2.3 Deep learning: from "subtracting" to "separating"
The modern mindset shifts the problem entirely — instead of estimating noise, the model learns what speech is and explicitly splits the signal into two streams:
a voice stream;
an everything-else stream (noise, reverb, background music).
Two dominant technical routes:
1) Time-frequency masking. On the STFT spectrogram, the model predicts a mask (e.g., an Ideal Ratio Mask, IRM) that decides, per time-frequency bin, how much of the energy belongs to speech, then applies the mask to recover clean vocals. U-Net-style encoder-decoder networks with skip connections are a common architecture.
2) Waveform-domain separation. Waveform-domain U-Nets such as Demucs separate signals end-to-end directly on the waveform, sidestepping the hard phase-estimation problem; they work particularly well for music source separation.
The key advantage is that these methods act on speech characteristics rather than a fixed noise fingerprint, so they handle non-stationary noise (wind, reverb, crowds, keyboards, traffic) while preserving the naturalness of the voice.
2.4 Vocal isolation: a "superset" of denoising
When you're dealing with songs, videos with background music, or live recordings, the need upgrades from denoising to source separation: split the vocals (acapella) from the instrumental. It shares the same underlying target-source-separation engine, just with finer-grained targets.
The instrumental can be used for karaoke, remixes, and sampling; the clean vocals feed covers, dubbing, re-mixing, or transcription.
2.5 Doing this step with DeVoice
The DeVoice Background Noise Remover combines three things into one in-browser workbench (it runs in the browser, requires no install, and works on Windows/macOS/iOS/Android):
AI background denoising — removes hiss, hum, wind, echo/reverb, crowd noise, keyboard clicks, traffic, and background music;
Vocal/instrumental separation — outputs acapella and instrumental stems for karaoke, remixes, and sampling;
One-click transcription — send the clean vocals straight to ASR without switching tools.
It accepts MP3 / WAV / M4A / FLAC / MP4; the guest free tier handles clips up to 5 minutes. The workflow is three steps: upload → AI processing → A/B preview and download.
Engineering note: placing denoising before ASR is a near-zero-cost optimization that meaningfully lowers downstream WER. The payoff is largest for high-noise recordings — phone footage shot outdoors, home recordings, and omnidirectional conference-room mics.
3. ASR internals: what happens between waveform and text
After denoising, we enter automatic speech recognition. A modern ASR system typically contains these modules.
3.1 Voice Activity Detection (VAD)
VAD decides which segments contain speech, cutting out silence and long pauses. Common implementations include the lightweight WebRTC VAD and the neural Silero VAD. It helps by:
reducing wasted computation (silence never reaches the engine);
lowering the memory pressure of long audio;
providing boundaries for chunking and speaker segmentation.
3.2 Feature extraction
The raw waveform (e.g., 16 kHz, 16-bit) isn't fed directly to the network. It's first converted into acoustic features:
framing (e.g., 25 ms frames, 10 ms hop) with a window function;
FFT/STFT to obtain the spectrogram;
a Mel filter bank and logarithm, yielding log-mel filterbank (fbank) features, often 80 dimensions.
The Mel scale mimics the human ear's greater sensitivity to low frequencies and captures the spectral envelope compactly.
3.3 Acoustic model and decoding
The mainstream end-to-end families include:
CTC, which introduces a blank label to solve frame-to-label alignment — fast to train and convenient for forced alignment;
RNN-Transducer (RNN-T), which naturally supports streaming (encoder + prediction network + joint network) and powers many online real-time transcription services;
Attention encoder-decoder / Conformer / Whisper-style models. Conformer combines CNN local modeling with Transformer global attention and excels on long, multilingual audio; large multilingual models such as Whisper fold recognition, language identification, and translation into a single framework.
Post-decoding steps include:
punctuation restoration (raw output often has none);
ITN (inverse text normalization), converting spoken forms into written numbers, dates, and currency;
capitalization and sentence segmentation.
3.4 Word-level timestamps
Text alone isn't enough for captions and navigation — you need a timeline. Timestamps usually come from CTC/attention forced alignment, mapping each token back to a frame and then to milliseconds. DeVoice provides word-level millisecond timestamps, meaning each word gets precise start/end boundaries rather than a rough sentence-level range. That's essential for generating SRT/VTT, building keyword jump-to-time, and rendering highlighted subtitles.
3.5 Multilingual identification
Global products handle 100+ languages. In practice, an LID (Language Identification) step first detects the language, then routes the audio to the right multilingual model branch. DeVoice auto-detects 100+ languages and can translate after recognition, which is especially useful for YouTube/TikTok content aimed at overseas audiences and for cross-border meetings.
4. Speaker diarization: answering "who said this"
Meetings, interviews, and support calls have multiple speakers. Diarization determines who speaks when, following a typical pipeline:
Segmentation — cut the audio into short, single-speaker segments;
Speaker embeddings — encode each segment into a vector that represents a speaker's voice, such as an x-vector;
Clustering — group similar vectors into clusters, each representing one speaker (Speaker 1, Speaker 2, …);
Alignment with ASR — tag each recognized utterance with its speaker.
DeVoice transcription offers optional speaker diarization; when enabled, the exported transcript carries Speaker 1 / Speaker 2 labels — a must-have for meeting notes, interviews, and call-center QA.
5. Post-processing: from raw transcript to a usable asset
The verbatim transcript is only the starting point. The real time savings come afterward:
Subtitles — SRT and VTT carry timelines and can be muxed or attached directly to video;
Multi-format export — TXT for reading, DOCX for editing, CSV with timestamps/speakers for programmatic analysis;
AI summaries — meeting minutes, key points, and action items by template;
Translation — multilingual subtitles and transcripts;
AI chat — ask questions over the whole transcript ("What was the customer's core complaint?");
Mind maps — structure long content into a tree outline for review.
On the results page, DeVoice lets you translate, summarize, chat with, and build a mind map from the transcript, then export SRT / VTT / TXT / DOCX / CSV as needed.
6. A hands-on DeVoice workflow
6.1 Denoising / vocal separation (3 steps)
Open the Background Noise Remover and upload a local file or video (MP3/WAV/M4A/FLAC/MP4);
Wait for AI processing — it denoises automatically and can split vocals from the instrumental on demand;
Use the before/after preview to confirm the voice stays natural (no watery artifacts), download the clean audio, or continue straight to transcription.
6.2 Audio-to-text (with batching)
Open Audio to Text and upload a file or paste a video/audio URL;
Process up to 5 tasks in parallel when batch uploading;
Configure options: speaker diarization on/off, AI summary type, and language (auto-detect available);
After recognition, translate, summarize, chat with, or build a mind map from the transcript;
Export SRT / VTT / TXT / DOCX / CSV for your use case.
Everything runs in the browser, so there's no heavy local software to install and no GPU environment to configure.
7. Developer use cases with runnable code
Note: DeVoice is an in-browser online tool that runs up to 5 tasks in parallel. The code below targets the two steps developers can actually automate: ① pre-upload preprocessing with ffmpeg, and ② batch post-processing of the exported SRT/CSV artifacts. Everything is Python 3 and ready to run.
7.1 Use case 1: automatic captions for short video / courses
Pain point: manually transcribing and timing captions takes several times the length of the video, and timelines are error-prone.
Solution: video → DeVoice transcription (with word timestamps) → export SRT/VTT → attach to the video or player.
(1) ffmpeg preprocessing into an ASR-friendly format
# Extract the audio track and convert to 16kHz mono 16-bit PCM (a common ASR input)
ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le output.wav
# Loudness normalization (EBU R128) to reduce recognition variance from level differences
ffmpeg -i input.wav -af loudnorm=I=-16:LRA=11:TP=-1.5 output_norm.wav
# Split long audio into 10-minute chunks for parallel upload (-c copy avoids re-encoding)
ffmpeg -i output_norm.wav -f segment -segment_time 600 -c copy chunk_%03d.wav
(2) Parse SRT and convert to VTT / plain text in Python
import re, csv
from pathlib import Path
def parse_srt(path):
"""Parse SRT -> [(index, start_ms, end_ms, text), ...]"""
raw = Path(path).read_text(encoding="utf-8-sig")
blocks = re.split(r"\n\s*\n", raw.strip())
items = []
for b in blocks:
lines = [l for l in b.strip().splitlines() if l.strip()]
if len(lines) < 3:
continue
idx = lines[0].strip()
m = re.match(r"(\d+):(\d+):(\d+)[,.](\d+)\s*-->\s*(\d+):(\d+):(\d+)[,.](\d+)", lines[1])
if not m:
continue
g = list(map(int, m.groups()))
start = ((g[0]*60 + g[1])*60 + g[2])*1000 + g[3]
end = ((g[4]*60 + g[5])*60 + g[6])*1000 + g[7]
items.append((idx, start, end, " ".join(lines[2:])))
return items
def ms_to_vtt(ms):
h, ms = divmod(ms, 3600000)
m, ms = divmod(ms, 60000)
s, ms = divmod(ms, 1000)
return f"{h:02d}:{m:02d}:{s:02d}.{ms:03d}"
def srt_to_vtt(items):
out = ["WEBVTT", ""]
for _, s, e, text in items:
out.append(f"{ms_to_vtt(s)} --> {ms_to_vtt(e)}")
out.append(text)
out.append("")
return "\n".join(out)
def find_keyword(items, keyword):
"""Locate a keyword: return every timestamp (seconds) where it appears."""
hits = []
for _, s, e, text in items:
if keyword.lower() in text.lower():
hits.append((round(s/1000, 2), text))
return hits
if __name__ == "__main__":
items = parse_srt("demo.srt")
Path("demo.vtt").write_text(srt_to_vtt(items), encoding="utf-8")
Path("demo.txt").write_text("\n".join(t for *_, t in items), encoding="utf-8")
print("subtitle cues:", len(items))
print("'pricing' appears at:", find_keyword(items, "pricing"))
With millisecond timestamps you can also make clickable captions that seek the video, auto-cut clips around keywords, or burn in highlighted subtitles.
7.2 Use case 2: meeting / interview notes
Pain point: a one-hour meeting takes 30+ minutes to turn into notes, and you can't tell who owned which action item.
Solution: enable speaker diarization → export the speaker-labeled transcript → use AI summaries for decisions and action items → export DOCX to circulate.
After exporting CSV, you can aggregate speaking time per speaker to understand the meeting's structure:
import csv, re
from collections import Counter
def load_transcript_csv(path):
"""Read a transcript CSV with start/end/speaker/text columns."""
rows = []
with open(path, encoding="utf-8-sig", newline="") as f:
for r in csv.DictReader(f):
rows.append(r)
return rows
def speaker_stats(rows):
speak_time, speak_words = Counter(), Counter()
for r in rows:
spk = r.get("speaker") or r.get("Speaker") or "Unknown"
try:
dur = float(r.get("end", 0)) - float(r.get("start", 0))
except ValueError:
dur = 0
speak_time[spk] += dur / 1000.0 # ms -> seconds
speak_words[spk] += len(re.findall(r"\w+", r.get("text", "")))
return speak_time, speak_words
if __name__ == "__main__":
rows = load_transcript_csv("meeting.csv")
st, sw = speaker_stats(rows)
for spk in st:
print(f"{spk}: spoke {st[spk]:.0f}s, {sw[spk]} words")
Combined with DeVoice's AI summaries, you get a structured "decisions / action items / owners / deadlines" note without re-watching the recording.
7.3 Use case 3: call-center / sales QA
Pain point: QA teams only sample-listen to calls, so coverage is low, subjective, and delayed.
Solution: batch transcription → diarization to separate agent and customer → keyword and talk-track detection → warehouse for full-population QA.
The web app runs up to 5 tasks in parallel, and locally you can batch-process the exported files with a thread pool using the same concurrency:
import json
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
# Align with the web app's "up to 5 tasks in parallel" to avoid over-subscription
MAX_WORKERS = 5
def analyze_one(srt_path):
"""QA for a single call: check compliance phrases and risk terms."""
items = parse_srt(srt_path) # reuse the parser from 7.1
full = " ".join(t for *_, t in items)
rules = {
"recording_disclosure": ["this call is recorded", "calls are recorded"],
"risk_terms": ["refund", "complaint", "lawyer", "cancel"],
}
result = {"file": srt_path.name, "duration_sec": round(items[-1][2]/1000, 1)}
for k, kws in rules.items():
result[k] = [w for w in kws if w.lower() in full.lower()]
return result
def batch(folder):
files = sorted(Path(folder).glob("*.srt"))
with ThreadPoolExecutor(max_workers=MAX_WORKERS) as ex:
return list(ex.map(analyze_one, files))
if __name__ == "__main__":
report = batch("calls")
Path("qa_report.json").write_text(
json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
print(f"QA complete for {len(report)} calls")
7.4 Use case 4: repurposing podcasts / global content
From a single English podcast or YouTube video, DeVoice can run denoising → transcription → translation → summary in one pass, then export:
SRT/VTT: multilingual captions that improve accessibility and watch time;
TXT/DOCX: rewritten into a blog post or article;
Mind map: chapter structure for show notes and chapter markers.
One source asset multiplies into captions, articles, short-video scripts, and social posts — the standard "produce once, distribute everywhere" approach.
8. Engineering pitfalls and optimization tips
Denoising isn't "more is better." Over-processing creates watery artifacts and swallows breath sounds and weak syllables, hurting naturalness and ASR. Always A/B-test by ear and keep natural breathing.
Leave headroom while recording. Clipping distortion cannot be fixed in post; keep peaks around -6 dB.
Normalize the sample rate. 16 kHz mono is a common ASR input — convert with ffmpeg before upload to avoid resampling issues.
Split long audio before parallelizing. Chunk at VAD boundaries or fixed durations to exploit the 5-task parallelism and to re-upload only one segment if something fails.
Apply a tolerance when merging chunk timestamps. Add each chunk's global start offset when concatenating subtitles, or the timeline will drift.
Break captions along semantic boundaries. Word timestamps give points; readable captions need re-wrapping by punctuation and line length so no line is too long.
Confirm the language for mixed speech. Code-switched audio with jargon benefits from auto-detection plus a manual language check.
Expect some diarization error. Overlapping speech, fast turns, and far-field voices cause segmentation mistakes; correct speaker labels manually for high-stakes transcripts.
9. Copyright and compliance
When separating vocals, removing vocals, or downloading online audio/video, make sure you own the content or have authorization; stay within fair use, licensing, and permitted derivative use.
When transcribing and QA-ing support calls or meetings, honor local recording-consent laws (e.g., "this call is being recorded") and handle personal data with storage controls and redaction.
Review the data-handling and privacy policy of any audio tool; evaluate sensitive recordings before sending them to a cloud service.
10. Wrapping up
The real engineering value of audio AI isn't the single "speech-to-text" step — it's connecting the whole pipeline. Deep-learning source separation cleanly strips vocals from noise and background music; modern ASR produces multilingual text with word-level timestamps; diarization recovers who spoke when; and captions, summaries, translation, and chat turn unstructured audio/video into searchable, analyzable, redistributable data assets.
For individual developers and small teams, you don't have to train models or deploy GPU services yourself. An in-browser tool like DeVoice bundles denoising, vocal separation, transcription, diarization, multilingual support, summarization, and multi-format export. As a developer, you only own the two ends — preprocessing and post-processing of the artifacts — to cheaply build meeting-notes, auto-captioning, and call-QA applications.
If you're building anything around audio/video, start by turning your next recording into a structured document with timestamps and speaker labels, and run the pipeline yourself.
Links:
AI audio to text: https://devoice.io/audio-to-text
Online background noise remover / vocal isolation: https://devoice.io/remove-background-noise
Top comments (0)