In-Depth Explanation of the Seed-VC Architecture — Decomposing Voice into 'Who, What, and How' in a 4-Stage Structure

📝 Originally published (in Japanese) at forge.workstyle.tech.

Understanding "Voice" — Breaking Down the Components

When trying to convert your recorded voice into someone else's, the first question that comes to mind is:

What exactly is a "voice"?

Even if the same words are spoken, different speakers produce different sounds. Even the same speaker produces different sounds depending on what they're saying. Adding intonation changes it further. In other words, speech is a signal composed of multiple independent pieces of information mixed together — at least: who is speaking (speaker identity), what is being said (content), and how it's being said (prosody).

The core challenge of voice conversion (VC) lies here. If you naively process speech to replace only the speaker identity, the content and intonation often get altered as well. In this article, we'll explain how Seed-VC, which we adopted for the backend of our "voice design" app, solves this problem — by separating and handling information using four modules, while walking through the actual model loading and inference code.

---

Design Philosophy: Delegate Separable Information to Dedicated Modules

At the heart of Seed-VC is the idea: "Don't make a monolithic model do everything." Instead, decompose the components of speech by type and assign each to a dedicated module, then recombine them at the end. The model loading process at startup looks like this:

model, semantic_fn, f0_fn, vocoder_fn, campplus_model, mel_fn, mel_fn_args = load_models(a)

While seven components are returned, they can be grouped into four functional layers:

1. whisper (semantic_fn) — Extracts content (what is being said) as semantic features
2. campplus (campplus_model) — Encodes speaker identity (who is speaking) into a single vector embedding
3. CFM/DiT (model.cfm) — A diffusion model that generates a mel-spectrogram conditioned on the above two
4. BigVGAN (vocoder_fn) — Converts the mel-spectrogram back into an audible waveform

Each role — "extract meaning", "extract speaker", "draw the spectrogram", "convert to sound" — is cleanly separated. This division of labor is why we can replace only the speaker identity while keeping everything else intact. Let's go through each step.

---

Step 1: whisper — Extract Only "What Is Being Said"

For content extraction, we use the encoder of the speech recognition model Whisper. We don't use the decoder (transcription). Instead, we take the intermediate representation (a sequence of semantic features) directly as a feature vector.

s_alt = _semantic(torchaudio.functional.resample(src_t, _S["sr"], 16000))

A key point: the input is resampled to 16kHz before processing. Since Whisper is designed for 16kHz audio, even if the main conversion model operates at 44.1kHz, we always downsample to 16kHz before semantic extraction. The resulting sequence contains little to no information about pitch or timbre — it represents a representation aligned with the spoken content. This is the prerequisite that allows us to freely replace speaker identity later.

Note: Whisper has a limitation — it can only process up to 30 seconds of audio at a time. For longer audio, chunking is required. This constraint itself was a major pitfall (see our other article: "The Culprit Behind the 'Slow Speech' Bug in Voice Conversion Was Whisper's 30-Second Limit").

---

Step 2: campplus — Condense "Who Is Speaking" into a 192-Dimensional Vector

Speaker identity extraction is handled by CAMPPlus, a speaker embedding model. It takes audio, computes fbank features, and outputs a single 192-dimensional vector.

feat = torchaudio.compliance.kaldi.fbank(w16, num_mel_bins=80, dither=0, sample_frequency=16000)
feat = feat - feat.mean(dim=0, keepdim=True)
return _S["campplus_model"](feat.unsqueeze(0)).squeeze().detach().cpu().numpy()

The fact that "speaker identity = a single vector" is crucial to our "voice design" app. Because vectors can be added, blended, or interpolated, we precompute embeddings from clean recordings of 18 speakers as "anchors", then blend them using weighted averages based on slider inputs.

w = design_weights(slider_values, bank, **kw)
emb = (w[:, None] * bank.embeddings).sum(axis=0)

We define sliders for interpretable axes like "age", "pitch", and "huskiness", compute weights w, and take a weighted average of anchor embeddings. This "design by blending voices" operation is only possible because speaker identity is modularized as a standalone vector. If speaker identity were entangled with content, blending would corrupt the spoken words.

---

Step 3: CFM/DiT — Use Diffusion to "Draw" the Mel-Spectrogram

Once we have the semantic features (content) and speaker embedding (identity), we proceed to synthesis. This is where the diffusion model (CFM: Conditional Flow Matching; implemented as DiT) comes in. It starts from random noise and gradually generates a mel-spectrogram conditioned on the inputs.

vt = _S["model"].cfm.inference(cat, torch.LongTensor([cat.size(1)]).to(dev),
                               mel2, style, None, STEPS, inference_cfg_rate=CFG)
vt = vt[:, :, mel2.size(-1):]

Looking at the arguments reveals how this stage integrates outputs from previous modules:

• cat — Semantic features (content). Concatenation of prompt audio (pc) and target condition (cond)
• style — Speaker embedding from Step 2 (= the blended design embedding)
• mel2 — Mel-spectrogram of the prompt audio. After generation, we trim the first mel2.size(-1) frames to keep only the new content
• STEPS / inference_cfg_rate — Number of diffusion steps and strength of classifier-free guidance

The model receives content (cat) and speaker identity (style) as separate arguments. It interprets them as instructions: "Draw this content in this speaker's voice." If you fix one and change the other, only the intended attribute changes. The separation design is reflected directly in the inference interface.

---

Handling Prosody: length_regulator and F0

"How it's being said" — prosody (tempo and pitch movement) is a bit different. It cannot be fully disentangled from content, so instead of a dedicated module, it's handled as a connection point between content and speaker identity.

First, tempo (duration). The semantic sequence from Whisper needs to be stretched to match the desired number of mel frames. This is handled by length_regulator.

cond, *_ = _S["model"].length_regulator(s_alt, ylens=torch.LongTensor([mel.size(2)]).to(dev),
                                        n_quantizers=3, f0=None)

By passing the target frame length (ylens), the semantic sequence is adjusted to the desired duration. Next, pitch (F0). In a 44.1kHz F0-conditional model, length_regulator can accept an f0 argument, enabling prosody control such as shifting the carrier voice's pitch range toward the target speaker's range.

lf[F0_alt > 1] = lf[F0_alt > 1] - m_alt + m_ori   # Shift carrier F0 to target median
shifted = torch.exp(lf)
cond, *_ = _S["model"].length_regulator(S_alt, ylens=tgt_len, n_quantizers=3, f0=shifted)

This is a simple operation: shifting in the log domain by the difference in medians. Prosody isn't treated as a separate module but as an injected parameter. This reflects a pragmatic compromise — attempting full disentanglement can lead to unnatural results.

---

Step 4: BigVGAN — Convert Mel Back to Audible Sound

The diffusion model generates a mel-spectrogram — a kind of "blueprint" of sound. The final step, converting it into a waveform humans can hear, is handled by the vocoder: BigVGAN.

wav = _S["vocoder_fn"](vt.float()).squeeze().detach().cpu().float().numpy().reshape(-1)

Though it's just one line, this step greatly impacts audio quality. Even with the same mel-spectrogram, different vocoders produce different sounds. Our 44.1kHz model can output "true broadband" because BigVGAN is trained at high sampling rates (see our other article: "22.05kHz vs 44.1kHz").

---

Pitfalls and Lessons Learned

Here are some key lessons from implementing and operating this four-layer architecture:

• Sampling rates differ per layer. Semantic extraction is fixed at 16kHz, the conversion model operates at 22.05kHz or 44.1kHz, and speaker embeddings use 16kHz fbank. Misunderstanding which layer expects which rate silently degrades quality. Explicitly resampling at each stage is safest.
• Speaker identity as a single vector unlocks applications. If speaker identity were embedded inside the diffusion model, we couldn't implement "voice blending". Modular separation isn't just about quality — it directly enhances product expressiveness.
• Prosody cannot be fully separated — design accordingly. Tempo and F0 are tied to content, so treating them as conditional parameters injected via length_regulator is the pragmatic solution. Forcing full disentanglement often leads to unnatural artifacts.
• Each layer is replaceable. The vocoder can be swapped between BigVGAN, HiFiGAN, or Vocos. The semantic extractor can be swapped between Whisper and CNHuBERT. The modular design grants extensibility — each stage can be upgraded independently.

---

Summary

• Seed-VC's voice conversion uses a four-layer architecture: whisper (semantic) + campplus (speaker) + CFM/DiT (diffusion mel generation) + BigVGAN (vocoder)
• The design philosophy is to decompose speech into "who, what, and how", assigning each to a dedicated module
• By representing speaker identity as a single 192-dimensional vector, we enable blending multiple speakers — the core of our "voice design" app
• The diffusion model receives content (cat) and speaker identity (style) as separate arguments — the separation design is reflected in the inference interface
• Prosody (tempo and F0) is not fully disentangled; instead, it's injected as a conditional parameter via length_regulator
• Each layer operates at different sampling rates and roles, and is replaceable. This modularity improves quality, expressiveness, and extensibility.