The Culprit Behind the 'Slow Speech' Bug in Voice Conversion Was Whisper's 30-Second Limit

📝 Originally published (in Japanese) at forge.workstyle.tech.

While building a voice conversion app, I received a report like this:

"The recording is normal, but the converted voice speaks way too slowly."

The input audio was at a natural pace, but the conversion result sounded drawn out, almost like the speaker was drunk. Interestingly, this didn't happen with short clips—only with long recordings. This article is a record of my investigation, which revealed that the cause wasn't the conversion model or the audio format, but the "30-second limit" of Whisper, which I was using for semantic extraction.

Anyone using Whisper as a feature extractor in a pipeline (voice conversion, TTS, lip-sync, subtitle generation, etc.) can fall into this same trap.

Premise: The Pipeline Structure

I was working on a voice conversion system based on Seed-VC. Roughly speaking, it decomposes audio into "who is speaking" (speaker identity) and "what is being said" (content), replaces the speaker identity, and resynthesizes the audio. In this process, I used the Whisper encoder to extract the speech content.

The flow was as follows:

1. Pass the input audio (carrier) through Whisper to obtain a sequence of semantic features representing the content.
2. Adjust that sequence to match the target length (number of frames in the mel-spectrogram).
3. Resynthesize the audio using a diffusion model combined with the speaker embedding.

The problem existed at the boundary between Step 1 and Step 2.

Observation: Slowness Proportional to Audio Length

First, I started isolating the cause. There are several typical reasons why audio might "become slow":

• Sampling Rate Mismatch: Playing 44.1kHz audio as 22.05kHz results in half-speed, low-pitched playback.
• Resampling Errors: An unexpected rate conversion happening somewhere in the chain.
• Time-Stretch Bug: A DSP error in the speaking rate adjustment.

However, after checking the output files, the sampling rate was correct and speech rate adjustment was off. Despite this, it was still slow. The deciding factor was that the longer the input, the slower the output became.

Here was the specific case I reproduced:

• Input (recording): approx. 131 seconds
• Output: Significantly more drawn out than the input; perceptually over 4x slower.

Short audio (a few seconds) had no issues at all. The fact that it "worsened in proportion to length" pointed directly to the root cause.

The Root Cause: Whisper Can Only "Hear" 30 Seconds at a Time

The Whisper encoder processes input by fixing it to a 30-second log-mel spectrogram. If you pass audio longer than 30 seconds, only the content of the first 30 seconds is captured as features (the rest is discarded).

Now, recall Step 2 of the pipeline: "Adjust to match the target length." The length regulator stretches the semantic sequence obtained from Whisper to match the total length of the mel-spectrogram.

So, when a 131-second recording was passed:

1. Whisper returned only the content for the first 30 seconds.
2. The length regulator stretched those 30 seconds to fit a 131-second duration.
3. Result: The content of 30 seconds is played over 131 seconds = approx. 4.4x slower.

The perceived "over 4x slower" matched perfectly with $131 \div 30 \approx 4.4$. The culprit wasn't model quality or audio formatting; it was the fact that semantic information was being stretched the moment the input exceeded 30 seconds.

It makes sense that this didn't happen with short audio, as there is no truncation or stretching within the 30-second window.

Solution: Extraction via Overlapping 30-Second Chunks

The standard solution is to split long audio into 30-second chunks, pass each through Whisper, and concatenate the results. However, simply cutting the audio every 30 seconds causes the context to break at the boundaries, leading to unnatural results. To fix this, I overlapped the chunks by several seconds and discarded the redundant portions during concatenation.

The original Seed-VC inference code follows this logic, so I modeled my implementation after it. Here is the semantic extraction helper I implemented:

def _semantic(w16):
    """Extracts semantic features. For long audio (>30s), splits into 30s overlapping chunks and concatenates.

    The whisper encoder can only handle a maximum of 30s at once. Passing long audio directly
    results in only the first 30s being captured. Stretching this to the original length
    causes "slow speech" output, so we process in 30s chunks (with 5s overlap) and join them.
    """
    sfn = _S["semantic_fn"]
    if w16.size(-1) <= 16000 * 30:          # Pass through if 30s or less
        return sfn(w16)
    ov = 5  # seconds (overlap between chunks)
    parts, buf, t = [], None, 0
    while t < w16.size(-1):
        chunk = w16[:, t:t + 16000 * 30] if buf is None else \
            torch.cat([buf, w16[:, t:t + 16000 * (30 - ov)]], dim=-1)
        s = sfn(chunk)
        parts.append(s if t == 0 else s[:, 50 * ov:])  # whisper ~50 tokens/s; remove overlap
        buf = chunk[:, -16000 * ov:]
        t += 30 * 16000 if t == 0 else chunk.size(-1) - 16000 * ov
    return torch.cat(parts, dim=1)

A few key points:

• Audio is handled at 16kHz (Whisper's input rate), so 16000 * 30 is the number of samples for 30 seconds.
• Whisper's semantic sequences are roughly 50 tokens per second. I remove the 5-second overlap (50 * 5 tokens) from the beginning of the second chunk onwards.
• Inputs under 30 seconds are passed through as before, so the behavior for short audio remains unchanged.

After this, I simply replaced all calls to semantic extraction (condition generation from carrier, F0-conditioned paths, etc.) with this _semantic() function.

Verification: 1.00x Ratio for 131s Recording

After the fix, I converted a 40-second segment taken from the problematic 131-second recording and measured the ratio between input and output length.

• Input: 40.00 seconds
• Output: 40.00 seconds (Ratio: 1.00)

Cases that would have been slowed down before are now generated at the correct length. Audio under 30 seconds continues to work as expected.

Lesson: Pitfalls of Using Whisper as a "Feature Extractor"

The lesson here is particularly relevant when integrating Whisper for purposes other than transcription—specifically as a feature extractor.

• 30 seconds is not a "crash boundary," but a "silent truncation boundary." No exceptions or warnings are thrown; it simply uses the first 30 seconds. This makes it very hard to notice.
• Symptoms appear downstream. In this case, the symptom looked like a DSP issue ("slow audio"), but the root cause was feature extraction three steps upstream. I was able to rule out "likely suspects" like sampling rates and time-stretching because I noticed the degradation was proportional to length.
• Handle long-form audio yourself. While the transformers pipeline has chunking features for long text, if you are using the encoder output directly, you must implement your own chunking, concatenation, and overlap logic.

Whenever you're tempted to blame "poor model quality," first check for implicit limitations in preprocessing or feature extraction. Instead of applying a quick fix to hide the symptoms, look for patterns (like proportionality) to find the root cause. It's a tedious process, but it's the fastest way to a real solution.

Summary

• The cause of the "slow speech" bug in voice conversion was the Whisper encoder's 30-second limit.
• For long audio, only the first 30 seconds of meaning were captured and then stretched to the full length, causing the slow-motion effect (e.g., 131s $\approx$ 4.4x slower).
• Fixed by implementing 30-second chunking (with 5s overlap) and concatenation, removing overlap at the token level.
• Verification showed a correction from slowed-down audio to a 1.00 ratio (40s input $\to$ 40s output).
• When using Whisper as a feature extractor, assume it will silently truncate at 30 seconds and prepare your own handling for long-form audio.