Learning from Failure: Why Couldn't 'Breathiness' Be an Independent Parameter?

📝 Originally published (in Japanese) at forge.workstyle.tech.

In my "Voice Design" app, I added several sliders to adjust voice characteristics. Implementing things like speech rate and pitch fluctuation was straightforward. Then, I added one more: "Breathiness." This parameter was intended to allow users to add "whisper-like" sweetness or expressive breathiness to a voice.

To get straight to the point: I failed to implement this slider. I tried almost every method imaginable—additive noise, band-limited noise, cross-synthesis, increasing aperiodicity in WORLD, additive difference, high-pass filtering, cross-fading with whispers, and finally, generating it via the model itself—but nothing could achieve a "natural breathiness that doesn't sound like noise." Ultimately, I removed the slider from the UI entirely.

Why did I fail? The reason was right in front of me from the beginning, but it took nearly ten rounds of trial and error to put into words: Breathiness (breathy voice) is physically inseparable from noise. This article is an honest, chronological record of that defeat. Since my git commit logs serve as a history of these failures, I will follow them.

Premise: What is a "Breathy Voice"?

First, let's clarify what "breathiness" or "breathy voice" is acoustically. Voiced sounds are produced by the periodic vibration of the vocal folds (pitch). In a breathy voice, the glottis does not close completely, resulting in turbulent noise from airflow mixed with the periodic vibration. A whisper is the extreme version of this, where the periodic component almost disappears, leaving only turbulent noise.

In short, the essence of breathiness is the turbulent noise itself. This single sentence foreshadows the entire conclusion of this article. However, at the time, I wasn't ready to accept that. I thought, "If I shape it correctly, I should be able to create an elegant breathiness that doesn't sound like noise."

Records of Attempts

1. Noise Addition → High-Frequency Boost (84cf63a)

My first attempt was adding Gaussian noise. Naturally, this just sounds like hiss. So, I changed tactics: instead of adding noise, I tried simply slightly boosting the high frequencies.

sos = butter(2, min(3000.0, sr / 2 * 0.9) / (sr / 2), btype="high", output="sos")
hf = sosfilt(sos, wav).astype(np.float32)
return wav + hf * (0.22 * amt)

The "hissing" sensation disappeared, but the sound just became brighter; it didn't actually sound "breathy." It wasn't "breathiness"; it was just an "equalizer."

2. Band-limited (2–8kHz) Breath Noise (8ea3279)

High-frequency boosting wasn't working. So, I tried to mimic the airflow turbulence directly: applying band-limited noise (2–8kHz) shaped by the volume envelope, applied only to the voiced sections. This prevented noise from appearing during silent sections, avoiding the "constant hiss" problem.

env = np.abs(wav).astype(np.float32)          # Volume envelope (breath follows the speech)
...
noise = sosfilt(lp, sosfilt(hp, noise))       # Band-limited to 2–8kHz
breath = noise * env * (0.35 * amt)

The ratio of high-frequency components (>2kHz) in the same audio went from 2.1% to 5.8%, so the "breath component" did increase. The metrics moved. But to the ear, it still sounded like noise on a separate layer from the voice.

3. Cross-Synthesis: Whispers shaped by the Voice Envelope (d10f696)

I hypothesized that the noise sounded like a "separate layer" because it wasn't passing through the vocal tract (formants). I switched to a cross-synthesis method: shaping the noise using the short-time spectral envelope of the speech to create a whisper component of that specific voice, then mixing it in. Theoretically, if it passes through the vocal tract resonance, it should sound like "breath mixed with the voice" rather than noise. The logic was sound, but turning up the intensity still left a heavy hiss.

4. Increasing WORLD's Aperiodicity (AP) (438ff68)

I decided to change direction completely. Instead of "adding" noise, I would reconstruct the voice itself to be breathier. Using the WORLD vocoder, I decomposed the signal into F0 (pitch), Spectral Envelope (SP), and Aperiodicity (AP), and then resynthesized it by increasing the aperiodicity. As you push the aperiodicity toward 1, the energy in those bands shifts from periodic components to breathy components. Unlike addition, this theoretically prevents a "separate layer" from appearing.

f0, t = pw.harvest(x, sr)
sp = pw.cheaptrick(x, f0, t, sr)
ap = pw.d4c(x, f0, t, sr)
ap2 = np.clip(ap + (0.45 * amt) * (1.0 - ap), 0.0, 1.0)   # Pushing AP toward 1 = more breath
y = pw.synthesize(f0, sp, ap2, sr, frame_period=5.0)

This looked promising, but new problems emerged one after another.

5. Double Vocoding → Additive Difference (f7bc4d9)

The input was originally audio generated by Seed-VC. Re-synthesizing the entire thing with WORLD resulted in double vocoding, which caused audio artifacts/distortion. To fix this, I tried extracting the difference between the original AP and the increased-AP version (i.e., just the added breathiness) and adding that to the clean original audio. The idea was that the degradation caused by WORLD would be canceled out by the subtraction.

6. Low-end Muddiness (Grit/Gravel) → High-frequency Limiting (2287e39)

Now, the low frequencies (harmonics) were being de-periodized as well, making the voice sound muddy and gravelly. Since breathiness is naturally high-frequency, I tried limiting the AP increase to frequencies >1.5kHz to keep the low-end voiced parts clean. The muddiness and distortion were resolved, but as my commit message notes:

Effectiveness is diminished (limitations of post-processing).

This was where the "ceiling of post-processing" became clear. If you turn it up, it gets muddy; if you stop the muddiness, it loses the effect. You cannot have both.

7. Increasing Intensity (1110567) → Cross-fading with a Whisper (e767575)

I tried pushing it harder despite the side effects (High-cut 1.0kHz, Strength 0.85). To avoid the dilemma—where limited-range AP is too weak and full-range AP is too muddy—I tried creating a "whisper version" of the voice by aperiodicity across all bands, then cross-fading it with the original. 100% would be a pure whisper; the middle would be breathy. Each component remained clean and non-muddy.

By this point, I had exhausted almost every post-processing method. There was one common failure point: The harder you try to increase the breathiness, the more noisy it becomes; the more you suppress the noise, the less breathiness you get.

8. Giving up and using Model-side Generation (e7c6438)

I thought: if post-processing has reached its limit, why not let the model generate the breathiness itself? If the model generates it, the vocoder should render it cleanly. I would blend the Seed-VC reference prompt (cref) with a WORLD-generated whisper version based on the breathiness amount, forcing the model to synthesize the breathiness itself.

# Blend reference prompt with a breathy (whisper) version → model generates breathy voice
cref = (cref * (1.0 - 0.85 * b) + _world_whisper(cref, sr) * (0.85 * b)).astype(np.float32)

Testing showed the voiced ratio dropped from 0.64 to 0.11 (at 100%) and spectral flatness stayed steady at 0.028—meaning the noise levels didn't increase, and we got more breathiness while remaining clean. This was numerically the best result. Because I had moved from post-processing to the generation stage, I used torch.manual_seed(1234) to ensure deterministic results for A/B testing.

Even so—strong breathiness still had that characteristic "breathy" noise. Of course it did. Even if you make the model draw "clean" breathiness, as long as breathiness is fundamentally turbulent noise, a "strong breathiness without noise" cannot exist.

9. Removal (85bbcd0)

So, I reached a conclusion:

Breathiness/whispering is turbulent noise itself. Concluded that it is 
impossible to achieve "strong breathiness without noise" via post-processing 
or the model. Removed breathiness slider from advanced settings (kept WS/pitch).

I removed the slider from the UI and reverted the backend to its pre-experiment state. I kept speech rate (WSOLA) and pitch fluctuation. Only the breathiness lost.

Why didn't I see this from the start?

The failure wasn't technical; it was a failure of assumptions. The UI label "Breathiness" had transformed in my mind into an unattainable goal of "elegant, noise-free breathiness." Acoustically defined, breathiness = breath noise. If you remove the noise, you remove the breath. This isn't a matter of implementation skill; it is physics. Whether via addition, aperiodicity, or model generation, I was just hitting the same wall from different angles.

Another lesson learned: improving intermediate metrics does not guarantee the goal is being met. High-frequency ratio, voiced ratio, spectral flatness—every number was "improving." But I was getting no closer to the self-contradictory goal of "natural breathiness that doesn't sound like noise." The better the metrics looked, the harder it was to see that I was chasing an impossible target.

Summary

• The essence of breathy and whisper voices is turbulent airflow noise. "Breathiness without noise" is an acoustic contradiction; it was a problem of physics, not implementation.
• I tried every method: Noise addition $\to$ High-frequency boost $\to$ Band-limited noise $\to$ Cross-synthesis $\to$ WORLD aperiodicity $\to$ Additive difference $\to$ High-frequency limiting $\to$ Whisper cross-fade $\to$ Model-side generation. All of them resulted in the same dilemma: "increase it and it gets muddy/noisy; decrease it and it's ineffective."
• While WORLD's aperiodicity and model-side generation improved intermediate metrics (voiced ratio, flatness), they did not get closer to the impossible goal. Improving metrics does not mean achieving the objective.
• The final decision was removal of the slider. It was more honest for the product to remove an ineffective or unnatural parameter than to keep it.
• The core of the failure was that the UI label implicitly defined an impossible goal. Before building a feature, you must define the term acoustically and verify if it is physically possible.