Creating Perceivable Vibrato/Pitch Fluctuations

📝 Originally published (in Japanese) at forge.workstyle.tech.

Adding a Vibrato Slider to the Voice Design App

We added a "pitch variation" slider to the "voice design" app to reduce the robotic feeling of the voice and add a natural vibrato. The implementation was straightforward, and with a sine wave, we could verify that the pitch was indeed fluctuating periodically. However, when we applied it to an actual voice, even with the slider set to maximum, the difference was barely audible to the ear. This is a typical problem in DSP, where the parameters are correct, but the effect is not perceivable.

This article discusses the mechanism of creating vibrato using delay modulation and the design of modulation depth to achieve a perceivable effect.

Premise: Why Delay Modulation Makes Pitch Fluctuate

There are several ways to fluctuate pitch, but here, we used a time-varying delay (fractional delay). This is the same principle as chorus and flanger effects.

The key point is that it's not the delay time itself, but the rate of change of the delay time that fluctuates the pitch. As the delay quantity increases over time, the playback position slowly moves backward, causing the pitch to decrease due to the Doppler effect. Conversely, if the delay quantity decreases, the pitch increases. By making the delay quantity oscillate with a sine wave, the pitch fluctuates up and down, creating a vibrato.

Mathematically, the instantaneous pitch change is roughly proportional to 1 - d(delay)/dt. Therefore, both the speed of the fluctuation (modulation frequency) and the depth of the fluctuation (maximum delay amount) determine the strength of the vibrato.

Implementation: Oscillating Delay Quantity with a Sine Wave

The implementation (_apply_pitch_variation) is as follows:

n = len(wav)
t = np.arange(n) / sr
rate_hz = 4.5
max_delay = (0.0030 * amt) * sr           # maximum ~3.0ms (100% for ± approximately 65 cents of vibrato)
delay = max_delay * (0.5 - 0.5 * np.cos(2 * np.pi * rate_hz * t))
idx = np.clip(np.arange(n) - delay, 0, n - 1)
i0 = np.floor(idx).astype(int)
frac = (idx - i0).astype(np.float32)
i1 = np.minimum(i0 + 1, n - 1)
return wav[i0] * (1 - frac) + wav[i1] * frac

Let's break it down:

• rate_hz = 4.5 is the vibrato speed. Human natural vibrato is around 4-7Hz, so we placed it near the lower end. If it's too fast, it sounds like a tremolo, and if it's too slow, it sounds out of tune.
• delay oscillates between 0 and max_delay with a sine wave. The delay quantity oscillates, so its derivative, the pitch, fluctuates up and down.
• idx is the "current sample position". It shifts backward by the delay amount. Since the position is not an integer, it becomes a fractional delay.
• The last three lines are linear interpolation. We interpolate between i0 (integer part) and i1 (next sample) with frac (fractional part). This interpolation is necessary to achieve fractional delay; without it, the sound becomes jittery and noisy.

amt (0-1) linearly affects max_delay. Currently, the setting is a maximum delay of ~3.0ms at 100% slider value.

Main Topic: Solving the "Inaudible" Problem with Depth

In the initial implementation, max_delay was 1.0ms. Verification with a sine wave was normal, but the effect was not perceivable with an actual voice.

The reason was simple: the modulation was too shallow. From the above relationship, the modulation depth was approximately ±24 cents. Since one half-tone equals 100 cents, 24 cents is less than a quarter of a half-tone. While it's noticeable with a sine wave, it's not perceivable with an actual voice that has overtones and formants. The cause of the "existent but not perceivable" problem was this shallow modulation.

We increased the maximum delay from 1.0ms to 3.0ms. This resulted in a vibrato depth of ± approximately 65 cents at 100% slider value, which was clearly perceivable. This change is also reflected in the git history:

fix: Pitch fluctuation is not perceivable. Increased modulation depth from 1.0ms to 3.0ms (100% for ± approximately 65 cents)
Sine wave verification showed the effect was present, but ±24 cents was too subtle and not perceivable.

Interestingly, even with increased depth, it doesn't sound out of tune. As long as the speed (4.5Hz) remains within the natural vibrato range, increasing the depth doesn't make it sound out of tune; instead, it's perceived as a "fluctuating expression." The roles are divided: depth (max_delay) makes the effect stronger, and speed (rate_hz) ensures it sounds natural.

Pitfalls and Lessons

• Sine wave verification only shows if it's working, but it doesn't guarantee the effect is perceivable with actual audio material. DSP requires separate evaluations for "operation correctness" and "perceivable effect".
• Perception is non-linear. In terms of the perceived pitch scale (cents), ±24 cents and ±65 cents differ by a factor of roughly 2.7, but the perceived difference is qualitative, changing from "imperceptible" to "clearly audible". Parameters should be designed based on perceived quantities, not physical quantities.
• Separate the roles of depth and speed. Effect strength is determined by depth (max_delay), while naturalness is ensured by speed (rate_hz). Adjusting both simultaneously makes it difficult to distinguish between their effects.
• Fractional delay requires linear interpolation. Rounding to the nearest sample introduces jitter and aliasing, making the sound noisy and ruining the vibrato effect.

Summary

• Vibrato, or pitch fluctuation, can be created using time-varying delay (fractional delay). The pitch is fluctuated not by the delay itself, but by the rate of change of the delay.
• The speed (rate_hz) should be within the natural vibrato range (4-7Hz), and the delay quantity should oscillate with a sine wave. Reading is done using linear interpolation for fractional delay.
• The main cause of the "existent but not perceivable" problem was insufficient modulation depth. ±24 cents was too subtle, while ±65 cents was clearly perceivable.
• Increasing depth doesn't make it sound out of tune if the speed remains within the natural range. Depth makes the effect stronger, and speed ensures naturalness.
• Verifying operation with a sine wave and verifying perception with actual audio material are distinct. Parameters should be designed based on perceived quantities (cents), not physical quantities.