Making Diffusion Model Generation Deterministic — Ensuring Consistent Voice from the Same Input

📝 Originally published (in Japanese) at forge.workstyle.tech.

While tuning parameters for a voice conversion app, I ran into a frustrating scenario:

Move a slider slightly $\rightarrow$ Generate $\rightarrow$ Move it back to the original position $\rightarrow$ Generate $\rightarrow$ For some reason, the voice sounds slightly different.

The slider was exactly where it was. The recording was identical. Yet, the resulting voice didn't match the previous output. "I think it sounded better a moment ago, but now I can't get it back"—this makes it impossible to verify exactly how a specific parameter is affecting the output. The cause was that the diffusion model starts generation from random noise. This article is about how I made the generation process deterministic to ensure "same input always equals same voice."

Why Diffusion Models Change with Every Execution

Generation via diffusion models (specifically CFM: Conditional Flow Matching) starts with random noise and gradually refines it according to specific conditions to approach the target output (in this case, a mel-spectrogram).

Because the starting point is random, even with the same conditions, a different initial noise seed means the final result will shift slightly every time. In voice conversion, this means that while the speaker identity and content remain the same, the timbre and breathing subtly fluctuate with every generation. While the quality itself might not be "bad," this lack of reproducibility is a fatal flaw in a production environment.

The relevant inference code looked like this. Without fixing the random seed, cfm.inference pulls a different initial noise state every time it is called.

vt = _S["model"].cfm.inference(cat, torch.LongTensor([cat.size(1)]).to(dev),
                               mel2, style, None, STEPS, inference_cfg_rate=CFG)

The Solution: Fixing the Seed for Determinism

The fix is simple: fix the PyTorch random seed immediately before generation.

# Make generation deterministic: Same input (recording + slider) 
# results in the exact same conversion every time.
# (Since CFM samples from random noise, it varies per execution if not fixed)
torch.manual_seed(1234)
if dev.type == "cuda":
    torch.cuda.manual_seed_all(1234)

It's only a few lines, but there are two key points here:

• CPU and GPU use different random number generators. torch.manual_seed() fixes the generator for the CPU, but the GPU (CUDA) side is separate. Therefore, when dev.type == "cuda", calling torch.cuda.manual_seed_all() ensures the results are consistent regardless of the execution environment.
• The seed must be fixed "immediately before generation." If you fix it only once during initialization, other processes may consume random numbers in the meantime, changing the noise for the next generation. To be certain, you must re-fix the seed right before calling cfm.inference every single time.

Now, "Same Recording + Same Slider = Same Voice" is guaranteed. The actual value (1234) doesn't matter; what matters is that it is fixed.

3 Practical Values of Determinism

"Getting the same result every time" might seem boring, but it provides immense value for both development and the final product.

1. Meaningful A/B Testing

When comparing whether Parameter A or Parameter B is better, non-deterministic generation makes it impossible to tell if the difference is caused by the parameter or just random jitter. By fixing the seed, everything except the modified part remains identical. The difference in output is purely attributable to the parameter change, establishing the fundamental requirement for comparative experimentation.

2. Isolated Verification of Parameter Effects

"How does the voice change if I increase the 'huskiness' slider by 10%?" To answer this, everything except huskiness must be frozen. With determinism, moving a single slider allows you to observe the specific impact of that axis on the sound. Tuning during design shifts from "gut feeling" to reproducible observation.

3. Reproducible Bug Reports

When a user reports "I got a weird sound with these settings," determinism allows you to reproduce the exact same issue from the same input. In a non-deterministic system, investigations often stop at "I can't reproduce this on my end." With a fixed seed, you can grab a bug and not let go, which drastically improves debugging productivity.

Separating Post-Processing from Determinism

I implemented one more design tweak. Post-processing DSP (such as speech rate, pitch fluctuation, and breathing) is decoupled from the model generation. I cache the raw generated waveform so that only the post-processing can be re-applied.

# Cache the raw generated waveform so post-processing DSP 
# can be re-applied without re-generating from the model.
_S["last_raw"] = wav.astype(np.float32).copy()

This removes the need to run the diffusion model every time a post-processing parameter is tweaked. Heavy generation (the deterministic, stable part) happens once, while light post-processing (the part you want to iterate on) is rapidly re-applied to the raw waveform. By separating the "layer that should be fixed/deterministic" from the "layer for rapid trial-and-error," I achieved both reproducibility and iteration speed.

Pitfalls and Lessons Learned

• "If the quality is good, non-determinism is fine" is a trap. Output quality and reproducibility are different axes. Even if quality is high, you cannot verify, compare, or debug without reproducibility. If you're aiming for a product, you should implement determinism early.
• The position of the seed fix is everything. If you fix it once at the start and stop there, random numbers will be consumed by other functions, and the output will drift. Be rigorous about fixing the seed immediately before generation.
• Don't forget the CPU/GPU divide. Fixing only one and wondering why "it doesn't reproduce on GPU" is a very common pitfall.
• Separate the fixed layers from the experimental layers. Don't make everything deterministic. Fix the heavy generation that requires stability, and use caching for the light post-processing that requires frequent iteration. This boundary determines your development efficiency.

Summary

• Diffusion models (CFM) generate from random noise, meaning results vary slightly even with the same input.
• Call torch.manual_seed(1234) (and torch.cuda.manual_seed_all for CUDA) immediately before generation to ensure determinism.
• Benefits of seed fixing: A/B testing, isolated parameter verification, and bug reproduction—all based on the premise that "everything except the variable is identical."
• Key points: Fix the seed right before generation and cover both CPU and GPU generators.
• Separate post-processing DSP by caching the raw waveform. Heavy generation is fixed and done once; light post-processing is re-applied quickly for rapid iteration.