I recently completed a browser-local, multilingual voice-cloning workflow for Timeline Studio.
The reference recording is not sent to an inference server. Audio decoding, speaker embedding extraction, tone-color conversion, preview, persistence, and timeline replacement all run inside the browser.
Repository: https://github.com/MartinDelophy/ai-video-editor
This post is an engineering record of the production workflow: model packaging, audio preprocessing, WebGPU and WASM execution, workers, cache behavior, tail cleanup, IndexedDB persistence, and the precision decision behind the shipped FP16 build.
The product workflow
The feature is deliberately split into two stages:
Text input
↓
Select language and base TTS voice
↓
Generate source speech
↓
Upload or record a reference voice
↓
Extract the target speaker embedding
↓
Run OpenVoice V2 tone-color conversion
↓
Preview the converted result
↓
Save to My assets or replace the current timeline clip
TTS owns pronunciation, language, prosody, and linguistic content. OpenVoice V2 performs the second-stage speaker tone-color transfer.
For Japanese text, the source speech is generated with a Japanese TTS voice. English uses an English source voice, and other supported languages follow the same route. A saved speaker profile can therefore be reused across languages without asking the converter to generate linguistic content itself.
This separation also keeps the editing workflow predictable: users can verify the base pronunciation first, then verify the cloned tone color as a separate operation.
Model packaging
The browser build uses an FP16 ONNX export of the OpenVoice V2 converter. The runtime is split into two artifacts:
| Artifact | Size |
|---|---|
| Reference Encoder | 1,637,269 bytes |
| Converter | 64,314,222 bytes |
| Total | 65,951,491 bytes |
The total download is approximately 65.95 MB, or 62.9 MiB.
The artifacts are hosted in project-owned Hugging Face and ModelScope mirrors and pinned to immutable provider revisions. Chinese and domestic sessions prefer ModelScope, while other sessions prefer Hugging Face. If the preferred provider fails, the loader falls back to the other mirror.
Both providers map to one provider-independent cache identity. Switching download sources does not create duplicate local copies of the same model.
Audio preprocessing
Uploaded files and browser recordings are decoded with the Web Audio API. Before inference, the runtime performs:
- mono downmixing;
- resampling to 22,050 Hz;
- conversion to internal Float32 PCM;
- level normalization;
- silence and low-energy tail analysis.
OpenVoice does not receive the raw waveform directly in this browser pipeline. The worker computes a short-time Fourier transform with the following parameters:
Sample rate: 22,050 Hz
FFT size: 1,024
Hop length: 256
Frequency: 513 bins
The reference encoder input is arranged as:
[1, frameCount, 513]
The converter spectrogram uses:
[1, 513, frameCount]
The remaining converter inputs are:
Frame mask: [1, 1, T]
Source embedding: [1, 256, 1]
Target embedding: [1, 256, 1]
Noise: [1, 192, T]
The noise tensor is generated from a deterministic seed. This makes repeated runs easier to reproduce during browser and audio-quality validation.
WebGPU and WASM execution
Inference runs through ONNX Runtime Web.
The reference encoder is small and uses WASM. The converter prefers WebGPU and falls back to a WASM-only session if WebGPU session creation fails.
Reference Encoder → WASM
Converter → WebGPU
↘ WASM fallback
The relevant runtime configuration is:
graphOptimizationLevel = "all"
WebGPU powerPreference = "high-performance"
WASM SIMD = true
WASM threads = 1–4
WASM multithreading is enabled only when the page is cross-origin isolated. This avoids promising a multithreaded execution path on pages where the required browser isolation headers are unavailable.
Model bytes are downloaded in parallel, but inference sessions are initialized serially:
Model downloads: parallel
Session initialization: serial
Parallel downloads reduce network wait time. Serial session creation limits the temporary memory pressure caused by constructing multiple ONNX sessions simultaneously.
Keeping inference away from the UI thread
STFT processing and model inference can block scrolling, button feedback, and timeline interactions when executed on the main thread.
The complete inference path therefore runs in a dedicated Web Worker:
Main thread
│
├─ transfers PCM, configuration, and embeddings
│
Web Worker
├─ computes STFT
├─ runs Reference Encoder
├─ runs Converter
├─ performs audio postprocessing
└─ transfers converted PCM back
PCM buffers use transferable ArrayBuffers, avoiding copies of large Float32 arrays between the UI and inference contexts.
Cancellation terminates the active worker and rejects pending requests. A boolean cancellation flag alone cannot reliably interrupt an ONNX call that is already executing.
The initialized worker remains alive for repeated conversions on the same page, so a second conversion does not present itself as another model setup operation.
Removing the quiet audio tail
During testing, some converted clips contained a long, very quiet tail after the actual speech ended.
This can come from residual low-energy output around padded spectrogram frames and waveform reconstruction. Cutting every result at a fixed duration would also remove natural word endings, so the cleanup is based on RMS activity.
The current parameters are:
RMS window: 20 ms
RMS hop: 10 ms
Minimum useful peak: 0.0025
Activity threshold: max(0.0015, peakRms × 0.035)
Tail retained: 160 ms
Cosine fade: 40 ms
The worker scans backward for the final active window, keeps 160 ms of natural tail, and applies a 40 ms cosine fade.
This removes low-level output that can otherwise continue for several seconds while preserving the audible ending of the sentence.
Output gain and limiting
Converted audio can be slightly quieter than its source TTS clip. Timeline Studio allows audio-clip volume from 0% to 400%.
Applying a linear gain of 4.0 without protection would create hard clipping, so the output path uses a soft limiter:
Volume range: 0–400%
Limiter: tanh
Drive: 1.35
The same gain envelope is used by preview and export. The volume control is therefore part of the real audio pipeline rather than a player-only adjustment.
Speaker profiles in IndexedDB
Reference audio, test results, and extracted embeddings are stored locally in IndexedDB.
A saved profile can contain:
id
name
referenceBlob
testBlob
speakerEmbedding
language
favorite
authorization
createdAt
updatedAt
This supports a reusable local voice library:
- upload a clean reference recording;
- record a reference voice in the browser;
- test the cloned result;
- save the profile;
- add or remove it from favorites;
- reuse it for later synthesis;
- delete the local profile.
Persisting the speaker embedding means the reference encoder does not need to run again every time the same voice is selected.
Cache quota is not an inference failure
The loader attempts to store model artifacts in a versioned Cache Storage entry. Some browsers and devices can still raise QuotaExceededError for a roughly 66 MB model.
The important design rule is that a cache write failure must not invalidate model bytes that have already been downloaded:
Download model
↓
Attempt Cache Storage write
├─ success → reuse on later visits
└─ failure → continue inference from memory
The application also requests persistent storage when available, reducing the likelihood of automatic cache eviction.
Users receive a contextual message explaining that the model is running in memory. A raw “Failed to fetch” or quota exception is not exposed as the product error.
Why the production build remains FP16
The released browser model uses FP16.
ONNX can represent Float8 formats such as E4M3 and E5M2, but current browser WebGPU and WGSL environments do not yet provide a sufficiently stable, general native FP8 execution route for this graph.
An FP8 artifact may still require parts of the graph to cast or dequantize into FP16 or FP32 at runtime. Shipping that path would require validation of:
- additional Cast and Dequantize nodes;
- unsupported operator fallbacks;
- CPU-to-GPU and GPU-to-CPU transfers;
- browser and GPU compatibility;
- changes in speaker similarity, noise, and quiet tails;
- a separate model and cache identity.
The current release decision is therefore:
Production model: FP16
FP8: isolated experiment
The decision is based on the effective browser execution path, not only the number printed on the weight format. FP8 will remain separate until execution coverage and audio behavior can be validated without silently expanding most of the graph back to a wider precision.
Connecting conversion to the editing timeline
A completed conversion does not automatically overwrite the current clip.
After previewing the output, users can explicitly choose to:
- save the result to My assets;
- run another conversion;
- replace the current clip;
- restore the original audio;
- download the audio file.
Replacing a clip preserves its timeline position and editing context while updating the underlying audio asset and duration. A voice-cloning test cannot silently destroy an existing edit.
Final browser pipeline
The resulting browser-local workflow is:
Multilingual TTS
→ browser audio decoding
→ resampling to 22,050 Hz
→ STFT
→ speaker embedding extraction
→ OpenVoice V2 tone-color conversion
→ low-energy tail cleanup
→ gain and soft limiting
→ IndexedDB voice persistence
→ timeline preview and replacement
→ local export
Converting a model to ONNX was only one part of the implementation. A usable browser feature also needed model delivery, fallback execution, worker transfers, cache degradation, audio cleanup, profile persistence, and safe timeline integration to behave as one coherent system.
The complete implementation and its development history are available in the repository:
Top comments (0)