Running AI Music Generation Entirely in the Browser: ONNX, WebGPU, Samplers, and Long Audio Stitching

Many developers think running AI models in the browser is hard because of inference performance.

After shipping local browser AI music generation in our open-source video editor Timeline Studio, I can confirm:

Model inference is the easy part.

The real difficulty lies in solving browser-specific constraints: model size, WebGPU memory limits, cache quotas, sampler correctness, long audio generation, and stable editor integration.

This article breaks down our complete production-grade browser AI music pipeline, including all pitfalls and fixes.

1. Why We Chose Stable Audio 3 Small (Q4 ONNX)

We use the Stable Audio 3 Small Music Q4 ONNX model.

Total size: 683MB

It consists of four core components:

Module	Purpose	Size
Text Encoder	Encode prompt into conditioning vectors	213MB
Number Conditioner	Encode target audio duration	<1MB
DiT Diffusion	Main denoising computation	380MB
Audio Decoder	Decode latents to 44.1kHz stereo audio	45MB

Browser-specific optimizations

1. Model sharding
   We split model weights into chunks under 100MB to avoid browser network timeout and parsing failures.

2. 4-bit quantization

3. MatMul / Linear → MatMulNBits

4. Embedding → GatherBlockQuantized

5. Other parameters remain FP32

Q4 drastically reduces download size and runtime memory usage.

Tradeoff: minor grainy artifacts in high-frequency and reverb tails — acceptable for fully local browser generation and runnable on 16GB M1 devices.

2. Parallel Download + Serial WebGPU Initialization

Parallel network fetch

Model files are downloaded in parallel to reduce first-time startup latency:

const responses = await Promise.all(
  paths.map(path => fetchModelFile(path))
);

Serial GPU initialization (critical fix)

While network requests are parallelizable, WebGPU session creation cannot.

Parallel initialization of multiple large ONNX sessions causes instant unified memory spikes, especially on Apple Silicon, often leading to WebGPU allocation failure or tab crash.

We enforce a strict sequential pipeline:

Text Encoder → Number Conditioner → DiT → Audio Decoder

Core engineering rule

Network I/O can be parallelized, but WebGPU resource initialization must be serialized.

Once initialized in a Web Worker, all GPU sessions are reused for subsequent generations, avoiding reloading overhead.

3. Multilingual Prompt Pipeline (Auto Translate + Structured Prompts)

Stable Audio models natively understand only English.

To support global users, we built a fully automatic multilingual prompt pipeline:

User native prompt
→ Language detection
→ Chrome built-in translation
→ Structured English music prompt

We also auto-append standardized metadata: style, mood, instrument, BPM, and no-vocal constraints.

Example transformation:

Raw Chinese input
melancholic piano in a rainy café

Final structured prompt

melancholic jazz piano in a rainy café,
cinematic soundtrack,
dreamy,
piano,
90 BPM,
instrumental music,
clean production,
no vocals

For browsers without translation support, we gracefully fall back and ask users for English input, avoiding silent generation failures.

4. Most Audio Quality Issues Are Not From Quantization

Initially, our generated audio suffered from:

• Grainy high-frequency noise
• Blurry instrument texture
• Unstable reverb tails
• Loose rhythm structure

We initially suspected Q4 quantization quality loss.

After deep investigation, the real root cause was incorrect sampler schedule implementation.

Correct Rectified Flow sampling logic

denoised = x - tCurrent * velocity;
xNext = (1 - tNext) * denoised + tNext * randomNoise;

The time parameter t must decay strictly from 1 to 0.

Our original schedule only decayed to ~0.27, leaving 27% residual noise in the final latent before decoding — this destroyed audio quality.

Fixed LogSNR schedule

const logSnr = 2 - t * 8.2;
const sigma = 1 / (1 + Math.exp(logSnr));

// Hard enforce clean endpoints
schedule[0] = 1;
schedule[steps] = 0;

// Zero-noise final step
if (tNext === 0) {
  x = denoised;
}

This single fix improved audio quality far more than increasing sampling steps.

Key takeaway

Poor local model generation quality is usually caused by preprocessing/scheduler bugs, not model quantization.

5. From Latent Output to Final WAV

Latent length is dynamically calculated based on target audio duration:

latentLength =
  Math.ceil((seconds + 6) * 44100 / 8192) * 2;

DiT outputs latent tensors, and the decoder produces:

• Shape: [1, 2, audioFrames]
• Format: Float32
• Range: -1 ~ 1
• Sample rate: 44100Hz

We convert float samples to standard 16-bit PCM WAV:

sample< 0 ? sample * 32768 : sample * 32767;

Finally, the audio asset is automatically:

• Converted to WAV blob
• Decoded for accurate duration
• Waveform-analyzed
• Saved with prompt/model metadata
• Imported into the editor asset library

6. Long Audio Generation: Why We Avoid Direct 120s Inference

Direct long-form audio inference (90s/120s) in browsers causes:

• WebGPU buffer allocation failures
• Excessive memory usage
• Tab freezing or system process killing
• Peak decoder memory pressure

To solve this, we use a segmented generation + intelligent looping strategy:

• 90s target → generate 45s segment
• 120s target → generate 60s segment

The final long audio is stitched from high-quality short segments, cutting VRAM and inference time nearly in half.

7. Seamless Audio Stitching (No Clicks, No Jumps)

Naive loop splicing creates obvious artifacts:

• Volume jumps
• Misaligned drum hits
• Phase discontinuity
• Audible clicks/pauses

We analyze the last 5 seconds of the source segment and score candidate cut points using:

• RMS energy
• Amplitude jump difference
• Waveform slope jump
• Shorten ratio penalty

Unified scoring formula:

score =
  rms * 0.7
  + amplitudeJump * 0.8
  + slopeJump * 0.25
  + shortenedRatio * 0.08;

Lower score = better loop boundary.

We also use dynamic fade duration (0.25s ~ 1.5s) based on local audio energy:

fadeSeconds = clamp(
  0.25 + localRms * 4,
  0.25,
  1.5
);

High-energy regions get longer fades; quiet regions use short transitions for natural looping.

8. Model Caching: Harder Than Inference

We use Service Worker + Cache Storage to cache the 683MB model persistently.

We fixed two production-critical browser cache issues.

Issue 1: Cache hit misjudgment

Custom HTTP headers cannot be reliably exposed cross-origin after Service Worker proxying.
This caused UI to show “downloading” even when models were loaded from local cache.

Fix: Directly query Cache Storage from the AI worker instead of relying on response headers.

Issue 2: QuotaExceededError

Dual-write from both AI Worker and Service Worker caused temporary duplicate model occupancy, exceeding browser storage quota.

Final cache architecture:

• Only Service Worker writes cache
• AI Worker only reads/validates cache
• Cache failure does not block generation
• Auto clean old model cache versions

Large model caching requires a single source of truth for writes. Cache logic must never break core inference.

9. Model Mirroring & Immutable Version Locking

Public Hugging Face repos are unstable for production use — authors may delete, overwrite, or update weights.

We fully mirrored and frozen the model:

• Mirror repo: haixin/stable-audio-3-small-music-onnx
• Fixed immutable commit hash: 0b8a05e0bc3511e674b4cb3413d3ef6c48880cdb

We verified:

• 26 model files
• Full 683MB byte integrity
• SHA256 hashes
• ONNX graph & weight shards
• License & notice files

Old cache entries are compatible, so existing users do not need to re-download.

10. Real Boundaries & Value of Browser-Native AI Music

Limitations

• Q4 quantization cannot match FP16/FP32 server-side quality
• Long audio relies on intelligent looping instead of full inference
• WebGPU support depends on device/browser implementation

Unique strengths

• Zero server-side inference cost
• 100% local processing (prompt + audio never uploaded)
• One-time model download + persistent cache
• Hot GPU session reuse for fast subsequent generation
• Native integration into video editor workflow
• Deployable via static hosting

Conclusion

Running production-ready AI music in the browser is not just about getting ONNX inference running.

It requires careful system-level engineering:

• Correct sampler scheduling
• WebGPU memory & concurrency control
• Intelligent long audio stitching
• Robust caching strategy
• Immutable model supply chain

With these fixes, a 683MB quantized music model can become a usable, stable, production-grade browser AI feature.

Open Source Project

If you are interested in browser-side AI, WebGPU inference, or local media generation:

GitHub: martindelophy/ai-video-editor