No upload, no render queue: how a song and a cover become an H.264 file with WebCodecs, and the parts that broke first.
Musicians who want a visualizer for a new single usually have two things: an unreleased master they would rather not upload anywhere, and a deadline. That combination makes a good case for doing the whole job on the client. The browser can decode the audio, analyze it, draw every frame and encode an MP4 without a server ever seeing the file. This post walks through the pipeline behind an audio visualizer built that way, and the places where real browsers pushed back.
Analyze The Audio Once, Offline
The tempting approach is to wire an AnalyserNode to the playing song and draw whatever it reports. That works for a live preview and fails for an export, because an export has to run faster or slower than real time and still produce the same frames every time.
So the audio is decoded to PCM once, folded to mono, and turned into a feature track with no Web Audio involved. Each feature frame holds:
- 48 spectrum bands, log-spaced from 30 Hz up to about 11 kHz, from a Hann-windowed FFT
- loudness, plus separate bass and mid envelopes
- a kick envelope with a 0.16 s decay and a minimum gap of 0.11 s between kicks
- onsets and an overall BPM estimate
The track is sampled at 60 feature frames per second. A 30 fps render interpolates between them, and a 60 fps render reads them directly. Because the analysis is a pure function from samples to arrays, it runs in a worker, on the main thread in slices when a worker is not available, and under node --test in CI. The same numbers drive a live spectrum analyzer view and the exported video, so what you preview is what you get.
Draw Frames From A Clock You Own
With the features precomputed, rendering frame n means drawing the scene at time n / fps. Nothing reads performance.now() or waits for requestAnimationFrame. That makes the export deterministic, and it lets a slow laptop take longer without dropping frames.
for (let i = 0; i < frameCount; i++) {
drawScene(ctx, features, i / fps); // pure: same inputs, same pixels
const frame = new VideoFrame(canvas, { timestamp: Math.round((i * 1e6) / fps) });
encoder.encode(frame, { keyFrame: i % (fps * 2) === 0 });
frame.close();
if (encoder.encodeQueueSize > 8) {
await new Promise((r) => encoder.addEventListener('dequeue', r, { once: true }));
}
}
await encoder.flush();
Two details matter more than they look. frame.close() is not optional: forget it and memory climbs until the tab dies somewhere around minute three of a long song. And the encodeQueueSize check is the only backpressure you get. Without it the loop happily queues hundreds of frames the encoder has not touched.
Ask Before You Offer 4K
Hardware support for H.264 at 3840×2160 varies by machine, browser and power state. Rather than guess, the app asks the encoder before the 4K option is shown:
const bitrate = Math.round(((width * height * fps) / (1920 * 1080 * 30)) * 6_000_000);
const { supported } = await VideoEncoder.isConfigSupported({
codec, // an H.264 profile string, e.g. 'avc1.640033'
width,
height,
framerate: fps,
bitrate,
});
The bitrate scales from 6 Mbps at 1080p30 by pixel rate, so 1080p60 asks for 12 Mbps and 4K30 for 24 Mbps. If the check fails for every candidate codec string, the option simply is not offered. That is a better experience than letting someone wait through a long render that dies at the first frame.
Do Not Re-Encode The MP3
The most common complaint about "convert MP3 to video" tools is that they decode the song and encode it again, a second lossy pass on audio that was already compressed. For a whole-song export there is no reason to. The MP3's packets can be copied into the MP4 untouched, the same idea as ffmpeg -c:a copy. That is what the mp3 to mp4 path does.
There is one catch. MP3 encoders add a short block of priming samples at the start, so the first audible sample is not at time zero. If you copy the packets and start the visuals at zero, the bars hit slightly before the kick. The fix is to read the encoder delay from the file and offset the visual timeline by that amount. When the source is WAV or FLAC, nothing can be copied, so the audio is encoded once to AAC from the lossless samples.
The muxing itself is handled by a library (Mediabunny in this case) that takes encoded video chunks and audio packets and writes a standard MP4 container.
Plan For The Browser Taking The Encoder Back
The failure that took longest to understand was an export that died halfway with a codec error on a machine that had exported fine a minute earlier. Browsers can reclaim a codec that looks idle or belongs to a tab in the background, and a long export that stalls between frames, for example while a heavy frame draws, can lose its encoder.
Two changes made this rare instead of common. Exports are serialized with the Web Locks API, so two tabs never compete for the same hardware encoder:
await navigator.locks.request('visualizer-export', async () => {
await runExport();
});
And when an export does fail with the specific reclaimed-codec error, it restarts once automatically instead of showing an error. If the second run completes, the person never sees that anything happened.
What It Adds Up To
The result is a pipeline where the only network traffic is the page itself. A full song renders to a 1080p youtube visualizer export on an ordinary laptop, often faster than real time for simpler styles, and the master stays on the machine it started on. The parts that needed the most care were not the visuals. They were the boring edges: backpressure, closing frames, asking the encoder what it can do, keeping the original audio, and expecting the browser to change its mind halfway through.
Top comments (0)