I Built an Audio Tool That Runs in the Browser. Here's What I Wish I'd Known First

Leo Dan — Sun, 16 Aug 2026 02:52:02 +0000

I Built an Audio Tool That Runs in the Browser. Here's What I Wish I'd Known First

A few months ago I put out ezaudio. It's a small set of audio tools, a vocal remover, a stem splitter, and an audio to text thing, and the whole point is that you use it straight from your browser. No app to install, no desktop setup. You open a tab, drop a file in, and get the result back.

I'm not here to sell it. The link is at the bottom and I'll say so plainly. I just want to talk through what building this actually taught me, because a lot of it I had to learn the hard way.

Why even put audio tools in a browser

The easy answer is the one everyone gives. No install step. Anything with a browser is a supported platform, so you skip the entire nightmare of building native apps for every OS. For one person or a tiny team that matters more than people admit. It's the difference between shipping and not.

But the reason people actually kept using mine wasn't the convenience alone. It was the zero friction entry. No downloads, no account wall, no "which version runs on my machine" questions. That low barrier is the whole reason a tool like this gets tried at all. I almost treated that as a footnote early on and that was a mistake.

The tradeoff is real though. You're at the mercy of the network and the backend. Every request has latency, and a flaky connection means a flaky product. You don't get to pretend the heavy work is free.

The browser is the front door, not the engine

This bit tripped me up. The Web Audio API is genuinely good at what it does in the browser: playing the result back, routing it, letting the user scrub and preview. If you've used AudioContext you know what I mean. decodeAudioData handles most of the decoding, AudioBuffer gives you something you can actually process and play, and OfflineAudioContext lets you run the graph without a live audio output. That last one is more useful than it sounds. It turns the browser into a batch processor for the parts you do want client-side, like preview rendering.

But the actual separation and transcription, the part that matters, that runs on the server. The heavy compute happens either on our own machines or through third party model APIs, and the browser just sends the audio up, shows progress, and plays the result. So Web Audio in my case is the plumbing, not the brain.

That changes how you think about the product. You're not shipping compute, you're shipping a thin client with a good UI. The hard problems move to the backend: latency, cost per call, and keeping the models honest.

The vocal remover and stem splitter aren't browser tricks

Here's the thing nobody tells you before you start: the interesting audio work isn't a browser problem at all. It's a machine learning problem wearing a web app costume.

Vocal removal and stem splitting both come down to source separation, and the models that do it well are trained on spectrograms, not waveforms. You take the mix, transform it into a time-frequency representation, and the model learns to predict a mask for each stem, a value per time-frequency bin that says "this is mostly vocals" or "this is mostly drums." Multiply the mix by the mask, invert back to audio, and you've got your stem.

Open source has gotten frighteningly good at this. Meta's Demucs family is the one people keep coming back to, and it splits into four stems, drums, bass, other, and vocals. There's a trick worth knowing: if you want a clean karaoke track, separating into all four and summing everything except the vocals usually sounds better than any dedicated two-stem vocal remover, because each instrument mask was learned with its own structure instead of one catch-all "accompaniment" blob.

None of this runs in a browser tab. These models want a GPU, or at least a very patient CPU. So the whole "browser tool" is really a server-side inference pipeline with a clean upload and playback wrapper. The browser does the parts it's good at, and the server does the parts it's actually good at.

Transcription has its own trap

Speech to text is the same story from a different angle. The model is open and excellent, OpenAI's Whisper family runs locally and fast, but production is where the edges show up.

Whisper has a documented habit of hallucinating. Give it silence, music, or heavy noise and it can invent speech out of nothing, sometimes repeating a phrase until the audio stops. The fix is voice activity detection, you gate what gets fed to the model so silence and music beds never reach it. That one change cut our weird outputs by a huge margin.

Long files are their own problem. Feed a two hour recording to a model with a context window and quality drifts, so you chunk with overlap and stitch the results back together. And if you want word-level timestamps, which people actually use for search and captions, you end up adding a forced alignment step on top of the base transcription. Every one of these is a known problem with a known fix, but they're all work, and they're all yours.

The stuff that bit me anyway

Mobile Safari is its own world. Things that are totally fine on desktop Chrome will just behave differently or refuse outright on iOS. Test there early.

Formats are inconsistent too. The browser can decode a lot but not everything, and of course the weird files are exactly the ones your users upload. We normalized everything server side before it touched a model, and that one habit saved us a dozen support tickets.

Large files will blow up upload time and memory if you're not careful, so you end up chunking and showing progress whether you like it or not. And there's a real tension between quality and how long the user waits. You can't fully win. You just pick which one your tool is for.

Privacy isn't a feature, it's the default

Audio is personal. People upload recordings of themselves, their meetings, their songs, and they notice when you keep them. Our posture is simple: files get auto-deleted within 24 hours, no permanent storage, no "keep it for your convenience." That's not a marketing line, it's an engineering decision that shapes the storage layer, the database, everything. It's also the cheapest trust you can buy.

If you're thinking about building something like this

Start with the UX. Get the upload, the preview, and the result flow feeling instant even when the backend is slow. A spinner that lies is worse than an honest progress bar.

Keep the client thin. Push as much logic to the server as you reasonably can, and treat the browser as a view layer.

Test on the worst device and the worst connection you can find, not the laptop on your fast wifi.

And be straight about limits. A tool that admits what it's bad at earns more trust than one that promises everything.

If any of this was useful, ezaudio is at ezaudio.io. Free to try, runs in your browser.

DEV Community: Leo Dan

I Built an Audio Tool That Runs in the Browser. Here's What I Wish I'd Known First

I Built an Audio Tool That Runs in the Browser. Here's What I Wish I'd Known First

Why even put audio tools in a browser

The browser is the front door, not the engine

The vocal remover and stem splitter aren't browser tricks

Transcription has its own trap

The stuff that bit me anyway

Privacy isn't a feature, it's the default

If you're thinking about building something like this