Most tutorials about vocal separation start with "run Demucs on your GPU". That is a fine answer if you own the GPU, have Python set up, and are willing to wait. It is a bad answer for a web product, where the upload arrives over HTTP from a phone and the user expects a progress bar, not a CUDA error.
This post is about the middle part of that pipeline: what actually happens between "the user dropped a 40 MB MP4 on the page" and "here are two MP3s". I build Voice Remover, so the numbers and the failure modes below come from running it, not from a paper.
Step 1: decode in the browser, upload only the audio
The first decision that shapes everything else is where the file gets decoded. Server-side decoding of video is expensive, and it means a phone video sits on your storage before you know whether it is even usable.
Instead, the page uses the Web Audio API to decode the container and extract the audio track client-side. For an MP4 or MOV, the video stream never leaves the device. A 40 MB phone video can end up as a 3 MB audio upload, which changes both your bandwidth bill and the number of uploads that survive a flaky mobile connection.
// decode locally, re-encode to WAV, upload only that
const arrayBuffer = await file.arrayBuffer();
const audioCtx = new OfflineAudioContext(1, 1, 44100);
const decoded = await audioCtx.decodeAudioData(arrayBuffer);
// ...encode `decoded` to 16-bit PCM WAV, then POST the Blob
Two things bite here. decodeAudioData support differs by codec and browser, so M4A and MOV are the risky pair, not MP3. And you must re-encode rather than uploading the decoded buffer raw, because what you send to the separators wants to be a real audio file in a real container.
Step 2: an async job, not a long request
Separation takes tens of seconds. If you hold an HTTP request open for that, every proxy in the world will eventually time out on you, and you have no way to show progress.
The shape that works: the upload lands in object storage, a row is written to the database with a status, a message goes onto a queue, and the browser polls or subscribes for the status of that row. On Cloudflare that is R2 for the bytes, D1 for the row, and Queues for the handoff. The work runs on a Worker, which gets a fresh CPU budget per invocation and cannot be trusted to hold state in memory.
upload -> R2 -> D1 row (status=queued) -> Queue message -> Worker
Worker: claim row -> call separator -> write outputs to R2 -> status=done
The row is the only source of truth. Everything else is replaceable, which matters when your queue redelivers a message and you need the second worker to notice the job is already claimed.
Step 3: treat the third-party API as hostile
Separation itself is not something you want to run on a Worker. The model is heavy, and Workers are for glue. So the audio goes out to an inference provider — htdemucs on fal in our case — and comes back as two stems.
The lesson from production is that a third-party inference call is not a function call. It is a thing that can: accept your job and then lose it, return two stems with slightly different lengths, return silence for a file it did not like, or time out in a way that tells you nothing. So:
- Store the provider's job id on your row before you await anything,
- put a hard deadline on the polling loop rather than trusting the provider's own timeout,
- and on any ambiguity, fail the row with a message a human can act on, rather than retrying three times and charging the user for a broken result.
Step 4: mix and encode on the edge, not in the model
The stems come back as raw audio and that is not what people want to download. Mixing and MP3 encoding happen back on a Worker. This separation of concerns is the whole reason the architecture is worth writing about: the model does one job, the plumbing does another, and you can change the model without touching the encoding.
Step 5: delete things on a schedule
Uploads get deleted when the job completes. Outputs get deleted after 24 hours. This is not only a privacy stance — although it is that too — it is cost control. Storage you forget about is the most common way a side project's bill climbs.
What this architecture actually costs you
Honest list of things that did not go smoothly:
- Phone video is a menu of edge cases. A MOV from an iPhone is not the same container as an MOV from a screen recorder. Some files carry multiple audio tracks; pick one deliberately instead of letting the decoder guess.
- The 10-minute file limit is a product decision disguised as a technical one. Longer files mean longer inference, more retries, more storage, and a worse free tier. Cutting at ten minutes kept the free tier honest.
- Separation quality is a property of the recording, not the pipeline. Heavy reverb on the vocal, a live crowd, an old mono master — all of these leave traces in the instrumental. No amount of queueing fixes that, and users will tell you about it in the first week.
- Backing vocals travel with the lead vocal. htdemucs gives you vocals and instrumental here; it is not a four-stem split. Anyone expecting drums and bass separately will be disappointed, so say so up front.
The one-line version
Decode in the browser, queue the job, keep one row as the source of truth, treat the inference provider as unreliable, encode at the edge, and delete on a schedule. If you are building anything that puts a heavy model behind a web form, the model is the easy part.
If you want to see the output rather than the diagram, the stem splitter shows the full two-track split, you can also isolate vocals from any song and get the acapella first, which is the view producers want, and there is a separate background music remover built on the same pipeline for speech-in-video use cases. The pricing page lists what the free tier actually covers, which is the number most people ask about first.
Top comments (0)