Shipping a polyphonic audio-to-MIDI converter that runs entirely in the browser

zheng — Mon, 21 Sep 2026 13:17:54 +0000

Almost every "audio to MIDI" tool follows the same shape: upload the file, wait, download a .mid. That is
fine until you notice what you just handed over — a demo, an unfinished song, a client's reference track.

So the constraint I set for Tonera was: the standard conversion must never upload the audio. Everything
below is what that constraint costs, and what I got for paying it.

What actually runs in the page

The transcription model is Spotify's Basic Pitch — an open-source
polyphonic pitch-detection model. It does not need a GPU cluster; it needs a fixed input shape and a bounded
amount of memory, which makes it a candidate for running client-side instead of behind an API.

The pipeline in the browser is deliberately boring:

decode the file the user dropped (AudioContext / WebAudio),
downmix to 22.05 kHz mono — the sample rate the model expects,
run the model, which returns pitch/onset/contour estimates,
turn those estimates into notes the user can inspect and edit,
export.

Steps 1, 2, 4 and 5 are ordinary deterministic code. Only step 3 is a model. Keeping that line sharp matters:
it means the editable part of the result (note boundaries, quantisation, notation) can be recomputed
instantly and reproducibly, while the model runs exactly once per file.

The ceiling nobody advertises: memory

The honest reason there is a ten-minute cap is RAM, not licensing. Decoded 22.05 kHz mono audio plus the
model's intermediate tensors have to fit in a browser tab; a phone and a desktop are very different budgets.
Instead of silently failing on a long rehearsal recording, the limit is stated up front.

The practical workaround is the same one audio engineers already use: split around musical sections and
convert the passage you actually need. You rarely want MIDI for the four minutes of tuning and talking at
the start of a rehearsal take.

Where transcription actually goes wrong

Not where people assume. It is not "MP3 bad, WAV good" — converting an MP3 to WAV before uploading changes
the container and restores nothing that the encoder already discarded.

What breaks transcription is evidence, not bitrate:

compression artefacts — short, blurred attacks in the high end look like note onsets;
reverb and room sound — tails smear note boundaries;
overlapping harmonics — a dense mix asks the model to separate instruments it was never told to separate;
quiet grace notes — the first thing an encoder spends fewer bits on.

That is why the result view is not a download button. It is a note map plus five controls: onset threshold,
frame threshold, minimum note length, and a low/high frequency range — with the discarded candidates still
visible, so raising a threshold is a decision you can see rather than trust.

Post-processing does the rest of the cleanup: quantise to a grid at the detected tempo, split into left/right
hand, smooth velocities, transpose. All of it re-runs instantly and free, because none of it touches the model.

Exports are where the tool earns its place in a workflow

One detection pass, several outputs, because the next step differs by person:

MIDI — straight into a DAW;
MusicXML — into notation software with real engraving;
PDF — for reading, printing, handing to a player;
ABC notation — for text-based / folk workflows;
chord chart — for the "I just need the changes" case.

MusicXML→MIDI, MIDI preview and WAV synthesis are deterministic converts, not generative: if you ask for the
same file twice you get byte-comparable behaviour, not a new opinion.

When server-side work is justified

Dense mixes are a genuine hard case, and pretending otherwise would be dishonest. So there is an optional
high-precision mode that runs HTDemucs stem separation before transcription — the audio is uploaded for
that path, it is labelled before it happens, and it costs credits because it consumes real compute.

Retention is stated rather than implied: temporary source objects are covered by a one-day lifecycle rule and
generated stems by a seven-day rule. If a workflow does not need that mode, nothing leaves the device at all.

Four things I would tell anyone building this

Do not promise one accuracy number. Results depend on instrumentation, room, compression and overlap. Publish the limits next to the tool; it is the difference between a demo and something a musician trusts.
Test on real material — phone memos, 128 kbps downloads, full mixes. A clean studio solo makes every model look good.
Expose the uncertainty. Discarded notes are as informative as kept ones; a threshold without visible consequences is just a slider.
Keep the model at one end of the pipeline. The moment transcription output is frozen into notes, the rest should be cheap, fast and reversible.

Tonera is where I put this together — it runs in the browser, the standard conversion is free and needs no
account, and the accuracy trade-offs are documented rather than buried:
https://tonera.app · method and limits: https://tonera.app/blog/basic-pitch-accuracy-guide

If you try it on something and the result is bad, that is useful information — the failure modes above are
exactly what I want to hear about.