By Tania Alwan
"Audio to MIDI" sounds like a single feature: drop in a recording, get notes out. Under the hood it is two very different problems. Turning one sung or played melody into MIDI is a solved-enough task you can script in about 30 lines of Python. Turning a full mix with chords, bass and drums into clean MIDI is still an active research area. This post explains why, and shows a working monophonic converter.
What MIDI actually stores
A MIDI file does not contain sound. It contains events:
- Note on / note off with a pitch number (0 to 127) and a velocity
- Timing, in ticks relative to a tempo
- Optional controller data (pitch bend, sustain pedal, and so on)
So audio to MIDI means answering three questions for every note: which pitch, when does it start, and when does it end. Pitch numbers map to frequency with a simple formula: MIDI note = 69 + 12 × log2(f / 440). A4 at 440 Hz is note 69, and each semitone adds 1.
Monophonic: one note at a time
When only one note sounds at any moment (a voice, a flute, a whistled melody), the job reduces to pitch tracking:
- Slice the audio into short overlapping frames
- Estimate the fundamental frequency (f0) of each frame
- Decide whether each frame is voiced (a note) or unvoiced (silence, breath, consonants)
- Group consecutive frames with the same rounded pitch into notes
Algorithms like YIN and its probabilistic version pYIN do the f0 step well on clean recordings. The hard parts are musical, not mathematical: vibrato, slides between notes, and short consonants that create false notes.
Polyphonic: why full songs are hard
In a mix, many notes overlap, and each note has harmonics at 2×, 3×, 4× its frequency. Those harmonics collide. A C and the G above it share harmonics, so an algorithm sees energy where both notes would be and has to guess which notes actually caused it. Add drums, reverb and a bass line and the guessing gets harder.
That is why polyphonic transcription moved to neural networks trained on aligned audio and MIDI:
- Onsets and Frames from Google's Magenta team targets solo piano and predicts note starts separately from sustained notes
- Basic Pitch from Spotify is an open-source, lightweight model that handles many instruments and outputs MIDI with pitch bends
Even these models work best on a single instrument. A common, practical trick is to separate the song into stems first (vocals, bass, other) and transcribe each stem separately.
A working monophonic converter in Python
This script uses librosa's pYIN to track pitch and pretty_midi to write the file. Install with pip install librosa pretty_midi.
import numpy as np
import librosa
import pretty_midi
y, sr = librosa.load("melody.wav", sr=22050, mono=True)
# Frame-level pitch tracking (pYIN default hop is 512 samples)
f0, voiced, _ = librosa.pyin(
y, fmin=librosa.note_to_hz("C2"), fmax=librosa.note_to_hz("C6"), sr=sr
)
times = librosa.times_like(f0, sr=sr, hop_length=512)
midi = np.round(librosa.hz_to_midi(f0))
# Group frames with the same rounded pitch into notes
notes, start, cur = [], None, None
for t, m, v in zip(times, midi, voiced):
if v and not np.isnan(m):
if cur is None:
start, cur = t, m
elif m != cur:
notes.append((start, t, cur))
start, cur = t, m
elif cur is not None:
notes.append((start, t, cur))
cur = None
if cur is not None:
notes.append((start, times[-1], cur))
pm = pretty_midi.PrettyMIDI()
inst = pretty_midi.Instrument(program=0)
for s, e, m in notes:
if e - s >= 0.08: # drop blips shorter than 80 ms
inst.notes.append(pretty_midi.Note(velocity=90, pitch=int(m), start=s, end=e))
pm.instruments.append(inst)
pm.write("melody.mid")
print(f"{len(inst.notes)} notes written")
Things you will notice when you try it on a real vocal:
- Vibrato splits notes. A wide vibrato can cross the rounding boundary and create rapid alternating notes. A median filter on the MIDI values before grouping helps.
- Slides create staircases. A scoop into a note shows up as two or three short notes. The 80 ms minimum length removes most of them.
- Reverb and backing tracks confuse pYIN. Feed it the driest, most isolated vocal you can.
Getting a clean input
For a melody you recorded yourself, record dry and close to the mic. For a melody inside an existing song, isolate the vocal first. Open-source separators like Demucs work locally if you are comfortable with Python; I also use the Voicestars stems splitter when I want a quick vocal stem without setting anything up. Either way, transcription accuracy improves noticeably once the drums and chords are gone.
For full songs, try Basic Pitch
If you need chords too, Basic Pitch is the easiest place to start:
from basic_pitch.inference import predict
model_output, midi_data, note_events = predict("piano_stem.wav")
midi_data.write("piano_stem.mid")
Expect to clean up the result in a DAW piano roll. Transcription gets you most of the way, and your ears do the rest.
Takeaways
- Monophonic audio to MIDI is pitch tracking plus note grouping, and runs fine on a laptop
- Polyphonic transcription needs trained models because harmonics of different notes overlap
- Separating stems before transcribing is the single biggest quality win
Disclosure: this article was written in partnership with Voicestars.
Top comments (0)