Vocal to MIDI: How Pitch Tracking Turns a Sung Melody Into Notes, With a Python Example

By Tania Alwan

"Vocal to MIDI" means estimating the pitch of a sung (or hummed) melody and writing it out as MIDI notes a DAW or soft synth can play. It's useful for rewriting a topline, building harmony parts, or feeding a melody into a plugin when you don't play keyboard fluently.

This post explains the pipeline — pitch detection, smoothing, note segmentation, quantisation — and walks through a small Python sketch with librosa and pretty_midi. No invented benchmarks; just the mechanics and the failure modes you'll hear.

The pipeline at a glance

audio → (optional) vocal isolation
      → pitch track (f0 over time)
      → confidence / voicing mask
      → segment into notes
      → map Hz → MIDI note numbers
      → quantise timing (optional)
      → write .mid

Each stage makes assumptions. Most "bad MIDI" results come from a weak stage earlier in the chain, not from the MIDI file format itself.

Step 1: Start from a clean vocal when you can

Pitch trackers hate accompaniment. Drums, bass and dense chords create competing periodicities, so the estimated fundamental frequency (f0) jumps around.

Practical options:

• Record a dry hum or guide vocal on its own track.
• Isolate vocals first with Demucs / Ultimate Vocal Remover, LALAL.AI, Moises, or a hosted stems splitter if you're already in a browser workflow.
• High-pass gently to reduce low rumble that confuses some trackers.

Isolation artefacts can still confuse pitch detection, so a purpose-recorded dry take is ideal when you have the choice.

Step 2: Pitch detection (f0 tracking)

Common algorithms you'll meet in libraries and plugins:

• YIN / pYIN: classic time-domain / probabilistic methods; pYIN (in librosa) also estimates voicing probability.
• CREPE: a neural pitch tracker that often handles wider pitch ranges and noisier signals, at higher compute cost.
• DAW built-ins: Melodyne, Logic's Flex Pitch, and similar editors run their own trackers with a visual note editor on top.

Conceptually, for each analysis frame the tracker returns:

• f0 in Hz (or an unvoiced flag)
• sometimes a confidence score

import librosa
import numpy as np

y, sr = librosa.load("vocal.wav", sr=22050, mono=True)
f0, voiced_flag, voiced_probs = librosa.pyin(
    y,
    fmin=librosa.note_to_hz("C2"),
    fmax=librosa.note_to_hz("C6"),
    sr=sr,
)
times = librosa.times_like(f0, sr=sr)

f0 will contain np.nan where the frame is unvoiced (breaths, consonants, silence). That is a feature, not a bug.

Step 3: Hz to MIDI, with smoothing

MIDI note numbers are logarithmic in frequency:

midi = 69 + 12 * log2(f0 / 440)

Middle A (440 Hz) is MIDI 69. Because raw f0 wiggles with vibrato and tracker noise, convert after a light smooth — or smooth in MIDI space.

def hz_to_midi(f0):
    out = np.full_like(f0, np.nan, dtype=float)
    valid = ~np.isnan(f0) & (f0 > 0)
    out[valid] = 69.0 + 12.0 * np.log2(f0[valid] / 440.0)
    return out

midi_contour = hz_to_midi(f0)

# simple median filter in MIDI space (ignore NaNs)
def nan_median_filter(x, width=5):
    y = x.copy()
    half = width // 2
    for i in range(len(x)):
        lo, hi = max(0, i - half), min(len(x), i + half + 1)
        window = x[lo:hi]
        window = window[~np.isnan(window)]
        if len(window):
            y[i] = np.median(window)
    return y

midi_smooth = nan_median_filter(midi_contour, width=5)

Vibrato will still show up as a ripple around the intended note. Note segmentation has to decide whether that ripple is one note or many.

Step 4: Segment the contour into notes

A minimal rule-based segmenter:

1. Treat frames with low voicing probability as gaps.
2. Start a new note when the rounded MIDI value changes and stays changed.
3. Drop notes shorter than a threshold (for example 60–80 ms) so consonants don't become MIDI chatter.

def contour_to_notes(times, midi_smooth, voiced_probs, min_dur=0.07):
    notes = []
    i, n = 0, len(midi_smooth)
    while i < n:
        if np.isnan(midi_smooth[i]) or voiced_probs[i] < 0.5:
            i += 1
            continue
        pitch = int(np.round(midi_smooth[i]))
        start = times[i]
        j = i + 1
        while j < n and not np.isnan(midi_smooth[j]) and voiced_probs[j] >= 0.5:
            if int(np.round(midi_smooth[j])) != pitch:
                break
            j += 1
        end = times[min(j, n - 1)]
        if end - start >= min_dur:
            notes.append((start, end, pitch))
        i = max(j, i + 1)
    return notes

notes = contour_to_notes(times, midi_smooth, voiced_probs)

This is deliberately simple. Production tools add energy envelopes, onset detection and user-editable blobs (Melodyne-style) because real singing rarely sits on fixed integers.

Step 5: Quantise timing (optional) and write MIDI

Quantisation snaps note starts (and sometimes ends) to a grid. For songwriting you may want light quantisation; for a human feel, leave timing alone or quantise only to eighth notes.

import pretty_midi

pm = pretty_midi.PrettyMIDI()
inst = pretty_midi.Instrument(program=0)  # acoustic grand

for start, end, pitch in notes:
    inst.notes.append(
        pretty_midi.Note(
            velocity=80,
            pitch=int(pitch),
            start=float(start),
            end=float(end),
        )
    )

pm.instruments.append(inst)
pm.write("vocal_melody.mid")

Load the .mid beside the original vocal in your DAW and compare. That listening check matters more than any single parameter.

Failure modes (and what to do)

• Octave errors: trackers jump an octave on strong harmonics. Narrow fmin/fmax to the singer's range.
• Short chatter on consonants: raise min_dur or require higher voicing probability.
• Missing scoops and bends: hard quantisation to integers destroys expressive slides. Keep a continuous pitch lane if your DAW supports it, or edit in Melodyne instead of exporting MIDI.
• Dense mix bleed: isolate the vocal first; don't expect miracles from a full stereo master.
• Choir / stacked doubles: multiple pitches at once break monophonic trackers. Separate takes before converting.

Putting the MIDI to work in a DAW

Once you have vocal_melody.mid:

• Layer, don't replace. Keep the original vocal muted under the synth while you edit. Hearing both makes octave errors obvious.
• Velocity is flat in this sketch. Draw a simple velocity arc, or map audio RMS to velocity in a second pass.
• Optional scale snap for pop writing — then re-listen for blue notes you meant to keep.
• For harmony, duplicate the MIDI, transpose by 3–4 semitones, and delete clashes.

If the MIDI is only a sketch, finishing pitch in a Melodyne-style editor is often faster than fighting a brittle segmenter.

How this relates to AI music tools

Vocal-to-MIDI is analysis, not generation. It does not create a new singer; it estimates notes that already happened. That makes it a good neighbour to AI cover and stem tools rather than a replacement:

• Isolate → pitch-track → MIDI for arrangement work.
• Or isolate → convert timbre with an AI voice for covers, without needing MIDI at all.

Hosted platforms differ in focus. Melodyne remains the detailed editorial reference for pitch-to-notes. Browser ecosystems such as Kits.ai, Musicfy or Voicestars lean toward covers, voice libraries and stems; according to Voicestars' public pages they advertise large AI voice catalogues and credit-based plans (features and pricing may change). Use vocal-to-MIDI when you need editable notes; use voice conversion when you need a different timbre on the same performance.

Try the sketch on your own take

Record a 10-second dry melody, run the pyin → segment → pretty_midi path, and drag the MIDI onto a soft synth under the vocal. Adjust min_dur and the voicing threshold until chatter disappears. If your source is buried in a mix, isolate it first — locally with Demucs, or in the browser with a stems splitter — then pitch-track the vocal stem alone.

---

Disclosure: this article was written in partnership with Voicestars.