DEV Community

Cover image for How I built AirStems: remixing a song's stems with your bare hands
Lluis Estape
Lluis Estape

Posted on

How I built AirStems: remixing a song's stems with your bare hands

#showdev #python #audio #machinelearning

🏆 AirStems won the LALAL.AI Special Prize at the Musixmatch Musicathon 2026.

A build log on turning LALAL.AI's separated stems into a real-time, gesture-controlled instrument.

What AirStems is

AirStems lets you conduct and remix a real song with your bare hands in front of a webcam. Raise or fold a finger and a stem — vocals, drums, bass, other — drops in or out. Make a fist and the whole track cuts; open your hand and it all comes back. Your left hand's height opens and closes a low-pass filter, and spreading your fingers adds reverb. Everything is locked to the beat, so a drop always lands musically, and synced lyrics scroll underneath, karaoke-style.

▶ 30-second demo video

It started at the Musixmatch Musicathon and is built on top of my open-source gesture instrument, Aetheric Geometry. The idea was simple: I already had an engine that let my hands play synthesised oscillators, so what if the thing my hands played was the actual vocals, drums and bass of a real track? LALAL.AI is what turns a finished song back into those independent parts, so it became the source the whole instrument plays.

LALAL.AI published an interview about the story and the idea; this post is the technical companion — how the stems come in, how they become a real-time instrument, and the beat-sync engine that ties it together.

System overview

There are two halves, and keeping them apart is the single most important design decision in the project.

Offline preparation (runs once per song, and is allowed to touch the network):

  • LALAL.AI API — separates the track into stems (WAV).
  • Musixmatch — time-synced lyrics (line- and word-level).
  • Cyanite — BPM / key / mood tags.

Each writes its output to disk, indexed by the song's name:

stems/<song>/vocals.wav  drums.wav  bass.wav  other.wav
lyrics/<song>.lrc   (or .richsync.json)
analysis/<song>.json
Enter fullscreen mode Exit fullscreen mode

Real-time app (runs every frame, and never touches the network):

webcam ─► MediaPipe Hands ─► gesture state ─► StemEngine (sounddevice callback) ─► speakers
                                                ▲
                            stems + beat grid, loaded into memory on song load
Enter fullscreen mode Exit fullscreen mode

The live loop only ever reads local files that are already in memory. No part of a network request is on the audio path, and that separation is what keeps the instrument glitch-free while you play.

Deep dive: stem separation → a real-time instrument

This is the core of the project, so I'll spend the most time here.

Getting the stems. The LALAL.AI flow is three steps: upload the file, request a split, poll until it is done, then download the WAVs.

sid = upload(path)                        # POST /upload/  -> source id
for stem in ("vocals", "drum", "bass", "piano"):
    tid  = split(sid, stem=stem)          # POST /split/stem_separator/
    node = wait(tid)                      # poll POST /check/ until success
    download_tracks(_tracks(node), out)   # save <stem>.wav
Enter fullscreen mode Exit fullscreen mode

I request each stem I want and save the result under stems/<song>/. The app is deliberately source-agnostic: it loads whatever WAVs are in that folder, so during testing I can drop in local Demucs output or files from the LALAL.AI website without changing the engine at all.

Turning files into something playable. When a song loads, every stem is read, forced to stereo, resampled to the engine's rate, and — importantly — padded so that all stems share the exact same length. That makes them sample-aligned from time zero, which is what lets me mix them sample-for-sample later.

data, sr = sf.read(path, dtype="float32", always_2d=True)
if data.shape[1] == 1:  data = np.repeat(data, 2, axis=1)   # mono -> stereo
if sr != SAMPLE_RATE:   data = resample(data, sr, SAMPLE_RATE)
...
length = max(len(a) for a in stems.values())
stems  = {k: pad_to(a, length) for k, a in stems.items()}   # align every stem
Enter fullscreen mode Exit fullscreen mode

Everything is loaded fully into memory up front, so the audio callback never has to read from disk.

Which stems, and why. For this instrument, four parts is the sweet spot: vocals, drums, bass, and a fourth ("other", or piano/guitar). Four maps cleanly onto four fingers, and each part is musically meaningful on its own — muting the drums or soloing the vocal are both instantly recognisable moves.

What mattered about the stems. For a live instrument, separation cleanliness matters far more than it does in a studio mix. In a mix, a little bleed between stems is masked because you hear everything together; here, the moment you solo the vocal or fully mute the drums, any bleed is exposed. Full-bandwidth, consistent-loudness stems delivered as lossless WAV are what make muting and soloing sound convincing in real time. As a useful side effect, the separated stems sum back to roughly the original track, so I can mix at close to unity gain without dividing for headroom.

Tradeoffs and honest feedback. Separation is not instant, which is exactly why it belongs in the offline stage — you would never want to run it inside a live session. Two things would have streamlined my integration: a single call that returns all stems as one task, rather than issuing and polling a separate split per stem; and a firmer response schema, since I ended up writing defensive parsing to locate the download URLs across a couple of possible shapes. Neither is a dealbreaker. The separation quality is the part that actually makes the project possible, and that was consistently strong.

The beat-sync engine

This is the most original piece, so it gets real space. The problem it solves: if a hand gesture takes effect the instant you make it, drops and returns land at arbitrary points and it sounds sloppy. The fix is to quantise every change to the song's own beat.

On load, I run librosa's beat tracker on the drums stem (the cleanest source of a pulse) to get a grid of beat timestamps and the BPM:

tempo, beats = librosa.beat.beat_track(y=drums_mono, sr=SR, units="time")
self.beat_times = beats          # array of beat timestamps, in seconds
Enter fullscreen mode Exit fullscreen mode

Then, in the audio callback, a gesture does not change the mix directly. The hands write pending gains, and those are only committed to target gains when a beat boundary falls inside the current audio block:

# does a beat land inside this block?
t0, t1  = pos / SR, (pos + frames) / SR
on_beat = np.any((self.beat_times >= t0) & (self.beat_times < t1))

# commit the hand-wanted gains on the beat (or immediately if quantize is off)
if on_beat or not quantize:
    self.target_gains = pending
Enter fullscreen mode Exit fullscreen mode

So the hand decides what changes, and the beat grid decides when. A key press turns quantisation off for instant, raw control. The same beat crossing also drives a pulse value that the on-screen display flashes with — a small touch that makes the whole interface feel locked to the music.

Hand tracking → mix control

Hand tracking is MediaPipe Hands. I assign one hand as left and one as right, and map them differently.

Right hand = stems. Each finger toggles a stem by comparing the fingertip's Y position to the knuckle below it. The difficulty is jitter: right at the threshold, a finger flickers on and off. So instead of a single line, I use two margins — the finger has to clearly cross to flip, and otherwise it holds its state:

if   tip_y < pip_y - UP_MARGIN:   finger_up = True     # clearly up
elif tip_y > pip_y + DOWN_MARGIN: finger_up = False    # clearly down
# else: hold the previous state  (hysteresis removes the flicker)
Enter fullscreen mode Exit fullscreen mode

Left hand = effects. Wrist height maps to the low-pass filter (lower hand, darker sound), and how open the hand is maps to reverb. Openness is measured in a scale-invariant way, as the mean fingertip-to-knuckle distance divided by palm length, so it behaves the same regardless of how far the hand is from the camera:

openness = mean(dist(tip, knuckle) for each finger) / palm_length
# roughly 0.3 for a fist, 0.9 for an open hand
Enter fullscreen mode Exit fullscreen mode

Both continuous controls are smoothed with a simple exponential moving average, so they feel responsive without following the natural tremor of the hand.

Gesture map

Hand Gesture Controls
Right fingers up / down stem 1–4 in / out
Right fist / open palm full drop / full mix
Left wrist height low-pass filter (down = dark, up = open)
Left open / close hand reverb (open = full, fist = dry)
Keys space · b · n play/pause · beat-sync on/off · next song

Audio engine internals

The engine is a single sounddevice output stream with a stereo, block-based callback. This is where the audio-focused details live.

Click-free mutes. Toggling a stem instantly would click, because the gain jumps. Instead, every block ramps each stem's gain from its current value to its target across the length of the block:

for name, audio in stems.items():
    ramp = np.linspace(curr[name], target[name], frames)   # per-block gain ramp
    mix += audio[idx] * ramp[:, None]
    curr[name] = target[name]
Enter fullscreen mode Exit fullscreen mode

The effects chain is reused directly from Aetheric Geometry: a first-order IIR low-pass driven by hand height, a Schroeder reverb (four comb filters) driven by hand spread, and a light tremolo. The reverb's wet level is smoothed inside the callback, so opening your hand fades the space in rather than snapping it on:

cutoff = 200.0 * (8000.0 / 200.0) ** filter_bright    # hand height -> cutoff in Hz
reverb_smooth += 0.08 * (reverb_wet - reverb_smooth)  # ease the wet level in/out
Enter fullscreen mode Exit fullscreen mode

Parameters cross from the main thread to the audio thread under a short lock, copied once at the top of each callback, so a gesture update never tears a block mid-render.

What was hard

  • Making it feel musical rather than glitchy. The beat-sync quantiser was the single biggest change: before it, hand-driven changes sounded messy; after it, everything lands in time.
  • Removing gesture flicker. Raw fingertip thresholds jitter, and the two-margin hysteresis above was what made stem toggles trustworthy.
  • Real-time safety. The callback copies its parameters under a lock and avoids heavy allocation. The per-sample IIR and comb-reverb loops are the CPU hot path, so block size becomes the main knob for trading latency against stability on a given machine.
  • Keeping the network off the audio path. Doing all separation, lyrics and analysis ahead of time, and loading the stems fully into memory, is what keeps playback from ever stalling.

Links & credit

Questions, or a gesture you think I should map next? I'd love to hear it in the comments.

Top comments (0)