DEV Community: Lluis Estape

How I built AirStems: remixing a song's stems with your bare hands

Lluis Estape — Thu, 03 Sep 2026 11:50:30 +0000

🏆 AirStems won the LALAL.AI Special Prize at the Musixmatch Musicathon 2026.

A build log on turning LALAL.AI's separated stems into a real-time, gesture-controlled instrument.

What AirStems is

AirStems lets you conduct and remix a real song with your bare hands in front of a webcam. Raise or fold a finger and a stem — vocals, drums, bass, other — drops in or out. Make a fist and the whole track cuts; open your hand and it all comes back. Your left hand's height opens and closes a low-pass filter, and spreading your fingers adds reverb. Everything is locked to the beat, so a drop always lands musically, and synced lyrics scroll underneath, karaoke-style.

▶ 30-second demo video

It started at the Musixmatch Musicathon and is built on top of my open-source gesture instrument, Aetheric Geometry. The idea was simple: I already had an engine that let my hands play synthesised oscillators, so what if the thing my hands played was the actual vocals, drums and bass of a real track? LALAL.AI is what turns a finished song back into those independent parts, so it became the source the whole instrument plays.

LALAL.AI published an interview about the story and the idea; this post is the technical companion — how the stems come in, how they become a real-time instrument, and the beat-sync engine that ties it together.

System overview

There are two halves, and keeping them apart is the single most important design decision in the project.

Offline preparation (runs once per song, and is allowed to touch the network):

LALAL.AI API — separates the track into stems (WAV).
Musixmatch — time-synced lyrics (line- and word-level).
Cyanite — BPM / key / mood tags.

Each writes its output to disk, indexed by the song's name:

stems/<song>/vocals.wav  drums.wav  bass.wav  other.wav
lyrics/<song>.lrc   (or .richsync.json)
analysis/<song>.json

Real-time app (runs every frame, and never touches the network):

webcam ─► MediaPipe Hands ─► gesture state ─► StemEngine (sounddevice callback) ─► speakers
                                                ▲
                            stems + beat grid, loaded into memory on song load

The live loop only ever reads local files that are already in memory. No part of a network request is on the audio path, and that separation is what keeps the instrument glitch-free while you play.

Deep dive: stem separation → a real-time instrument

This is the core of the project, so I'll spend the most time here.

Getting the stems. The LALAL.AI flow is three steps: upload the file, request a split, poll until it is done, then download the WAVs.

sid = upload(path)                        # POST /upload/  -> source id
for stem in ("vocals", "drum", "bass", "piano"):
    tid  = split(sid, stem=stem)          # POST /split/stem_separator/
    node = wait(tid)                      # poll POST /check/ until success
    download_tracks(_tracks(node), out)   # save <stem>.wav

I request each stem I want and save the result under stems/<song>/. The app is deliberately source-agnostic: it loads whatever WAVs are in that folder, so during testing I can drop in local Demucs output or files from the LALAL.AI website without changing the engine at all.

Turning files into something playable. When a song loads, every stem is read, forced to stereo, resampled to the engine's rate, and — importantly — padded so that all stems share the exact same length. That makes them sample-aligned from time zero, which is what lets me mix them sample-for-sample later.

data, sr = sf.read(path, dtype="float32", always_2d=True)
if data.shape[1] == 1:  data = np.repeat(data, 2, axis=1)   # mono -> stereo
if sr != SAMPLE_RATE:   data = resample(data, sr, SAMPLE_RATE)
...
length = max(len(a) for a in stems.values())
stems  = {k: pad_to(a, length) for k, a in stems.items()}   # align every stem

Everything is loaded fully into memory up front, so the audio callback never has to read from disk.

Which stems, and why. For this instrument, four parts is the sweet spot: vocals, drums, bass, and a fourth ("other", or piano/guitar). Four maps cleanly onto four fingers, and each part is musically meaningful on its own — muting the drums or soloing the vocal are both instantly recognisable moves.

What mattered about the stems. For a live instrument, separation cleanliness matters far more than it does in a studio mix. In a mix, a little bleed between stems is masked because you hear everything together; here, the moment you solo the vocal or fully mute the drums, any bleed is exposed. Full-bandwidth, consistent-loudness stems delivered as lossless WAV are what make muting and soloing sound convincing in real time. As a useful side effect, the separated stems sum back to roughly the original track, so I can mix at close to unity gain without dividing for headroom.

Tradeoffs and honest feedback. Separation is not instant, which is exactly why it belongs in the offline stage — you would never want to run it inside a live session. Two things would have streamlined my integration: a single call that returns all stems as one task, rather than issuing and polling a separate split per stem; and a firmer response schema, since I ended up writing defensive parsing to locate the download URLs across a couple of possible shapes. Neither is a dealbreaker. The separation quality is the part that actually makes the project possible, and that was consistently strong.

The beat-sync engine

This is the most original piece, so it gets real space. The problem it solves: if a hand gesture takes effect the instant you make it, drops and returns land at arbitrary points and it sounds sloppy. The fix is to quantise every change to the song's own beat.

On load, I run librosa's beat tracker on the drums stem (the cleanest source of a pulse) to get a grid of beat timestamps and the BPM:

tempo, beats = librosa.beat.beat_track(y=drums_mono, sr=SR, units="time")
self.beat_times = beats          # array of beat timestamps, in seconds

Then, in the audio callback, a gesture does not change the mix directly. The hands write pending gains, and those are only committed to target gains when a beat boundary falls inside the current audio block:

# does a beat land inside this block?
t0, t1  = pos / SR, (pos + frames) / SR
on_beat = np.any((self.beat_times >= t0) & (self.beat_times < t1))

# commit the hand-wanted gains on the beat (or immediately if quantize is off)
if on_beat or not quantize:
    self.target_gains = pending

So the hand decides what changes, and the beat grid decides when. A key press turns quantisation off for instant, raw control. The same beat crossing also drives a pulse value that the on-screen display flashes with — a small touch that makes the whole interface feel locked to the music.

Hand tracking → mix control

Hand tracking is MediaPipe Hands. I assign one hand as left and one as right, and map them differently.

Right hand = stems. Each finger toggles a stem by comparing the fingertip's Y position to the knuckle below it. The difficulty is jitter: right at the threshold, a finger flickers on and off. So instead of a single line, I use two margins — the finger has to clearly cross to flip, and otherwise it holds its state:

if   tip_y < pip_y - UP_MARGIN:   finger_up = True     # clearly up
elif tip_y > pip_y + DOWN_MARGIN: finger_up = False    # clearly down
# else: hold the previous state  (hysteresis removes the flicker)

Left hand = effects. Wrist height maps to the low-pass filter (lower hand, darker sound), and how open the hand is maps to reverb. Openness is measured in a scale-invariant way, as the mean fingertip-to-knuckle distance divided by palm length, so it behaves the same regardless of how far the hand is from the camera:

openness = mean(dist(tip, knuckle) for each finger) / palm_length
# roughly 0.3 for a fist, 0.9 for an open hand

Both continuous controls are smoothed with a simple exponential moving average, so they feel responsive without following the natural tremor of the hand.

Gesture map

Hand	Gesture	Controls
Right	fingers up / down	stem 1–4 in / out
Right	fist / open palm	full drop / full mix
Left	wrist height	low-pass filter (down = dark, up = open)
Left	open / close hand	reverb (open = full, fist = dry)
Keys	space · b · n	play/pause · beat-sync on/off · next song

Audio engine internals

The engine is a single sounddevice output stream with a stereo, block-based callback. This is where the audio-focused details live.

Click-free mutes. Toggling a stem instantly would click, because the gain jumps. Instead, every block ramps each stem's gain from its current value to its target across the length of the block:

for name, audio in stems.items():
    ramp = np.linspace(curr[name], target[name], frames)   # per-block gain ramp
    mix += audio[idx] * ramp[:, None]
    curr[name] = target[name]

The effects chain is reused directly from Aetheric Geometry: a first-order IIR low-pass driven by hand height, a Schroeder reverb (four comb filters) driven by hand spread, and a light tremolo. The reverb's wet level is smoothed inside the callback, so opening your hand fades the space in rather than snapping it on:

cutoff = 200.0 * (8000.0 / 200.0) ** filter_bright    # hand height -> cutoff in Hz
reverb_smooth += 0.08 * (reverb_wet - reverb_smooth)  # ease the wet level in/out

Parameters cross from the main thread to the audio thread under a short lock, copied once at the top of each callback, so a gesture update never tears a block mid-render.

What was hard

Making it feel musical rather than glitchy. The beat-sync quantiser was the single biggest change: before it, hand-driven changes sounded messy; after it, everything lands in time.
Removing gesture flicker. Raw fingertip thresholds jitter, and the two-margin hysteresis above was what made stem toggles trustworthy.
Real-time safety. The callback copies its parameters under a lock and avoids heavy allocation. The per-sample IIR and comb-reverb loops are the CPU hot path, so block size becomes the main knob for trading latency against stability on a given machine.
Keeping the network off the audio path. Doing all separation, lyrics and analysis ahead of time, and loading the stems fully into memory, is what keeps playback from ever stalling.

Links & credit

Live page: https://lluisestape-upc.github.io/AirStems/
Demo video: https://youtu.be/BnNcGabujgc
Source: https://github.com/lluisestape-upc/AirStems
The story (interview): https://www.lalal.ai/blog/how-airstems-uses-lalal-ai-api/
Built on my open-source gesture instrument Aetheric Geometry.
Stems by LALAL.AI, lyrics by Musixmatch, analysis by Cyanite.

Questions, or a gesture you think I should map next? I'd love to hear it in the comments.

Building BoardRoom on Qwen Cloud for the Global AI Hackathon Series. (Track 3: Agent Society)

Lluis Estape — Sat, 18 Jul 2026 13:01:28 +0000

I built a society of AI agents to review circuit boards. Then I tried to prove it was better, and nearly fooled myself three times.

Every hardware engineer knows the specific dread of clicking "order boards." Three weeks
and a few hundred euros later, a package arrives, and you discover you forgot a
decoupling capacitor, or your I²C bus has no pull-ups. Nothing to do but respin.

So for the Qwen Cloud hackathon I built BoardRoom: not one AI reviewer, but a
society of five — power integrity, signal integrity, connectivity/ERC, DFM & layout,
and firmware bring-up — that inspect a real KiCad project through the KiCad MCP server,
file findings that must cite tool evidence, and argue with each other under a Moderator
before signing off.

The build went fine. It's what happened when I tried to prove the society was better
than a single big agent that turned out to be the interesting part. Three separate times,
my benchmark was quietly lying in my favor. Here's each one, because I think the bugs are
more instructive than the architecture.

The setup

The measurable claim Track 3 asks for is "a demonstrable efficiency gain over
single-agent baselines." So:

Corpus: stock KiCad demo boards.
Seeding: a script injects reproducible defects with a ground-truth manifest — removed decoupling cap, swapped SDA/SCL, missing I²C pull-ups, floated enable pin, renamed power rail. One board (a pure-analog valve preamp) gets no defects, as an honesty check that nobody invents findings.
Two configs, same inputs, same token gateway: the society (five cheap specialists on qwen-flash/qwen3-vl/qwen3-coder, one qwen3-max Moderator) versus a baseline of one qwen3-max agent with all the tools and a generalist prompt.

Then I ran it. Recall: 0.00. The society caught zero of the seeded defects.

Lie #1: my agents were right, my scoring was wrong

Rather than tune anything, I dumped the actual findings. And there it was:

[critical] connectivity_erc: Hierarchical label VCC_PIC has no matching sheet pin
                             in the parent sheet, creating an unconnected rail

That's exactly the defect I'd seeded. The society found it. The scorer called it a miss.

Why? The matcher compares structured fields — affected_nets, affected_components —
and the agents had left them empty, naming the net only in the prose of the claim.

That's not a benchmark artifact. That's a product bug I'd have shipped: the report's
headline visualization — a blast-radius graph linking findings to the nets and parts they
touch — is built entirely from those fields. My centerpiece visual was silently empty
and I hadn't noticed, because I'd been reading the claims, which looked great.

The prompts already asked for those fields politely. qwen-flash ignored it. Making the
final instruction blunt ("a claim that names a net while leaving these arrays empty is
INVALID — the impact graph is built from these fields, not from your prose") took
structured coverage from 0/9 findings to 9/15, and recall from 0.00 to 0.50.

Lesson: when a model "ignores" a field, check whether anything downstream actually
depends on it. Mine did, and the benchmark found the bug my eyes didn't.

Lie #2: I'd accidentally built a strawman

Now the baseline. It scored 0.00, produced zero findings, and burned 6.8K tokens in 29
seconds. I could have shipped that. "Single agent finds nothing, society wins" — a
beautiful chart.

Except: 6.8K tokens is nothing. A generalist reviewing a whole board can't be done in
29 seconds. So I looked.

I'd named the baseline agent moderator so it would inherit the wildcard "all tools"
entry from my default allowlist. But registry.yaml overrides the moderator down to
three overview tools — because in the society, the chair doesn't do analysis. My
"single agent with all 24 tools" had been reviewing boards nearly blind.

A tool-starved baseline isn't a baseline. It's a strawman, and it's exactly the thing
a judge should nuke a submission for. Fixed: the baseline now builds its allowlist from
defaults, genuinely getting every tool. Its token use went 6.8K → 63K.

Lie #3: I was grading the baseline on handwriting

Still 0.00, but now doing real work. Fine — maybe one agent really does drown in 24
tools? That's a great narrative. I nearly wrote it.

Instead I printed its raw output:

{ "id": "B-001",
  "title": "Hierarchical label mismatch for VCC_PIC",
  "description": "...breaking connectivity.",
  "evidence_ids": ["EV-0005"] }

The baseline found the defect too. Every finding was thrown away at the schema
boundary because it wrote title/description/evidence_ids instead of
claim/evidence/recommendation.

And whose fault was that? Mine. Each specialist prompt carries a full worked JSON example
of the finding shape. The baseline prompt I'd dashed off didn't. I was measuring prompt
quality and calling it architecture.

I gave the baseline the same worked example the specialists get. I also normalized net
names in the matcher — KiCad legitimately calls the same net VCC_PIC or
/pic_sockets/VCC_PIC, and a correct answer shouldn't lose on a naming convention.

Notice the pattern: all three bugs pointed the same direction. None of them made the
society look worse. That's not coincidence — it's what motivated reasoning looks like
from the inside. You stop debugging when the number agrees with you.

What the honest numbers say

Six boards, twelve seeded defects, two independent runs per config, because LLMs are
nondeterministic and one run isn't a measurement.

	society	baseline (1× qwen3-max, all tools)
Seeded-defect recall (mean)	0.29	0.21
Cost per corpus	$0.147	$0.429
Prompt tokens	792K	312K
Findings surfaced	49–56	6–10
Hallucination rate	0.00	0.00

Read the token row and the cost row together, because that's the whole story:

The society burns 2.5× MORE tokens and costs 2.9× LESS.

Five cheap specialists doing more total work are dramatically cheaper than one
expensive generalist doing less. The win isn't "multi-agent is smarter." It's that
decomposition lets you route by role instead of by model size — you only pay
flagship prices for the one job that actually needs a flagship: adjudicating conflicts.
That number reproduced to within 2% across runs.

And the thing I'm not claiming: the society led on recall in both runs, but by exactly
one defect out of twelve. With n=2, that's noise — I watched the same config score
0.50 and 0.00 on the same board across runs. So the honest claim is "comparable, possibly
slightly better," and that's what the README says.

The most useful result is the one that makes my project look bad

Absolute recall is low for both configs — 0.21 to 0.29. I could have quietly picked
easier defects. Here's why it's low instead:

Most of the defects I seeded are invisible to every tool that exists. Removing a
decoupling cap creates an absence — no KiCad tool reports "a part that should be here
isn't." A swapped SDA/SCL is electrically valid, so ERC has nothing to say. The only
class anything reliably caught is rail-rename/floating-pin, because ERC and net listings
actually surface it.

That's not an indictment of the agents. It's a finding about EDA tooling: the reasoning
layer is now ahead of the observability layer. These models can absolutely reason about
a missing bypass cap — they just have no instrument that reports its absence. The
bottleneck isn't intelligence. It's that we never built the tool, because no human needed
one to eyeball a schematic.

I think that's the most interesting thing I learned, and it only showed up because I
reported a bad number instead of engineering around it.

What I'd tell someone starting one of these

Evidence gating works. Every finding must cite a real cached tool-call ID, and uncited claims are rejected at the boundary — not discouraged in a prompt. Hallucination rate: 0.00 across every run, both configs. Make it structural and you stop arguing with the model about honesty.
Enforce scope in code, not prose. Each specialist's tool allowlist is enforced before the call. Prompts are suggestions; allowlists are physics.
Route by role, not by size. This is where the 2.9× lives.
Debug the results that flatter you. All three of my bugs produced better numbers. If I'd stopped at the first good chart, I'd have submitted a rigged benchmark with a straight face.

One last confession: BoardRoom's negotiation engine — bounded two-round debates,
evidence-cited rulings, fully unit-tested — never fired on real data. Across the whole
corpus, no two specialists ever produced conflicting findings. Their scopes are narrow
enough that they rarely collide. The repo ships a clearly-labeled synthetic fixture to
demonstrate the viewer, and says so in the README, because the alternative is faking a
fight and calling it a demo.

That's a real limitation of the design, discovered by measuring instead of assuming. Which
is, I think, the actual point.

BoardRoom is open source (MIT): https://github.com/lluisestape-upc/boardroom — built on
Qwen models via Alibaba Cloud Model Studio, with the KiCad MCP server. Benchmark method,
raw numbers and limitations: docs/BENCHMARK.md.