Your Hand Is a Continuous Signal. A Scale Is Not. That Gap Is the Whole Problem.

Gesture Synth Weld — Sat, 05 Sep 2026 02:47:07 +0000

A webcam hand tracker hands you a position, thirty or sixty times a second, as a float
between 0 and 1. A musical scale hands you seven notes per octave. Building a
browser hand-gesture synthesizer is mostly the work of
getting from the first thing to the second thing without it sounding like a fax machine.

I want to write down the specific failure modes, because they are not obvious until you
have shipped one and watched somebody else use it.

The naive mapping is unusable, and it is unusable immediately

The obvious first version is one line:

const noteIndex = Math.floor(handY * scale.length);

Map vertical hand position onto the scale, floor it, play that note. This works
perfectly in your head and falls apart the moment a real hand is in front of the camera.

The reason is that a hand does not hold still. Even braced against a desk, the landmark
estimate jitters — partly real micro-movement, partly the model's own frame-to-frame
noise. If your hand happens to sit near a boundary between two notes, that jitter
straddles the boundary, and Math.floor faithfully converts a 2-pixel tremor into a
stream of alternating note-on events. You get a trill you did not ask for, at whatever
rate your tracker runs.

This is not a tuning problem you can smooth away with a lower threshold. It is
structural: you are sampling a noisy continuous signal with a hard comparator.

Hysteresis is the fix, and it costs you something real

The standard answer is hysteresis: make the boundary for entering a note different from
the boundary for leaving it. Once note 4 is playing, you have to move meaningfully past
the 4/5 line before note 5 takes over. A Schmitt trigger, borrowed from analog
electronics and applied to pitch.

It works. It also introduces a genuine trade-off that no amount of cleverness removes:

narrow deadband   →  responsive, but chatters near boundaries
wide deadband     →  stable, but notes feel "sticky" and fast runs get eaten

Where you land depends on what you think the instrument is for. A theremin-style
continuous mode wants almost no quantization at all — the sliding pitch is the point.
A scale-locked mode wants a deadband wide enough that a beginner can hold a note without
concentrating. These are different instruments wearing the same interface, which is why
the scale modes are a mode switch rather than a
slider: they need different deadbands, not different values of one deadband.

Latency is a budget, and the tracker spends most of it

End-to-end latency for a gesture instrument is roughly:

camera exposure + capture     ~16-33 ms   (frame rate bound)
hand landmark inference        ~10-30 ms   (model + device bound)
your mapping logic             <1 ms
Web Audio scheduling           ~5-20 ms   (buffer size bound)

Musicians start noticing around 20 ms and start compensating around 40 ms. You are
already at the edge before you have written a line of your own code, and the two big
line items are not yours to optimise — they belong to the camera pipeline and the
inference model.

The practical consequences:

Do not add smoothing filters casually. A 5-frame moving average on the landmark position is 80 ms at 60fps. You just doubled your latency to fix jitter that hysteresis fixes for free.
Do not schedule notes "now". Web Audio's currentTime plus a small fixed lookahead is more stable than firing immediately, because it decouples your note timing from whenever the render quantum happens to land.
Frame rate matters more than model accuracy. A slightly worse landmark at 60fps plays better than a slightly better one at 24fps. For this application, precision is cheaper than latency.

MIDI export is a different clock, and mixing them up will bite you

Recording the performance for export looks trivial — push {note, time} on every event —
until you notice you have three candidate clocks in the room:

performance.now() — wall clock, drifts against audio
audioCtx.currentTime — the audio clock, what you actually heard
MIDI ticks — musical time, relative to a tempo you have to choose

If you record wall-clock timestamps and write them out as MIDI ticks, the file will be
subtly wrong in a way that is very hard to hear in isolation and very obvious once it is
sitting next to a drum track in a DAW. The audio clock is the correct source, because
it is the one that produced the sound the performer was responding to.

The tempo choice is the other trap. A gesture performance has no metronome — the player
was not thinking in bars. Writing the file at an arbitrary 120 BPM means every note
lands on a fractional tick and the DAW's quantize function becomes useless. Picking a
high tick resolution and writing the true durations is more honest: the
exported .mid then represents what was played,
and the DAW can impose a grid afterwards if the user wants one.

The keyboard fallback is not a consolation prize

Every camera-based instrument needs a no-camera mode, and the reflex is to treat it as
an accessibility checkbox. It is more useful than that:

It is the only way to A/B your audio engine without the tracker in the signal path. When something sounds wrong, keyboard mode tells you within seconds whether it is the synth or the hand tracking.
It works in the situations where a webcam does not — bad lighting, a shared office, a locked-down machine, someone who does not want to grant camera permission to a page they found thirty seconds ago.

That last one is worth dwelling on. Asking for camera access is a large request. A page
that does something useful before it asks, and that never uploads a frame, is making a
much smaller one. Local-only processing here is not a privacy feature bolted on — it is
what makes the permission prompt reasonable in the first place.

What I would tell someone starting one

Build the audio engine first and drive it from the keyboard. Get it sounding good with zero tracking involved.
Add the tracker as a second input source, not as the foundation.
Put hysteresis in before you put smoothing in. You will probably not need smoothing.
Record on the audio clock from day one. Retrofitting this is miserable.
Decide early whether you are building a theremin or a keyboard. Trying to be both with one deadband produces something that is bad at both.

The interesting part of this problem was never the machine learning — the hand tracking
is a solved component you import. The interesting part is that a hand is an analog
controller with no detents, and music mostly is not. Everything that makes
a gesture instrument feel like an instrument rather than
a demo lives in how you handle that mismatch.