Building an open-source webcam-to-VRM studio: lessons from monocular tracking

I have been working on VTuber Studio Pro, a free, open-source app for animating a VRM avatar from an ordinary webcam. The current Windows portable release is v1.2.5. It supports VRM 0.x and 1.0 models and can provide a transparent avatar view for an OBS workflow.

The interesting part has been making a single-camera setup usable without pretending it can measure everything a depth camera can.

What the camera can observe

A webcam gives us a 2D image. Face, shoulder, arm, hand and finger landmarks can be estimated from that image, then mapped onto the avatar rig. The application runs the tracking locally. For someone trying to get started with a VRM model, this keeps the hardware requirement low: a webcam and a computer.

Where the guesswork begins

Depth is ambiguous in a monocular image. A hand moving toward the lens can look similar to a change in scale or body pose. I damp depth motion to reduce distracting jumps, but this necessarily trades responsiveness for stability. When an arm or shoulder leaves the frame, the pose also needs a fallback rather than trying to invent an exact position. Lighting and face occlusion affect expression tracking too.

These are current limitations, not features I would claim to have solved. If you test the app, I would especially like to hear about models with unusual rigs, hands near the camera, and tracking in imperfect lighting.

Try it and report a concrete failure

1. Download the portable Windows v1.2.5 release: https://github.com/ydjdydjfbfuduxbxiehsjd-source/vtuber-studio/releases/tag/v1.2.5
2. Load your own VRM 0.x or 1.0 avatar and enable your webcam.
3. Try face movement, both arms and fingers, then a transparent OBS scene.
4. In a GitHub issue, include your VRM version, camera/lighting setup, the motion that failed and what you expected.

The source and issue tracker are here: https://github.com/ydjdydjfbfuduxbxiehsjd-source/vtuber-studio

I am sharing this as an early release and would value reproducible feedback. If you have built webcam-driven avatar tools, which stability tradeoff has been hardest for you?