Real-Time Webcam-to-VRM Retargeting with MediaPipe Holistic: 17.3 FPS on M1 Max

Hello, everyone.

If I raise my hand in front of a webcam, can a 3D avatar raise its hand too? More specifically, can a browser do this without dedicated motion-capture hardware?

Today, I am testing MediaPipe Holistic as a way to estimate the body, face, and both hands, then retarget the result to a VRM 1.0 avatar in real time.

The short answer is that an Apple M1 Max processed a 1280×720 webcam stream at an effective 17.3 FPS using CPU/WASM. Every one of the 141 frames with a detected pose updated the VRM skeleton. Body tracking worked, but this short measurement did not establish stable simultaneous capture of the face and both hands.

What Is MediaPipe Holistic?

MediaPipe Holistic is a pipeline that Google announced on December 10, 2020 for estimating the body, face, and both hands from one camera. A landmark is a point representing a location such as a shoulder, elbow, or fingertip. The announced version returned 543 points: 33 for the pose, 468 for the face, and 21 for each hand. The bundle used here includes Face Mesh V2 with ten additional iris points, bringing the face output to 478 and the total to 553.

The 13.7 MB holistic_landmarker.task used here contains seven TensorFlow Lite model files. Their roles can be summarized as follows.

Model	Role
pose detector	Finds the person in the image
pose landmarks detector	Estimates 33 body points and 3D coordinates
face detector	Finds the face region
face landmarks detector	Estimates 478 facial points, including the irises
face blendshapes	Produces 52 coefficients such as eye blink and jaw opening
hand ROI refinement	Corrects the hand crop for a closer look
hand landmarks detector	Estimates 21 points and is shared by the left and right hands

Holistic first uses the body position to propose approximate face and hand regions. It then crops those areas from the full-resolution input. This multi-stage design avoids trying to read small fingertips from the already-downscaled whole-body image.

I pinned the float16 task bundle updated on December 21, 2023. Its component documentation dates BlazePose to April 2021, Hand Tracking to October 2021, Face Mesh V2 to September 2022, and Blendshape V2 to November 2022. The model cards for BlazePose GHUM 3D, Hand Tracking, Face Mesh V2, and Blendshape V2 all specify the Apache License 2.0. MediaPipe itself and @mediapipe/tasks-vision also use Apache License 2.0.

For the lab's bundled avatar and the measurement, I used Seed-san, an official sample model by VirtualCast, Inc. Its embedded settings point to the VRM Public License 1.0 and require credit. Redistribution and redistribution of modified data are allowed; its avatar permission is everyone, and its commercial-use setting is corporation.

The demo video uses the tool's custom-VRM loading feature to display AvatarSample_A, a VRoid Studio sample model. Its official conditions of use allow free commercial and noncommercial use without attribution. The model is not CC0, and its copyright has not been waived. The @pixiv/three-vrm and Three.js libraries use the MIT License.

What I Tested

The experiment asks five questions.

• Can the browser version of Holistic Landmarker process a webcam stream in real time?
• Can 33 body landmarks be converted into VRM 1.0 bone rotations?
• Can 21 landmarks per hand drive the wrists and fingers?
• Can facial coefficients be converted into VRM expressions?
• Does smoothing keep the response interactive while reducing jitter?

The complete code and measurement notes are available in the mediapipe-holistic-vrm lab in kiarina/labs.

Reproducing the Lab

You will need mise, a browser with WebGL and camera-input support, and an internet connection for the first download. mise installs Node.js 22.22.0.

git clone --depth 1 --filter=blob:none --sparse \
  https://github.com/kiarina/labs.git
cd labs
git sparse-checkout set .gitignore .mise/tasks Makefile mise.toml \
  2026/07/18/mediapipe-holistic-vrm
mise -C 2026/07/18/mediapipe-holistic-vrm run
mise -C 2026/07/18/mediapipe-holistic-vrm run preview

Open the displayed localhost URL and select カメラを開始 (Start Camera). The initial setup downloads the MediaPipe bundle and Seed-san, then verifies their SHA-256 hashes. Camera input and inference remain inside the browser and are not sent to an external server. You can also load another VRM file with the file picker or by dropping it onto the viewer.

Data Flow

webcam (1280×720)
  -> Holistic Landmarker (body, face, and both hands)
  -> landmarks and 52 face blendshapes
  -> MediaPipe-to-Three.js coordinate conversion
  -> directions and body, face, and palm orientation
  -> 34 VRM 1.0 bones and expressions
  -> interpolation against the previous frame to reduce jitter
  -> Three.js rendering

Retargeting means transferring an estimated human pose to another skeleton. For an arm or leg, I align the direction between two landmarks, such as shoulder to elbow, with the corresponding bone's rest direction in the VRM. The hips, shoulders, face, and palms use multiple points to build three axes, providing an approximation of torso rotation and palm orientation as well.

The hand chains map MediaPipe joints to VRM finger bones. For the face, mappings include eyeBlinkLeft to blinkLeft and jawOpen to aa. This is not speech recognition; it is a simple conversion from facial-shape coefficients to avatar expressions.

Small changes in estimated points look like shaking when applied directly to an avatar. The implementation therefore holds changes below two degrees for normal bones and below three degrees for wrists and fingers, then interpolates larger movements over time. Leg landmarks below 0.65 visibility are rejected. If the whole pose is lost for more than 0.5 seconds, the avatar returns toward its neutral pose.

Results

I connected to the local server from the Codex in-app browser on a MacBook Pro with an Apple M1 Max and 64 GB of memory, running macOS 26.5.2. Inference ran through browser WASM using the CPU XNNPACK delegate.

The following values came from a 12-second measurement. I did not control the subject to remain fully visible throughout the run.

Metric	Observed
Input	1280×720
Processed frames	207
Effective throughput	17.3 FPS
Mean inference time	49.17 ms
Median inference time	59.60 ms
Inference time p95	70.60 ms
Final inference rate	23 FPS
Final render rate	43 FPS
Pose detected	141 / 207 frames
Pose applied to VRM	141 / 141 detected frames
Right hand detected	1 / 207 frames
Left hand detected	0 / 207 frames
Face detected	0 / 207 frames

The end-to-end path from webcam input through Holistic inference and coordinate conversion to VRM 1.0 loading and bone updates worked. The 49.17 ms mean measures the model call's wall time. It is not glass-to-glass latency including camera exposure and display scanout.

The demo video shows the tool in operation with VRoid Studio's AvatarSample_A, rather than the Seed-san model bundled with the lab. The raw camera feed can be hidden without stopping inference, which makes the landmarks and avatar response easier to see.

How to Read the Zero Hand and Face Counts

These zeros are not an accuracy result saying that the model cannot detect hands or faces. During the short whole-body measurement, the face and hands were small and were not kept in a controlled, continuously detectable framing. A separate short run did produce a frame where the body and face were active together.

All finger-bone and expression conversions are implemented. Synthetic-input tests confirmed finger rotation and the jawOpen -> aa mapping. However, this experiment does not tell us how stable simultaneous face, hand, and full-body tracking is with a real camera.

What the Throughput Means

At 17.3 FPS, the pose can be updated about 17 times per second. That is usable for checking movement and interactive demos, but it is not smooth 60 FPS motion capture.

In @mediapipe/tasks-vision 0.10.35, detectForVideo is synchronous. Rendering on the same main thread therefore waits during inference. A frame that takes the measured p95 of 70.60 ms also blocks rendering for that time; inference and rendering are not fully isolated.

A Plain-Language Reading of the Results

The findings come down to three points.

1. A normal webcam drove a 3D avatar.

All 141 frames with a detected pose reached the VRM update. The basic path from body movement to a browser avatar worked without a dedicated sensor.

1. The speed was suitable for a demo, not smooth motion capture.

Effective throughput was 17.3 FPS and mean inference time was 49.17 ms. The response is visible, but fast movement or live-production use may show stutter.

1. This was not yet a complete face-and-finger evaluation.

The 12-second run was not a controlled framing test. Successful body retargeting and stable simultaneous tracking of the body, face, and both hands are separate claims.

Limitations

This retargeter is a lightweight directional approximation. Aligning two points leaves rotation around the bone axis unresolved, so forearm, wrist, and ankle twist is not exact. It also lacks hip translation, foot locking, and floor-contact IK. IK, or inverse kinematics, calculates joint angles backward from a target hand or foot position. Without it, the feet can slide while the root remains fixed.

The evaluation is also limited to one M1 Max, 12 seconds, and 207 frames. I did not measure model accuracy, different lighting and backgrounds, multiple people, long-run stability, or full glass-to-glass latency.

My Takeaway

MediaPipe Holistic removed much of the plumbing required to connect separate body, face, and hand estimators. It provided a practical foundation for reaching a VRM avatar entirely inside the browser. Seeing visible body response from CPU/WASM on an M1 Max was more usable than I expected.

At the same time, the ability to output 553 points does not guarantee stable simultaneous full-body, facial, and finger capture from one monocular camera. A design centered on the body tracking demonstrated here, with face and hand tracking used according to framing and purpose, could work well for lightweight avatar demos or in-browser gesture interfaces.