Making a VRM Character Dance from a Single Live-Action Video: Fixing Leg, Arm, and Finger Clipping After Estimation

📝 Originally published (in Japanese) at forge.workstyle.tech.

Creating a Pipeline to Make VRM Characters Dance from Live-Action Videos

We are building a pipeline to make VRM characters dance from live-action videos. However, when we applied the estimated motion to the character, the feet penetrated the floor, the arms intersected with the body, and the thumb passed through the index finger.

In our dance material, we were able to reduce the maximum penetration into the floor from 148mm on the left and 94mm on the right to 0.4mm on the left and 1.5mm on the right. While correcting the grounding, we also confirmed that the original motion was not overly trimmed.

The input is a video shot with a fixed camera, and the output is a VRM animation (VRMA) and a browser viewer. This series will introduce the process of measuring and correcting the defects that remain after estimation, divided into six parts. The first part will cover the overall pipeline and design decisions.

Comparing Before and After Correction with Numbers

We compared the before and after correction using dance material (1,789 frames, 60fps). The slip is the total of the entire clip, and the penetration is the maximum value. p90 is the 90th percentile when the values are arranged in order from smallest, and f represents frames. The numbers are not the direct output of the solver, but were measured by writing the correction back to the Blender rig and re-observing it.

Item	Before Correction	After Correction
Foot Slip (Total, Left/Right)	9366 / 8084 mm	1362 / 1537 mm
Penetration into the Floor (Maximum, Left/Right)	148 / 94 mm	0.4 / 1.5 mm
Foot Acceleration p90 (Left/Right)	6.8 / 9.7 mm/f²	7.2 / 9.6 mm/f²

The foot acceleration p90 was 6.8→7.2 on the left and 9.7→9.6 mm/f² on the right. While reducing slip and penetration, this indicator maintained a close value. However, with p90 alone, it cannot be determined whether individual steps or kicks were preserved. The slip did not become zero, and some correction remained at the boundary of the grounding interval.

The slip indicator selects the target vertex based on height conditions, so it cannot be determined whether the number of evaluation targets changed based on the total value alone. This point will be discussed in Part 3.

It's easy to "eliminate slip" by just stopping the feet, but that would also make the motion flat. The goodness or badness of the correction should be judged not only by whether the desired error is reduced but also by whether the quantity that should not be eliminated remains. The numbers that appear in this series are mostly in sets of these two.

What we originally wanted to do was simple: "make the VRM avatar dance with the same motion as the video we took." If we use commercial motion capture products, it would be faster, but we wanted to create it from a single video shot with a handheld camera without a suit or markers. Monocular pose estimation (a model that restores 3D humans from a single video) has become quite practical in recent years.

The estimation itself is surprisingly accurate. The defects that occur are after that, and what we actually observed were symptoms such as:

• Feet penetrating or floating from the floor
• Knees bending even when standing
• Arms intersecting with the body
• Thumbs passing through index fingers
• Wrist skin being crushed like candy wrapper

This series is a record of measuring and correcting each of these defects one by one.

Note that the process of applying the estimated motion to a character with a different physique is called "retargeting." In this article, we refer to the process of reassembling the bone direction to match the character's bone length as "lifting," and the process of creating a pose from bone rotation as "FK application."

Flow from Estimation to Correction and Verification

0. Material video (fixed camera, full-body)
1. Estimation        Convert GVHMR result to joint points, pelvis height, and line of sight
1b. Coordinate system  Unify to "camera coordinate system" aligned with gravity
1c. Image integration  Reload wrist, ankle, elbow, and knee onto 2D detection ray
2. Score            Detect landing, takeoff, and impact numerically
3. spec            Support foot schedule and correction specification (JSON)
4. Lifting      Reassemble direction only, using rig length
5. FK application  Orient all bones on Blender, explore waist height
5.1 Range of motion  Keep all joints within anatomical range
5.1c Grounding      Detect foot grounding interval, solve foot trajectory and waist at once
5.2 Interference      Push arms away from body
5.3 Hand            Finger range of motion, place thumb above index finger
6. Comparison      Measure difference by eye, comparing with original video
7. QA          Numerical gate (defect, slip, drift)

We output the corrected animation to VRMA and verify it with a viewer.

The joint points in step 1 are not in the format directly output by the estimation model. An adapter in this repository composes MediaPipe-style 33 points from SMPL joints and facial mesh vertices. The downstream process only looks at these 33 points, so even if the estimator is replaced, the same code can be run from there.

Among these, steps 1b, 1c, and 5.1 onwards are layers that were added after actually moving and finding them to be insufficient. It wasn't in this form from the beginning.

Transferring Bone "Direction"

If we apply the estimated joint positions to the character as is, defects will occur because the physique is different. On the other hand, a method that adds and subtracts differences also breaks down when the correction amount becomes large. The original documentation has a record that with a 20cm correction, the hand interval shrunk from 0.25m to 0.08m, and the wrist angle changed from 7° to 123°.

In this pipeline, we use the estimated bone direction and reassemble it using the actual measured value of the rig length.

# pipeline/lift_to_mixamo.py:441-449
l_elbow = l_arm + seg(mp, i, "left_shoulder", "left_elbow") * LEN["l_arm"]
l_wrist = l_elbow + seg(mp, i, "left_elbow", "left_wrist") * LEN["l_fore"]
l_hand  = l_wrist + hand_dir(mp, i, "left") * LEN["l_hand"]

l_knee  = l_upleg + seg(mp, i, "left_hip", "left_knee") * LEN["l_upleg"]
l_ankle = l_knee  + seg(mp, i, "left_knee", "left_ankle") * LEN["l_leg"]

seg() is a unit vector between two points, and LEN is the actual measured length of the rig. The position of the shoulders and hip joints is also not taken from the estimation, but the offset from the rest pose is applied to the estimated base of the waist.

This method ensures that the bone length does not shift by definition. However, the problem of the character's hands and feet being thicker than the performer remains. The bone length is summarized in rig_profile.json, and when the character is replaced, only this one file is replaced.

Avoiding Blender's IK Constraint and Writing to Rotation Key

With the rig and settings used this time, applying Blender's IK constraint caused the feet to fly away to -81m. Therefore, we configured it to write the corrected pose as the rotation of each bone. We also use our own analytical 2-bone IK (intersection of two circles) for calculations that solve leg and arm positions. Putting the feet on the floor is also done by "exploring the waist height to find the height at which the support foot touches the ground," rather than using a constraint.

There are other premises that should be noted regarding the rig.

• The armature is rotated 90° around the X-axis and scaled to 0.01. If we create a pose matrix in world coordinates and enter it into the pose bone, the hands will fly away 100 times. The target is to decide in armature space (centimeter system).
• The only moving bone is Hips. All others are quaternions, and location is reset to 0 every frame (old keys will remain and cause defects).
• World positions are read after view_layer.update() using (arm.matrix_world @ pose_bone.matrix).to_translation(). When comparing positions, the coordinate space and evaluation timing must be aligned.

These points will be discussed in Part 6.

Caching Estimation Results and Repeating Correction on CPU

GVHMR took 52 seconds to estimate 300 frames on an RTX 4080 Laptop (VRAM 12GB), with a VRAM peak of 4.2GB. We cache the estimation results and repeat the correction using NumPy and Blender.

The only part that uses the GPU is the pose estimation. There are two backends for hand finger estimation: MediaPipe, which runs on CPU, and WiLoR, which runs on GPU. If WiLoR is chosen, the GPU is also used for this part.

This division has practical benefits.

• Estimation only needs to be done once, and the results can be cached. Subsequent trial and error do not compete for the GPU.
• Grounding processing is written only with NumPy, so changing a constant and re-measuring takes only a few seconds.

If it takes several minutes to make one round trip with Blender, this difference becomes the difference in the number of trial iterations. In fact, the grounding processing is divided into three segments: "Blender observation → NumPy solution → Blender writing," and only the middle part was repeated dozens of times to decide on the constants.

For reference, the actual time for each segment when passing a 4-minute material (14,373 frames, 60fps) is as follows. Hand and facial expression estimation are not included in this table.

Segment	Time
Pose estimation (GPU)	Several minutes
Lifting + FK application	About 10 minutes
Grounding processing (observation → solution → writing → re-observation)	About 35 minutes
Arm interference (NumPy part is 3 minutes)	About 25 minutes
Thumb	About 1 hour
VRMA output	About 30 minutes

The total is about 2.5 hours.

spec Includes Material Information and Correction Specifications

Each clip has one JSON spec that summarizes material information, support foot schedule, standing pose adjustment, and override of estimated intervals that went wrong. Automatically generated content is also corrected as needed.

{
  "name": "fountain",
  "landmarks": "plates/fountain/landmarks.json",
  "src_fps": 29.97,
  "dst_fps": 60,

  // Support foot. Intervals are all written in the frame number of the original video.
  "plant": [
    { "src": [1, 70],    "support": "left"  },
    { "src": [71, 126],  "support": "both"  },
    { "src": [127, 129], "support": "none"  },   // Floating
    { "src": [130, 145], "support": "left"  }
  ],

  // Standing pose adjustment (knee in/out, stride, elbow)
  "stance": { "knee_in_pct": 3.5, "toe_in_deg": 14 },

  // Override estimated intervals that went wrong, with human reading
  "arm_overrides": [
    { "side": "right", "src": [76, 163], "ramp_src": 6,
      "wrist_local": [-0.17, -0.12, 0.33],
      "elbow_local": [-0.25, 0.05, 0.12] }
  ]
}

We are aware of the following three design considerations:

Unify frame numbers to the original video's frame number. When a person looks at the video and says "the right arm is strange at frame 27," it can be written directly into the spec. If the frame number after conversion to 60fps is inserted, it becomes easier to mix up the frame numbers. In fact, we made a mistake here.

Correction must always have an interval and a ramp. Corrections such as tightening the elbow only for a certain interval or overriding the arm position only for a certain interval will have a step difference at the boundary if switched abruptly. We use ramp_src to smooth it out.

There are scenes where human reading wins over estimation. In intervals where the arm is hidden behind the body, the estimation error becomes large. In such cases, it is faster for a person to look at the video and write it. However, only for that interval, and without touching other joints.

The support foot schedule is troublesome to write by hand, so we automatically generate it from the pelvis acceleration and correct only the necessary parts. When processing 41 materials in bulk, we did not make individual adjustments, and everything was automatic generation.

Compared Three Estimators and Returned to the First One

We set up two other estimators (PromptHMR-Vid, Human3R) for comparison. All of them can write landmarks.json in the same format, and the downstream process runs unchanged.

In this comparison, the slip and penetration after grounding processing were at the same level for the three estimators.

Estimator/Setting	Slip p90 (mm/f, Left/Right)	Penetration Maximum (mm, Left/Right)	Apply Error p50 (mm, Left/Right)
GVHMR	3.8 / 5.4	6.1 / 8.1	11 / 6
PromptHMR	4.5 / 4.7	5.7 / 2.1	5 / 4
PromptHMR raw	2.6 / 5.7	4.5 / 5.6	7 / 5
Human3R	4.1 / 2.9	2.2 / 1.9	5 / 4

We note the following three conditions. This comparison is for the same dance material passed at 30fps, and the fps and evaluation process are different from the table at the beginning. Both cannot be read as the same numerical value. PromptHMR raw is the estimation result without PromptHMR's own post-processing optimization (translation and camera re-fitting).

The remaining difference is the nature of the estimator itself. With this material and setting, PromptHMR-Vid had 1.3-1.7 times more jerky foot and waist acceleration, and the jump disappeared (no floating intervals remained) when the post-processing optimization was applied. Human3R estimated that the camera moved 3.5°/0.29m even though it was a fixed camera clip.

So, we stuck with the default GVHMR. Another thing we found out from this comparison is that none of the estimators can make the feet touch the ground. The lowest vertex of the shoe sole was floating from the ground by 45-85mm on average, and the frames where it touched the ground were less than 1%.

By the way, the initial version of this comparison table had an error. The other three columns were processed without "walk-away correction," leaving a difference of 0.4-1.1m, which appeared as 85-143mm of fake penetration. If the comparison conditions are not aligned, apparent differences will be generated in this form.

Output: VRMA and Viewer

The final output is VRMA (VRM Animation), and we are using a three.js viewer to play it back. We set up a local server and made it possible to switch materials with a pull-down.

What comes into effect here is the problem of which correction reaches which route. Bone pose corrections are written as animation keys to VRMA. Skinning weight corrections, which will be discussed in Part 6, belong to the avatar's mesh and are used in the corrected VRM in the viewer. Whether the effect of Blender's modifiers or constraints reaches depends on the output route and whether baking is performed.

We also had an accident due to the difference in routes. The processing of finger range of motion and line of sight was only in the rendering segment and did not run once in the VRMA creation route. The viewer had a character with "eyes that don't follow and fingers that bend more than 90°." We solved this by moving the processing to right after the arm interference.

What This Series Will Cover

Each part will be written to be readable independently.

Part	Content
Part 1 (this article)	Making VRM characters dance from live-action videos ── Correcting defects in feet, arms, and fingers that remain after estimation
Part 2	The difference in head and chest angles was 0.000° for all frames ── A story about how motion measurement was broken
Part 3	Shortening the leg by 1% made the knee bend by 16° ── A pitfall in VRM's grounding correction
Part 4	Arm penetration correction flips front and back ── Deciding the direction to escape using dynamic planning
Part 5	The thumb rests on the index finger, not pushed away ── Correction to maintain grasping and smoothing
Part 6	The cause of the wrist skin being crushed was skinning, not pose

The next part will start with the abnormality that "the difference in head and chest angles was 0.000° for all frames." It's a story about doubting measurement before motion when appearance and numbers contradict.

Supplement: The Foundation is Someone Else's Implementation

The skeleton of this pipeline (estimation → score → spec → lifting → FK application → QA, and comparison with the original video) is an implementation of squall01337/mixamo-llm-mocap (MIT). The part referred to as "fork origin" in this article refers to this repository. We explicitly state the reference to avoid confusion with repositories of the same name.

The parts that we added after forking, which are quoted in this article, are not published. Please note that it is not possible to reproduce and verify the implementation by looking at the repository (the quotes include file names and line numbers).

The unification of coordinate systems, range of motion, grounding, arm interference, hand processing, and migration to VRM are all parts that we added afterwards. The code quotes are necessary for explanation and are excerpted from the implementation. They are not code that can be pasted and run as is, but rather calculations that were taken out to illustrate the point.