When I Shortened the Leg by 1%, the Knee Bent by 16 - A Pitfall in VRM Grounding Correction

📝 Originally published (in Japanese) at forge.workstyle.tech.

Retargeting Character Movement from Estimated Actor Movement

After retargeting the estimated actor movement to a character with a different body type, the character's foot was penetrating the floor by 148mm. When trying to fix the foot's reach distance by adding a 1% margin, the knee bent by 16-18° just by standing.

To address this, the approach was changed to create the grounded foot's trajectory first and then adjust the hip position to match. The difference in knee angle estimation decreased from 5.7° to 2.1° (median) and from 18° to 8° (p90), where p90 is the 90th percentile of the values in ascending order.

This is the third installment in a series on creating VRM animations from fixed-camera live-action videos. In the second part, the measurement target and coordinate system were aligned. This time, the focus is on finding the grounded interval, preventing foot slippage, and adjusting the body to match.

The key takeaway is that foot height alone is not enough to determine grounding.

There are three main tasks:

1. Determine where the foot is grounded
2. Create a non-slip trajectory for the grounded foot
3. Align the body with the foot (hip position)

1. Foot Height Alone is Not Enough to Determine Grounding

First, let's look at detecting grounding.

A straightforward approach might be to consider a frame where the foot is near the floor as grounded. However, at the stage where the estimated results were retargeted to the character, the floating foot was penetrating the floor by 148mm. In this state, it's clear that relying solely on foot height to determine grounding is insufficient. Instead, the approach involves checking the movement of the shoe sole's vertices closest to the floor.

# Grounded foot: some point on the shoe sole is stationary
    # - For a flat foot, the entire sole
    # - For a toe spin, the toe
    # - For a heel roll, the rolling point
# Floating foot: all points on the shoe sole are moving

Thus, the detection is based on finding the vertex with the minimum movement among those closest to the floor in consecutive frames. The key point here is taking the minimum value, which allows the detection of both toe spins and heel rolls.

# pipeline/contact/segment.py:56-68 (docstring omitted)
def anchor_speed(sole, cfg):
    z = sole[:, :, 2]
    low = z < cfg.hover
    out = np.full(len(sole), np.nan)
    for i in range(1, len(sole)):
        both = low[i] & low[i - 1]
        if both.sum() >= cfg.min_verts:
            d = sole[i, both, :2] - sole[i - 1, both, :2]
            out[i] = np.linalg.norm(d, axis=1).min()     # ← min is the main part
    return out

Height is also used here. In this implementation, vertices within hover = 25mm from the floor that are present in both the current and previous frames (min_verts = 6 points) are considered, and then the minimum movement that is below v_stick = 3mm/frame (for 60fps material) is used to determine grounding. Height serves as a pre-condition to narrow down candidates before the movement-based judgment.

This is not a physical definition of grounding but rather the judgment rule for this pipeline. Scenes where the foot is grounded but slipping or appears stationary in mid-air do occur.

Interestingly, the function for measuring slippage later on has a very similar form, except for using .mean() at the end.

# pipeline/contact/metrics.py:25
out[i] = np.linalg.norm(sole[i, both, :2] - sole[i - 1, both, :2], axis=1).mean()

min checks for the presence of a stationary point, while mean looks at the average movement of the vertices in question. Here, the latter is used as an indicator of slippage, but since it can include rotations during grounding, it's checked visually as well.

How to Fill Gaps

Grounded intervals are sometimes broken by noise. Two rules are used to fill these gaps:

• Short gaps that don't go anywhere (within 0.25 seconds and less than 6cm movement) are considered noise within a grounded interval.
• Gaps where the foot doesn't float and isn't in a hurry (within 1.5 seconds, within 3.5cm from the floor, and less than 0.25m movement) are also considered part of a grounded interval.

The second rule exists to account for depth fluctuations in the estimator. In the image, the foot may appear grounded, but in the estimation, it might move ±10cm in World Y.

Only the part checking height and speed is shown below. Conditions for time and movement distance are in the caller.

# pipeline/contact/segment.py:78-82 (docstring omitted)
def _on_floor_gap(sole, speed, a, b, cfg):
    low = sole[a:b, :, 2].min(1)
    return bool(low.max() < cfg.lift and speed[a:b].max() < cfg.step_v)

2. Preventing Grounded Foot Slippage

Once the grounded interval is determined, the foot's trajectory is created. The design principle can be stated in one line:

Determine the grounded interval first, then create a foot trajectory that integrates through that interval.

Within the grounded interval, the "quietest frame" in that interval is used as a starting point, and then translation is integrated outward. The goal is to keep the common grounded vertices in the same position in world coordinates across consecutive frames.

# pipeline/contact/foot_path.py:163-173 (excerpt)
for i in rng:
    w = loc[i] @ Rp[i].T
    Ci = _contact(w[:, 2], cfg.contact_band)
    both = Ci & prev_C
    if both.sum() >= cfg.min_common:
        t[i, :2] = prev_world[both, :2].mean(0) - w[both, :2].mean(0)
    else:
        j = i - direction
        t[i, :2] = t[j, :2] + (foot_t[i, :2] - foot_t[j, :2])
    t[i, 2] = cfg.floor - w[:, 2].min()
    prev_C, prev_world = Ci, w + t[i]

What's being aligned is the average horizontal position of the common grounded vertices. If there are enough common grounded vertices, their average horizontal position is made to match that of the previous frame. While this can stop a flat foot from slipping, it doesn't make all vertices' slippage zero, especially during rotations or when the contact point changes. In fact, there's residual slippage in the actual measurement mentioned at the beginning.

A Small Difference Makes the Axis Foot's Orientation Change

A nasty bug was encountered here.

For flat foot grounding, the orientation (yaw) of the interval was fixed based on the "quietest frame." However, dancers spin on their axis foot. In measurements, the estimated foot orientation spun 20-80° in 11 grounded intervals.

Furthermore, the choice of the starting frame was so sensitive that a difference of 0.01mm/frame could result in choosing a completely different pose. "The quietest frame" could switch between two candidates, leading to vastly different outcomes:

• The left foot could be 44° inward and fixed for 2.5 seconds, or
• It could be 22° outward and fixed.

It's not surprising that the choice of starting frame could make such a large difference. The fix involves introducing a dead zone and making the orientation follow the estimation smoothly if it spins beyond a certain threshold.

# pipeline/contact/foot_path.py:133-139 (excerpt)
seg = np.unwrap(yaw[p.start:p.end])
ref = seg[p.anchor - p.start]
d = box(seg - ref, win)
w = smoothstep((d.max() - d.min() - dead) / dead)
hold = ref + w * d

Where to Place the Foot is a Separate Issue

While the movement within the grounded interval is fixed, where to place that grounded interval in the world remains a degree of freedom.

A straightforward approach might be to place it at the estimated position of the quietest frame. However, this was found to be incorrect. During long grounded intervals, the estimation could drift, and the other foot, which steps out during this interval, would land based on the drifted position. In the image, the feet would be closed, but in the result, they would be 70mm apart.

So, the positions of both feet's groundings are solved together in a least-squares manner. There are two terms:

• Being close to the estimated position on average
• Keeping the distance between feet as estimated when they are close

The unknowns are 2 per grounding (xy), and the system is linear because the offsets of floating frames are blends of neighboring groundings.

# pipeline/contact/placement.py:105-113 (excerpt, relative terms)
gap = np.linalg.norm(est[a] - est[b], axis=1)
w = cfg.rel_gain * (1.0 - smoothstep((gap - cfg.close_full)
                                     / (cfg.close_none - cfg.close_full)))
M = np.zeros((n, P)); M[:, col[a]] = C[a]; M[:, col[b]] = -C[b]
R = E[a] - E[b]
A += M.T @ (w[:, None] * M)
B -= M.T @ (w[:, None] * R)

3. Aligning the Body with the Foot - Hip Position

Once the foot is determined, the body follows. The answer to what happens when the leg doesn't reach the foot is sequential:

1. Lift the heel (only when the toe is grounded). The foot doesn't move, but the ankle moves towards the hip.
2. Make the hip follow the foot's displacement
3. Match the extension of both legs to the estimation
4. If still not reaching, move the hip to compensate

The first step has a condition because it makes sense. For a flat foot, lifting the heel would be a lie. In the image, if it's a flat foot, the reason the leg is short is that the knee is bent, not that the heel is floating.

The Knee Bends 16.2° When the Leg is 1% Shorter

The fourth step, "moving the hip to compensate," had a limit set to "99% of the leg length." This can be read as a 1% margin.

This is a side view of the same frame, differing only in the upper limit of the hip to foot distance. It wasn't just a matter of margin; it was necessary.

For nearly straight knees, the knee angle is proportional to the square root of the leg length deficit. In this calculation example, using a thigh length of 0.394m and a shin length of 0.445m, shortening the leg by 1% results in a 16.2° knee bend. The actual material had the knee bent 16-18° just by standing. The estimation side had a bend of 2-3°.

Three Processes to Solve the Hip

First, the hip is horizontally moved to match the foot's correction amount. The influence of the supporting foot is strongly reflected, and that of the floating foot is minor (pipeline/contact/body_solve.py's follow_feet).

Next, the extension of both legs is matched to the estimation. This is done using Gauss-Newton method in batch for each frame.

# pipeline/contact/body_solve.py:184-196
for _ in range(cfg.extension_iters):
    JTJ = np.tile(np.eye(3) * cfg.extension_reg, (n, 1, 1))
    g = cfg.extension_reg * (t - t0)
    for s, leg in legs.items():
        v = ankles[s] - leg["H"] - t
        d = np.linalg.norm(v, axis=1)
        u = v / np.maximum(d, 1e-9)[:, None]
        JTJ += u[:, :, None] * u[:, None, :]
        g -= u * (d - leg["extension"])[:, None]
    step = np.linalg.solve(JTJ, g[:, :, None])[:, :, 0]
    norm = np.linalg.norm(step, axis=1, keepdims=True)
    t = t - step * np.minimum(1.0, cfg.extension_step / np.maximum(norm, 1e-12))

d = |A-H-t|'s Jacobian regarding t is -u. The code's g -= ... also corresponds to this sign. In JᵀJ, the signs cancel out, resulting in the outer product of unit directions. Each frame's 3×3 system of linear equations is solved together using NumPy.

Lastly, for the side where the leg's reach exceeds the limit, the hip is projected towards the foot in an alternating manner (project_reach, alternating projection).

Balance: Tilting the Hip Doesn't Correct the Rear Lean

Another issue was that even when standing on both feet, the body would lean backward in a way that a person couldn't stand.

The average position of mesh vertices, used as an approximate balance indicator, was 50mm behind the foot sole's range. This depends on vertex density and isn't the actual center of mass. However, the existing correction only kicks in if the overhang exceeds 90mm, which doesn't happen here. Moreover, even if it did, the lean wouldn't be corrected. The estimation itself had the torso and legs leaning backward by 16° and 8°, respectively. Hip parallel movement doesn't correct the lean.

The fix involves rotating the upper body around the ankle as a pivot, but only for frames where the character is standing still and both feet are grounded. The legs are then re-solved to match the new ankle position.

# pipeline/contact/body_solve.py:238-245 (excerpt)
dist, near = hull_distance(com[i, :2], convex_hull(pts))
h = com[i, 2] - pivot[i, 2]
if dist <= cfg.lean_out or h <= 1e-3:
    continue
u = np.array([*(near - com[i, :2]) / dist, 0.0])
angle = min(np.arctan((dist + cfg.lean_target) / h), np.radians(cfg.lean_max_deg))
rv[i] = np.cross(Z_UP, u) * angle

In the actual measurement, the lean was corrected from 7° to 3° by tilting the body 3.9°.

This correction is limited to standing still because, during movement, the center of mass can naturally deviate from the foot sole's range, such as during kicks, jumps, or wide stances. In the material used, there were instances where it deviated by 20-30cm, and such deviations aren't uniformly corrected.

Writing Back to Blender Introduced a 90mm Discrepancy

Finally, a pitfall on the writing side.

The FK rig relationship can be simplified as follows:

matrix = parent.matrix @ inv(parent.bone.matrix_local) @ bone.matrix_local @ basis

basis represents the local movement, rotation, and scale of the bone, which are the values kept by the location/rotation keys. In the writing process, since the parent's changes were evaluated before setting the matrix to the child, simply writing pose_bone.matrix would use the old parent pose to inversely calculate basis. To address this, the parent's pose as solved in this frame is explicitly passed to calculate basis (pipeline/contact/bones.py's parent_frame).

Moreover, troublesome is that the bone's parent differs depending on the rig. In the ARP-remapped rig, the foot has no parent and is reached by the shin via IK, while in the original FK rig, the foot is a child of the shin. Writing the latter as "no parent" results in the foot's position being correct, but the foot rotates with the shin. In a dance kick, this resulted in a maximum rotation of 88° and an apply_error (distance between the written position and the solved position upon re-reading) of 90mm p50 (p50 is the median).

This discrepancy isn't detectable in the reading process during calculations. Therefore, the final verification involves re-observing the written .blend file. The solver's output is not trusted.

Conclusion

• Grounding is determined by combining height conditions and shoe sole movement.
• Create the grounded foot's trajectory first, then align the hip.
• The axis foot spins. Fixing its orientation can lead to vastly different outcomes based on minor differences in the starting frame choice.
• Small changes in leg length significantly affect the knee angle near full extension.
• Lean isn't corrected by hip parallel movement. For standing still, rotate around the ankle.
• Finally, verify by re-observing the saved .blend file.

Even with the feet aligned, the arms penetrate the body. The next part (Part 4) will address the issue of arms flipping front to back by dynamically planning which direction to evade.

Series Index

• Part 1: Creating a VRM from a Live-Action Video - Fixing Breakage in Feet, Arms, and Fingers After Estimation
• Part 2: The Difference in Head and Chest Angles Was 0.000° - A Story of How Motion Measurement Was Broken
• Part 3 (this article): The Knee Bends 16° When the Leg is 1% Shorter - Traps Encountered in VRM Grounding Correction
• Part 4: Arm Penetration Correction Flips Front to Back - Dynamically Deciding the Evasion Direction
• Part 5: The Thumb Rests on the Index, Not Pushed Away - Correction to Maintain Grip and Smoothing
• Part 6: The Cause of the Wrist Skin Crease Was Skinning, Not Pose

Supplement

This pipeline is based on squall01337/mixamo-llm-mocap (MIT). The referenced code, except for parts added after forking, is not publicly available. Please note that the repository cannot be used to reproduce the results described, as the necessary parts are not included.

The code excerpts provided are necessary for the explanation and have been simplified. Initializations and auxiliary functions not crucial for understanding have been omitted. For the complete implementation, including omitted parts, please refer to the referenced files and line numbers.