Elbow Interference Correction Jumps Back and Forth

📝 Originally published (in Japanese) at forge.workstyle.tech.

A character with hands clasped behind their back was supposed to remain still, but instead, their right hand moved forward by 17cm, then backward and upward by 15cm, and forward again by 17cm. The process intended to fix arm clipping was causing the arm to bounce.

The desired depth to move the forearm backward is 74mm, and the desired depth to move the hand forward is 73mm. If the deepest direction is chosen for each frame, a 1mm fluctuation causes the direction to switch. Therefore, we adopted a design that uses dynamic programming to select the optimal path for the entire clip from the candidates of each frame.

In Part 3, we aligned the feet with the ground. This time, we address arms clipping into the body due to differences in physique. The focus is on stabilizing the push direction and preserving the hand-clasping posture as much as possible.

This is the fourth installment in a series on creating VRM animations from fixed-camera live-action videos.

The previous article was Part 3: "Shortening the Legs by 1% Caused the Knees to Bend 16° ── Pitfalls in VRM Grounding Correction". Links to all parts of the series are provided at the end.

The Shape of the Push Process

First, let’s establish the premise. During retargeting (mapping an actor’s movements to a character of a different physique), arms may clip through the body. This occurs because the character’s limbs might be thicker or the torso broader.

On the left, the hand is clipped into the waist in the uncorrected state. The right image shows the corrected state. This article focuses not on "whether to push" but on "which direction to push."

Sample points are placed along the upper arm, forearm, and hand bones. For each frame, the movement required to position each part at a distance of its radius plus 6mm from the body surface is calculated. The forearm and hand samples with the largest push move the wrist, while the forearm and upper arm samples with the largest push move the elbow pole. The elbow is then resolved using 2-bone IK (maintaining bone lengths). This trial is repeated several times, as the moved forearm may clip into other parts.

This approach works well, but the challenge lies in determining the push direction.

Coin Flip Between Front and Back

In the first 120 frames of a dance clip, the actor stands with their hands clasped behind their back. The estimation shows the hands clipped into the waist by 5cm. Despite being stationary, the result was as follows:

Right hand: 17cm forward → 15cm backward and upward → 17cm forward again → ...

The direction flips back and forth with each frame.

The issue stems from choosing the direction of the deepest sample. In a lowered arm position:

• Forearm (clipped into the waist side): Wants to push backward and upward with a depth of 74mm
• Hand (clipped into the thigh): Wants to push forward with a depth of 73mm

The depths are nearly identical. A 1mm fluctuation determines which direction wins. As long as the decision is made independently for each frame, it’s a coin flip.

Solution 1: Hold Candidates and Select Path for Entire Clip

Up to 3 candidate solutions are held for each frame (KEEP = 3). The optimal path for the entire clip with the minimum cost is selected using dynamic programming. The chosen path is the best among the generated candidates, not necessarily the best among all possible arm postures.

There are three types of candidates:

1. Solution pushed from the estimated arm
2. Solution pushed from the previous frame’s solution (scanned once forward and backward, with the carryover amount decayed by 0.8, so offsets no longer needed by the body fade away)
3. Solution started from a position beyond the body in the image-indicated direction (discussed later)

The cost is calculated as follows (all distances in meters):

• Unary: Sum of squared wrist and elbow movements + 10 × (remaining depth inside the body) + side preference
• Transition: 2.0 × (squared change in solution from the previous frame)

# pipeline/clearance/modes.py:42-45
def _unary(c):
    dW, dK, inside = c[:3]
    side = c[3] if len(c) > 3 else 0.0
    return float(dW @ dW + dK @ dK) + INSIDE * inside + side

# pipeline/clearance/modes.py:118-125
for i in range(1, n):
    prev, cur = cands[i - 1], cands[i]
    step = np.array([[turn * float(np.sum((a[0] - b[0]) ** 2) + np.sum((a[1] - b[1]) ** 2))
                      for b in prev] for a in cur])
    total = cost[-1][None, :] + step
    j = np.argmin(total, axis=1)
    cost.append(total[np.arange(len(cur)), j] + np.array([_unary(c) for c in cur]))
    back.append(j)

Remaining clipping is penalized by 10 per unit distance, prioritizing solutions that move the arm out of the body over those that minimize movement. However, the penalty is finite, so non-clipping is not guaranteed. As shown in the table below, some frames remain clipped even after correction. Since distances are in meters, the coefficients 10 and 2.0 for transitions are also scale-dependent. Adjust these values when using a different unit system.

Solution 2: Weakly Prefer Camera Side When Differences Are Small

Dynamic programming alone tends to select the solution with the smallest movement. In this posture, pushing forward results in slightly less movement, causing the hands to move in front of the body. This is incorrect, as the video shows the hands clasped behind the back.

To resolve this, hand detection is used as a tiebreaker when differences are small. If the hand is detected, the candidate pushing toward the camera is weakly preferred; if not detected, the candidate pushing away from the camera is preferred.

It’s crucial to distinguish between camera side and anatomical front of the body. This rule is based on occlusion in the material, not a definitive judgment of hand position.

# pipeline/clearance/modes.py:48-54 (docstring omitted)
def _side(dW, toward, seen):
    if toward is None:
        return 0.0
    a = float(dW @ toward)
    return SIDE * (seen * max(0.0, -a) + (1.0 - seen) * max(0.0, a))

The weight SIDE = 0.3 (per meter) is kept low. Hand detection can fail due to motion blur or small hands, so the weight is minimized to influence the choice only when other costs are similar.

Another necessity was generating candidates for both sides. Local pushing alone cannot find solutions that escape to the opposite side of the body. Thus, a candidate starting from a position beyond the body in the image-indicated direction by the clipping depth plus 5cm is added.

# pipeline/clearance/modes.py:86-92 (docstring omitted)
def _seed(i):
    if toward is None or depth0[i] <= 0.0:
        return []
    d = toward[i] * (1.0 if seen[i] >= 0.5 else -1.0) * (depth0[i] + SEED)
    return [_run(H[i], K[i], W[i], T[i], field, radii, margin, i, (d, d), *cam(i))[0]]

Result: Hands Stabilize Behind the Back, Reducing Flipping

In the first 120 frames, both hands remained stable behind the waist, 10cm back, without flipping front to back.

The aggregated results are as follows. The evaluation is based on the same dance clip, with "Before" and "After" columns showing results both before and after applying dynamic programming and camera-side preference.

Item	Before Dynamic Programming + Camera Preference	After
Front-back flips (within 10 frames) Left / Right	65 / 29	29 / 11
Frames still clipped after correction Left / Right	73 / 148	16 / 70

Clipping frame counts are per arm (frames_inside_after from run_solve, frames with depth > 0.1mm). "Flips within 10 frames" are not precisely recorded in the implementation, so focus on the reduction trend rather than exact counts.

Computation time for one dance clip (14,373 frames, both arms) was 3 minutes, measured using NumPy’s solver alone, excluding Blender observation and write-back. This differs from the pipeline time reported in Part 1, as the measurement scope varies.

Exclude Arm-Following Vertices from Push Surface

Now, let’s discuss creating the body surface for pushing. Prepared for each frame, the entire mesh cannot be used directly, so a three-step filter is applied:

1. Select vertices dominated by torso, thigh, and head bones (same logic as treating skirt brushing the floor as non-foot)
2. Remove vertices with inward-facing normals. VRoid models have inner clothing surfaces, and using inward normals leads to incorrect push directions
3. Remove vertices following arm/shoulder bones by more than 15%. Skin above the deltoid moves partially with the arm, so it’s neither body nor arm

After filtering, a 2.5cm grid is applied for decimation. Arm thickness is not fixed but measured from each bone’s vertices (using a single radius for the entire forearm causes the wrist to appear as thick as the elbow, making placed hands float).

This push process approximates the nearby body surface as a plane and checks if sample points lie behind it. It’s not an exact inside/outside test for the entire mesh. Part 2 discussed avoiding capsule approximation for measurement verification, but this is an approximation for correction, serving a different purpose. Final appearance is confirmed through rendering.

Determine Push Direction Using 16 Vertices

Since the approximation is planar, neighbor selection matters.

Initially, the nearest vertex determined the direction. However, when the arm slid over a rounded body, the nearest vertex switched, causing the push direction to jump by the inter-vertex angle, resulting in visible "jitter."

Measurements from the desk material show that the peak of the arm’s (wrist) trajectory second derivative (in mm/frame²) increased from 3 to 9 after this step. Note that the exact method of measuring "peak" (maximum value vs. percentile) isn’t recorded, so focus on the worsening trend and magnitude.

The fix was to use a distance-weighted average of 16 neighboring vertices to create the plane.

# pipeline/clearance/geom.py:136-145
w = np.exp(-((dist - dist.min(1, keepdims=True)) / self.SOFT) ** 2)
w = w / np.maximum(w.sum(1, keepdims=True), 1e-12)
n = np.einsum("sk,skj->sj", w, N[j])
ln = np.linalg.norm(n, axis=1, keepdims=True)
n = np.where(ln > 1e-9, n / np.maximum(ln, 1e-12), N[j][:, 0])
s = np.einsum("sk,skj,sj->s", w, P[:, None, :] - V[j], n)
depth = np.where(near, np.maximum(-s, 0.0), 0.0)
need = np.where(near, np.maximum(np.asarray(clearance, float) - s, 0.0), 0.0)
return n * need[:, None], depth

The normal n is a weighted average of 16 vertex normals, and the signed distance s is also a weighted mix of each vertex’s plane value. Weights are measured by relative distance from the nearest vertex to prevent underflow when all neighbors are distant.

The same "3 → 9" issue reappeared when keyframe decimation was applied to pushing frames. Pushing only keyed frames creates gaps between keyed and non-keyed frames (in desk material, 85% of frames are pushed). Currently, all frames are keyed. This illustrates how the same symptom can have different causes.

Moving the Elbow Is Essential to Avoid Over-Pushing

Push assignment is as follows:

# pipeline/clearance/solve.py:109-117 (comments omitted)
pf = _largest(pushes[:nf])
ph = _largest(pushes[nf:nf + nh])
pu = _largest(pushes[nf + nh:])
dW = _largest(np.stack([pf, ph]))
dK = _largest(np.stack([pf, pu]))

The key is that the forearm sample pf contributes to both wrist movement dW and elbow pole movement dK. Fixing the elbow and moving only the wrist means the elbow-proximal quarter of the forearm receives only about a quarter of the wrist movement. In this approximation, pushing that point requires moving the wrist about four times as much. Moving the elbow allows the forearm to exit the body in a more parallel manner.

Only the upper arm’s distal 30% is tested (UPPER_T = (0.7, 1.0)). The shoulder-proximal side is both "arm and body," and testing it causes conflicts with the inhabited torso.

Hand-Clasping Posture: Pushing Pulls Hands Apart

Another issue arises with hand-clasping.

When hands are clasped in front of the waist, pushing each arm independently causes the hands to separate. This occurs because the rounded abdomen has left and right surfaces sloping in opposite directions. Each hand tries to exit its nearest surface, pulling the left hand left and the right hand right.

In tests, hands initially 5.3cm apart were pushed to 19.5cm apart after interference processing, undoing prior work.

"Sharing Only the Separation Component" Was Incorrect

The initial implementation averaged only the separation component of the push vectors. This discarded horizontal pushing, leaving 2,509 frames clipped inside the body on a rounded abdomen.

The correct approach is the opposite: share the direction and let each hand manage its distance. The left and right push vectors are added, and their direction becomes the common escape direction. Correction strength uses each hand’s original length, blended with the original vector based on coupling strength amount. The natural direction to exit a rounded body is "straight out," which is the average of the two pushes.

# pipeline/clearance/couple.py:40-51
dL = np.asarray(push_left, float)
dR = np.asarray(push_right, float)
w = np.asarray(amount, float)[:, None]
u = dL + dR
n = np.linalg.norm(u, axis=1, keepdims=True)
u = np.divide(u, np.maximum(n, _EPS), where=n > _EPS)
u[(n <= _EPS).ravel()] = 0.0
out = []
for d in (dL, dR):
    m = np.linalg.norm(d, axis=1, keepdims=True)
    out.append(d + w * (u * m - d))
return out[0], out[1]

The coupling amount is explored per frame. This correction method cannot keep hands coupled while exiting the body in some frames, especially on the rounded abdomen’s midline. In such cases, coupling is weakened to prioritize exiting the body. The body takes precedence.

# pipeline/clearance/run_solve.py:49 (comments omitted)
amt = couple.ease(_allowed(field, PL, PR, got, meta["radii"], margin, amt))

_allowed explores how much coupling each frame can accept while exiting the body, and couple.ease smooths the result temporally.

Do Not Smooth Explored Values Afterward

Explored results are step-like. Smoothing them normally would exceed capacity limits or fall below necessity thresholds. It’s crucial to treat upper and lower bounds separately.

Thus, one-sided smoothing is used:

• Upper limit (capacity): Contract → Blur → Minimum (ease). Stays within original capacity
• Lower limit (necessity): Expand → Blur → Maximum (hold). Stays above original necessity

Actual usage follows this pattern. Capacity for coupling hands uses ease, while necessity for finger escape uses hold.

This processing preserves explored upper and lower limits. The exploration itself doesn’t guarantee collision avoidance, a concept revisited in the next part on thumbs (where three mistakes were made).

Finger Penetration Into the Opposite Hand

Finally, a compromise rather than a fix: finger interlocking isn’t handled in hand estimation and correction. Each hand is estimated independently, unaware of the other, causing fingers to penetrate the opposite hand when clasped.

The solution is to symmetrically open fingers along the wrist-connecting line until fingertips no longer penetrate the opposite wrist. Positive values allow penetration, while negative values stop fingers before the wrist. The adopted value of -3.5cm was chosen after comparing renders of three values.

Finger Penetration Allowance	Appearance
+2.5cm	Fingers clearly protrude from clasped hands
0	Penetration eliminated
-3.5cm	Gap between hands (adopted)

Uninterlocked fingers are better separated than fused. This compromise prioritizes appearance over fidelity. Handling finger interlocking in estimation would eliminate this issue, but it’s a current implementation limitation.

Summary

• Determining push direction with a single nearest vertex causes jitter (arm trajectory second derivative peak increased from 3 to 9 mm/frame² in desk material). Use a weighted average of 16 vertices
• Forearm clipping requires moving the elbow to avoid over-pushing due to leverage
• Direction determined clip-wide, amount determined per frame
• For near-equal depths, weakly prefer the camera-indicated side (detected hand presence), but this is "camera side," not "body front"
• Dynamic programming selects the best among generated candidates; finite penalties don’t guarantee non-clipping
• Smooth explored values with one-sided operations to preserve limits
• For unsupported features (interlocked fingers), document compromise choices

Next (Part 5), we address thumbs. Simply moving them away from the index finger results in a thumbs-up grip. The focus will be on target placement and smoothing pitfalls.

Series Links

• Part 1: Animating VRM from Live-Action Video ── Fixing Remaining Issues in Feet, Arms, and Fingers
• Part 2: Head-Chest Angle at 0.000° for All Frames ── Motion Measurement Breakdown
• Part 3: 1% Shorter Legs Caused 16° Knee Bend ── Pitfalls in VRM Grounding Correction
• Part 4 (this article): Arm Clipping Correction Flipping Front to Back ── Direction Determination via Dynamic Programming
• Part 5: Moving Thumbs Away Results in Thumbs-Up ── Grip Preservation and Smoothing
• Part 6: Wrist Skin Creasing Caused by Skinning, Not Pose

Notes

This pipeline is based on squall01337/mixamo-llm-mocap (MIT). The pipeline/clearance/ code quoted is an addition to that base.

The quoted code, added after forking, is not publicly available. Testing cannot be replicated based on the repository (file names and line numbers are provided for reference).

Code is excerpted for explanation. Refer to the linked sections for full implementations, including omitted initializations and helper functions. Line numbers reflect the time of writing.