DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

Thumbs Up with a Twist - Correction and Smoothing of Grip

#python #3dcg #blender

📝 Originally published (in Japanese) at forge.workstyle.tech.

Introduction to the Fifth Article in the Series

This is the fifth article in a series on creating VRM animations from live-action videos. The previous article discussed how to determine the direction to move the arm to avoid intersection.

The Problem of Thumb Penetration

When the thumb penetrates the index finger, simply pushing it away can change the original gesture. The goal is to position the thumb on top of the index finger.

Defining the Target Pose

The target pose is defined as the point 18mm away from the middle of the index finger, towards the palm. This definition is based on the observation that when grasping, the thumb tends to rest on top of the index finger.

Measuring Distance

The distance between the thumb and index finger is measured using the minimum distance between line segments. The threshold for penetration is set to 18mm, which is the sum of the radii of the thumb and index finger.

# pipeline/clearance/thumb.py:25
CLEARANCE = 0.018     # thumb radius + finger radius, metres
Enter fullscreen mode Exit fullscreen mode

Four Attempts to Push the Thumb Away

Initially, four attempts were made to push the thumb away by rotating the base joint. However, this resulted in the thumb standing up, creating a different gesture.

Turning the Thumb Towards the Index Finger

The correct approach is to turn the thumb towards the index finger, rather than pushing it away. This is based on the observation that when grasping, the thumb tends to rest on top of the index finger.

Smoothing the Gate

The gate, which represents the amount of correction needed, is smoothed over time to remove short bursts and fill short gaps.

# pipeline/clearance/thumb.py:134-137
n = max(int(open_frames), 1)
g = _rank(_rank(g, n, np.max), n, np.min)     # close: fill the short gaps
g = _rank(_rank(g, n, np.min), n, np.max)     # open: drop the short bursts
return _blur(g, sigma)
Enter fullscreen mode Exit fullscreen mode

Five Failures in Smoothing the Gate

There were five failures in smoothing the gate, including:

  1. Using a binary threshold, which resulted in the thumb moving in and out of the index finger.
  2. Making the gate too weak, which resulted in the thumb not moving enough.
  3. Adding a margin to the gate, which did not improve the result.
  4. Using a step function to adjust the gate, which resulted in the thumb moving in a step-like motion.
  5. Applying the gate to the wrong pose, which resulted in the thumb not moving correctly.

Smoothing the Hand Track

The hand track is smoothed using a median filter and a Gaussian filter.

# pipeline/hand_smooth.py:68-73
if w > 1:
    for k in range(half, m - half):
        out[k] = np.median(A[k - half:k + half + 1], axis=0)
if sigma > 0:
    from scipy.ndimage import gaussian_filter1d
    out = gaussian_filter1d(out, float(sigma), axis=0, mode="nearest")
Enter fullscreen mode Exit fullscreen mode

Conclusion

The key takeaways from this article are:

  • Define the target pose for the thumb based on the observation that it tends to rest on top of the index finger when grasping.
  • Smooth the gate over time to remove short bursts and fill short gaps.
  • Apply the gate to the correct pose to ensure the thumb moves correctly.

The next article will discuss how to fix the skinning issue that causes the wrist to crease.

Series of Articles

Additional Notes

The code snippets provided are excerpts from the actual implementation and may not be complete. The references to the code files and line numbers are provided for further reading. The pipeline is based on the squall01337/mixamo-llm-mocap repository, and the modifications made to the code are not publicly available.

Top comments (0)