The angle between the head and chest was 0.000 in every frame: The time my motion tracking was broken

📝 Originally published (in Japanese) at forge.workstyle.tech.

The character's head would tilt along with the body whenever it twisted. When I measured the angle difference between the head and chest, it was 0.000° for every frame. I ended up redoing the head orientation logic downstream three times before realizing the cause.

The issue was that the nose and ear landmarks used for measurement were synthesized from the torso's base vector. I thought I was measuring the head and chest separately, but I was actually comparing the same orientation.

This is the second installment in a series about creating VRM animations from fixed-camera live-action videos. This time, I'll organize the lessons learned from actual mistakes regarding measurement targets, coordinate systems, and verification methods.

In the previous article, I covered the entire pipeline and design decisions. Links to each installment in the series are provided at the end.

What Are We Measuring?

0.000° Means "Measuring the Same Thing Twice," Not "Fixed"

In this material, the live actor's head moved about ±10° relative to the chest, so I was suspicious of the 0.000° result for every frame. Upon investigation, I found that the head and chest measurements were not independent.

Here's the dependency diagram:

Old: Shoulder line ──→ Torso base ──┬──→ Nose/Ear (synthesized)──→ Head orientation
                             └───────────────────→ Chest orientation
                             (Two outputs from the same input)

New: SMPL mesh actual vertices (nose/ear)─────────────→ Face orientation
    Torso base ──────────────────────────────→ Chest orientation

By definition, the ear line matches the shoulder line. This means the head orientation can never be anything other than the chest orientation in the calculation. Subtracting them always results in 0. So 0.000° didn't mean "the head is fixed to the chest," but rather "I was subtracting two values created from the same input."

I ended up redoing the downstream logic three times, thinking the issue was with how the head orientation was applied in the FK (forward kinematics) stage.

The fix wasn't in the estimator itself, but in the adapter (pipeline/estimate_pose_gvhmr.py) that converts GVHMR outputs to landmarks for this pipeline. It reads the actual nose and ear vertices from the SMPL mesh and outputs a unit vector connecting their midpoint every frame. The implementation name is gaze, but it's actually an approximation of face orientation, not the eyeball's line of sight.

I also added two checks to the QA process: if the standard deviation of the head-chest angle difference is less than 2°, or if the average face elevation angle exceeds 12°, a confirmation prompt is triggered. These thresholds are designed to catch anomalies in this pipeline. They don't flag intentionally fixed head movements or constant upward gazes as defects.

Capsule Approximation Said "4mm Clearance"

This happened when creating a video of two characters fighting. To the human eye, the kicking leg was penetrating the opponent's head. The comparison script reported "4mm clearance."

The comparison used a model approximating the torso as a cylinder with a 0.16m radius and the head as a sphere with a 0.11m radius. This model is lightweight and fast, useful for estimating distances and reach. However, it doesn't represent the character's silhouette. The hood, shoulders, gloves, and boots are all outside this approximation.

When checking for overlaps with the actual skinned mesh, I found intersections in 4 frames, with a maximum of 230 overlapping face pairs.

Capsule approximation: 4mm clearance
Actual mesh: 230 face intersections

From then on, I used Blender's BVH to check for face intersections between actual meshes for penetration confirmation in this scene. BVH checks for "face intersections," not all contacts or enclosures. There's a distinction between approximations for corrections and final visual confirmation (pushing with approximate shapes is covered in Part 4).

The lesson here, in one sentence, is:

When visuals and numbers conflict, first verify if the numbers are measuring the same thing as the visuals.

Same Point Name Doesn't Mean Same Location

Measurement breakdowns often occur when two compared points aren't actually the same point. I encountered three such cases in the same session.

Comparison/Judgment	Mismatch	Observed Impact
Actor's nose vs character's Head bone	Face position vs skull base	Compared points were ~10cm apart
Shoulder line vs Neck bone	Shoulder vs neck height	Compared points were 5-10cm apart
Contact judgment using only leg endpoints	Endpoints vs line segment	Missed shin penetration when knee and ankle were outside

The third case is particularly problematic. Relying only on endpoints means you can't detect "shin penetrating the head" if both the knee and ankle are outside the target.

Another issue is mistaking body proportions for pose errors. In one clip flagged for "hand position too high relative to face," 4.6cm of the 14.8cm difference was due to the character's head being lower relative to the shoulders compared to the actor. The comparison script now separates body proportion ratios from pose errors.

Which Coordinate System Are We Measuring In?

Camera "Up" Isn't Gravity

I received feedback that a kick "lacked height." Measurements confirmed it was low, but the issue wasn't the kick itself - I was reading the height in the camera coordinate system without correcting for the camera's tilt.

The camera was tilted 4.2° downward. Without adjusting for this tilt, there was a 0.14m error in the kick's peak height relative to the head. Before judging the kick as insufficient, I needed to align the comparison coordinate systems.

In this material, the camera tilt wasn't noticeable just by watching the video. Even when mounted on a tripod, this level of tilt is common. When measuring height, establish the gravity direction first - leading to the next point.

Compare Two People in a Shared Coordinate System

This is from a record of handling a two-person fight video (found in the forked repository's documentation).

Measuring the distance between the kicking foot and the opponent's head gave 0.18m. However, both the video and independently measured image quantities showed 0.29m. The difference between 0.18m and 0.29m is significant - "barely missing" versus "clearly missing."

The cause was that the estimator's world coordinate system normalized each person so "frame 0 faced forward." With two people, there are two separate coordinate systems. Any difference in their starting orientations creates a misalignment. Arranging their limbs in a single space produces results that seem plausible but are subtly incorrect.

For this comparison, I aligned the estimation results of both individuals to a common camera coordinate system. Note that the scale conversion from image quantities to meters and the assumptions used are not recorded in the original documentation. To reuse the 0.29m value, that conversion would need to be redone.

World and In-Camera Are Correct for Different Things

The pose estimator (GVHMR) produces two types of outputs for the same body:

• World: Estimation results in a gravity-aligned coordinate system. In this material, position and height drift were problematic
• In-camera: Estimation results in the camera coordinate system. Easier to compare with images, but depth estimation errors and camera tilt need separate handling

Both are estimation results. It's not a contrast between "world as prediction and in-camera as measurement." Mixing these two can lead to bugs with unclear failure modes. Here are some real-world examples (different materials and comparisons in each row, so don't compare numbers directly):

Material	Symptom	What Was Compared	Numbers
Plaza	Ground rises over time	World height time change	0.25m over 3 minutes
Fountain	Jump height not captured	World upward movement vs image upward movement	0.02-0.07m vs 0.12-0.26m
Plaza	Backwards-starting material flips entirely	Normalized orientation vs video orientation	180°
Material with approach (241 frames)	Insufficient step-in distance	World movement vs actual step-in	Reported 0.68m for 0.92m, final 0.16m marker mismatch

The last row is a clear example. The world output seems physically consistent and "correct" intuitively, but it drifts due to model-based estimation. Meanwhile, the in-camera output matched independently measured image quantities (waist pixel count ÷ body pixel count) within 2cm for all 241 frames. Again, the conversion from pixel ratios to meters isn't recorded, so the match accuracy needs retesting.

Suspect Common Causes When Applying Multiple Fixes

At one point, I was applying separate corrections for orientation flipping, lateral movement disappearance, and jump disappearance. All three issues stemmed from using the estimator's world output for position, height, and orientation.

I discarded all the fixes and unified the coordinate system. A constant rotation is calculated from the in-camera and world joint configurations to align with gravity. This rotation transforms the camera coordinate system estimation results, while translation uses the camera fit values.

The rotation is solved using the Kabsch method. The key is subtracting the center of mass each frame, which eliminates translation and drift, leaving only orientation.

# pipeline/frames.py:37-44
def camera_to_world_rotation(joints_incam, joints_world):
    a = (joints_incam - joints_incam.mean(axis=1, keepdims=True)).reshape(-1, 3)
    b = (joints_world - joints_world.mean(axis=1, keepdims=True)).reshape(-1, 3)
    return kabsch(a, b)

Since the camera is fixed, I assume constant rotation for the entire clip here, treating all frames as one least-squares problem. Finally, I add a yaw rotation to make the lens point along +z, completing the output coordinate system with y as up and y=0 as the floor.

In this coordinate system, downstream processes like "re-estimating orientation" and "restoring movement" become unnecessary.

Two Discarded Alternatives

I tested two other approaches that seemed better but ultimately discarded them:

1. Position from image plane: When the camera is tilted, actual depth movement leaks into height. In dance_full's approach, it was 0.16m lower
2. Depth from lowest foot ray-floor intersection: Performs poorly when the pelvis ray is nearly horizontal. For chest-height shots (most of this material), depth fluctuated ±17% (fit itself was ±4%)

In conclusion, the fit's own depth was the least bad option. Remaining errors are "depth error × sin(camera tilt)," which is a few centimeters in these materials. Compared to the world's 0.25-0.35m drift, it's sufficiently small.

Verify with Quantities Not Used in Fitting

Here's an effective verification pattern.

I wrote logic to transform hand estimation results (camera coordinates) to the rig's coordinate system. The transformation rotation is calculated by matching four torso points (shoulders and hips) between the two coordinate systems.

The challenge is verifying if this rotation is truly correct. You can't confirm its applicability to other parts using only the fitted points. Naturally, the rotation matches well when verified with the torso used for fitting. While not an exact match due to shape differences between the rig and actor, this verification is weak.

So I verified with the forearms, which weren't used in fitting. The camera-coordinate forearm orientation is transformed using this rotation and compared to the rig's forearm. In the desk material, the directional difference was 3.6°. Matching corresponding points and consistency across different parts are checked separately.

The implementation calculates rotation using SVD and adds a matrix sign to prevent mirroring. Without this, SVD fits can happily return flipped rotations.

Before Removing Based on Old Measurements, Retest if Reproducible Now

There's another temporal pitfall in measurement stories: your own past measurements.

Depth errors in fitting scale the entire translation, meaning depth mistakes also raise/lower foot height and produce tilted floors relative to depth. Actual measurements showed dance_full at +4° and plaza at -13°.

I didn't measure the actual floor tilt. The tilt appearing in estimation results is corrected to align foot heights. Floor plane fitting uses iterative reweighted least squares on the lower envelope, with 3 rounds, using only the lower 20% of residuals each time, and a ridge term (walking along one line makes x and z collinear).

# pipeline/frames.py:126-134
A = np.stack([x - x.mean(), z - z.mean(), np.ones_like(x)], axis=1)
sel = np.ones(len(low), dtype=bool)
lam = np.diag([0.05, 0.05, 0.0])   # ridge: a walk along one line leaves x and z collinear
for _ in range(rounds):
    As, ys = A[sel], low[sel]
    coef = np.linalg.solve(As.T @ As + lam * len(ys), As.T @ ys)
    res = low - A @ coef
    sel = res <= np.percentile(res, keep * 100)

At one point, based on an old measurement claiming "floor tilt estimation was counterproductive in all three materials," I decided to remove this processing and actually deleted it.

Later, when retesting with 12 clips containing tilts, the old measurement wasn't reproducible. The metric was the 5th-95th percentile width of the lowest foot height (ankle or toe, whichever is lower). Smaller values mean more consistent foot heights. 9 of 12 clips worsened without correction, 2 (desk1, desk1b) improved 5-8%, and the remaining clip's result wasn't recorded.

Material	With correction: Lowest foot height p5-p95 width (m)	Without correction: Same (m)
dance_full	0.112	0.144
Clip showing 29° tilt	0.195	2.98
desk1b	0.092	0.085

Based on these results, I restored the processing and recorded the retest conditions and process in the code. When making changes based on old measurements, first verify if they're reproducible with the current code.

Appendix: 2D Detection Consistency and Smoothing Side Effects

Here are two implementation points that come into play after aligning measurements, summarized briefly.

2D Detection Matches Image Positions Better

3D body fitting can be off by centimeters for hand and foot tips. In dance_full, wrist reprojection error was 20px (p50) to 40px (p90) - 3-7cm for a ~1000px actor. Hands that touch in the video appear 5cm apart.

In this material, ViTPose's 2D detection (already run by GVHMR as preprocessing) matched image joint positions better than reprojecting 3D fit points. However, 2D detection alone doesn't determine depth.

So I keep the 3D fit's depth but use 2D detection ray points as correction targets. In practice, correction amounts toward these targets are interpolated and smoothed. The code below uses camera coordinates (x right, y down, z depth), with p[:, 2] as depth.

# pipeline/frames.py:194-196
target = np.stack([(uv[:, 0] - cx) * p[:, 2] / f,
                   (uv[:, 1] - cy) * p[:, 2] / f], axis=1)
delta = _fill_and_smooth(target - p[:, :2], ok, window)
out[:, j, :2] = p[:, :2] + delta      # z (depth) untouched

Since delta is smoothed before addition, corrected points aren't strictly on the ray each frame. Two subtle but effective tweaks:

• Skip frames with confidence <0.7. Missing frames are interpolated from neighbors and median-smoothed over 5 frames. Frame-level 2D detection jitter shouldn't affect the character
• Skip waist and shoulders. Different "waist" and "shoulder" definitions in SMPL vs COCO. Elbows and knees are corrected after subtracting clip-wide median offsets

This is detailed work checking "which joints, which definitions, which coordinate systems" one by one. Being vague here leads back to the failures in the first section.

Smoothing Halved Deceleration

I was applying a 7-frame Savitzky-Golay filter as preprocessing - a noise reduction standard. Testing on dance_full showed:

Metric	With filter (7 frames)	Without
Wrist deceleration: |acceleration| p99 (m/s²)	32	64
"Stop" event count (times)	29	51
Noise median	6.5	10.7

The "noise" in row 3 isn't defined in the records - treat the ratio as indicative.

In this material, deceleration dropped from 64m/s² to 32m/s², and "stop" events from 51 to 29. While reducing noise, it may also weaken desired sharp movements. The event reduction rate doesn't directly equal "lost sharpness," but strike deceleration is clearly dulled.

I also measured kick peaks. For a roundhouse kick peaking in ~6 frames, a 7-frame window reduced shin angle by 5° and height by 0.04m. A 5-frame window reduced angle by 1.4°.

It's now off by default. If the estimator is noisy, re-enable via spec, but it wasn't a default processing step. The same topic appears in Part 5 (hand smoothing).

Key Points

• When numbers align suspiciously, check measurement independence and rounding
• Distinguish between approximate shapes being sufficient and needing actual mesh verification
• Align not just point names but position definitions and coordinate systems when comparing
• When applying multiple fixes, investigate common upstream causes
• Before removing processing based on old measurements, retest if reproducible now

The checklist for questioning numbers has settled on these five:

1. Comparing the same points (by position definition, not just name)
2. Same coordinate system and scale (gravity direction established? conversion assumptions documented?)
3. Measurements independent (two values not created from the same source?)
4. Looking at current output (not relying on old logs or measurements?)
5. Verified with quantities not used in fitting?

Next Preview

Next time, with measurements aligned, we'll place feet on the floor. Starting from an estimation where the swinging leg penetrates the floor by 148mm, we'll cover how to detect ground contact and align feet and waist.

Series Installments

• Part 1: Animating VRM from a Single Live Video
• Part 2 (this article): Head-Chest Angle Difference of 0.000° in Every Frame - Broken Motion Measurement
• Part 3: 1% Shorter Legs Resulted in 16° Bent Knees
• Part 4: Arm Penetration Correction Jumps Front to Back
• Part 5: Thumb "Placed" on Index Finger, Not Pushed Away
• Part 6: Wrist Crease Caused by Skinning, Not Pose

Implementation & Sources

The materials and code in this article are based on squall01337/mixamo-llm-mocap (MIT). The coordinate system discussion for two-person comparisons comes from the forked repo's documentation. Coordinate unification, floor leveling, and 2D consistency were added later.

Code quoted in this article that was added after forking isn't publicly available. Assume retesting isn't possible when reading (quotes include file names and line numbers).

Code excerpts are for explanation - see the referenced files (pipeline/frames.py, pipeline/wrist_heading.py, pipeline/estimate_pose_gvhmr.py) for full implementations including initialization and helper functions.