The Boundary for Head Replacement Should be at the Chest, Not the Jaw

📝 Originally published (in Japanese) at forge.workstyle.tech.

They sent me a screenshot and said:

What I'm talking about is this situation. Do you understand? The jaw and the face are made of separate parts and are clearly misaligned, right?

At first, I thought it was a background removal issue. It wasn't.

Why Replace Just the Head?

I'm using InfiniteTalk to make portraits talk. It's a model that creates a video of a moving mouth from a single image and audio, but when it talks, the hands move too. It raises its hands in front of its face as if gesturing.

I don't want the body to move. So, I was synthesizing it like this:

Body   … Freeze on the first frame
Head   … Extract ○% from the lip-sync video and overlay it

This "○% from the top" becomes the boundary. At first, I placed it around the jaw and neck. Because it's the smallest area where the mouth and jaw move.

Misalignment at the boundary

When they speak, the head sways slightly. Everything above the boundary is video, while everything below is a still image. When the head moves horizontally by a few pixels, the outline becomes disjointed at the boundary. If this occurs at the jawline, the face and the chin look like separate parts.

I measured the horizontal offset from the first frame both above and below the boundary.

Pichai       Boundary 0.34  Max shift  +9px
Dario (seated) Boundary 0.52  Max shift +12px
Sam         Boundary 0.34  Max shift  -5px
Ive         Boundary 0.41  Max shift  +4px

Even at 4px, in a caricature with thick outlines, the shift appears as a visible step.

Lowering the Seam to the Chest

I lowered the seam to the chest area, which has fewer patterns (ratio relative to the image height).

Pichai     0.34 → 0.46 (plain sweater chest)
Ive        0.41 → 0.55
Dalio      0.34 → 0.42
Seated Dalio 0.52 → 0.57 (below this, the top of the laptop overlaps)
Sam        0.34 → 0.44 (hoodie chest)

I further blurred the seam by 34px.

format=rgba,geq=...:a='255*min(1,(H-1-Y)/34)'

The chin now moves as a single unit with the face. The misalignment wasn't eliminated but moved to a less noticeable location. On a plain chest, a few pixels of misalignment go unnoticed.

This Time, the Hand Floating Issue

Lowering the crop boundary expands the area captured from the video side. The hand raised by InfiniteTalk ends up positioned above the crop line. Since the rest of the body remains stationary, only the severed portion of the hand appears to float in front of the chest.

This issue cannot be fixed during compositing. Instead, we modified the generation process to prevent the character from raising their hand.

a hand drawn caricature character talking, mouth clearly moving with the speech,
calm and still posture, hands stay low at the waist,
no hand gestures, no raised arms, arms stay down

For seated characters, the prompt specifies "keeping both hands on the keyboard"; for standing characters, it uses "keeping hands clasped in front of the body." Regenerating the lip-sync video with these adjustments ensured that the hands remained below the crop boundary.

The Boundary Set in Common Settings Was Affected in Another Scene

I encountered another issue later on.

I had set the boundary value in the common settings for each character's image. Sam's standing picture was set to 0.44. However, this value also affected another scene where Sam spreads his arms and gestures, causing his raised hand to be cut off and float beside his body.

The boundary should be determined by the scene's direction, not the nature of the image. I removed it from the common settings and only wrote it in the script for the scene where only the head moves.

Summary

• The border is placed not at the "minimum range where the mouth moves", but at a "place where it won't be noticeable even if it's shifted". A plain chest is preferable
• If the border is lowered, the movement (hands) that enters that range should be stopped on the generation side
• The border value is held by the direction of the episode, not by the picture
• A shift of several pixels will be overlooked in a series of still images. The difference from the first frame should be measured numerically