You animate a still portrait with LivePortrait. The face moves, the mouth talks, the eyes blink. And then you notice it: the head floats. It drifts a little off the neck, the jaw slides past the shoulders, and the whole thing reads like a paper mask on a stick. The puppet look.
I hit this in Wunjo Make while wiring up the portrait animation path. The fix is not a bigger generator or a fancier warp. It is three small multi-layer perceptrons that predict tiny corrections to the face keypoints. One keeps the head attached to the body. One controls the eyes. One controls the lips. They are so small you could miss them in the repo, and they are the reason the output stops looking like a puppet.
This post walks through what those networks take in, what they output, and how a handful of numbers gets added back to the keypoints to fix the artifact. All of it from the actual code in the repo.
The input is one still photo. Everything after is keypoints and deltas. Photo: Unsplash
The one network class, used three ways
Here is the whole network definition. It lives in portable/src/visual_processing/portrait_animation/modules/stitching_retargeting_network.py:
class StitchingRetargetingNetwork(nn.Module):
def __init__(self, input_size, hidden_sizes, output_size):
super(StitchingRetargetingNetwork, self).__init__()
layers = []
for i in range(len(hidden_sizes)):
if i == 0:
layers.append(nn.Linear(input_size, hidden_sizes[i]))
else:
layers.append(nn.Linear(hidden_sizes[i - 1], hidden_sizes[i]))
layers.append(nn.ReLU(inplace=True))
layers.append(nn.Linear(hidden_sizes[-1], output_size))
self.mlp = nn.Sequential(*layers)
def forward(self, x):
return self.mlp(x)
That is it. Linear, ReLU, Linear, ReLU, Linear. No convolutions, no attention, no recurrence. A plain stack of fully connected layers. The interesting part is that this same class is instantiated three times with different sizes, and each instance solves a different problem.
The sizes come straight from the model config in live_portrait_wrapper.py:
'stitching_retargeting_module_params': {
'stitching': {
'input_size': 126, 'hidden_sizes': [128, 128, 64], 'output_size': 65
},
'lip': {
'input_size': 65, 'hidden_sizes': [128, 128, 64], 'output_size': 63
},
'eye': {
'input_size': 66, 'hidden_sizes': [256, 256, 128, 128, 64], 'output_size': 63
}
}
Those numbers are the whole story. To read them you need one fact: LivePortrait represents a face as 21 implicit 3D keypoints. The config sets num_kp: 21. Twenty one points, each with an x, y, z. So one full set of keypoints is 21 * 3 = 63 numbers. Hold onto 63 and 21, because every size below is built from them.
Stitching: keep the head on the shoulders
Think about what causes the floating head. The animation pipeline takes the source keypoints (the still photo) and the driving keypoints (the motion you want), warps the source feature volume from one to the other, and decodes an image. The warp is good at the face. It does not know or care that the face has to connect to a neck and shoulders that are not moving. So the face can shift as a block and detach from the body.
Stitching is a learned correction that nudges the driving keypoints so the face lines up with where the body already is. Plain version: before we render, we ask a small network "given where the face started and where you want it to go, what small shift puts it back on the shoulders?" Then we add that shift.
The input is 126 numbers. That is 63 for the source keypoints plus 63 for the driving keypoints, flattened and concatenated. Here is the concat, from utils/helper.py:
def concat_feat(kp_source: torch.Tensor, kp_driving: torch.Tensor) -> torch.Tensor:
# kp_source: (bs, k, 3), kp_driving: (bs, k, 3) -> Return: (bs, 2k*3)
feat = torch.cat([kp_source.view(bs_src, -1), kp_driving.view(bs_dri, -1)], dim=1)
return feat
So the stitching MLP sees both poses at once: where the head is anchored, and where the animation wants it. Now the output, 65 numbers. Not 63. The extra two matter. Here is how the wrapper unpacks them in stitching():
def stitching(self, kp_source, kp_driving):
bs, num_kp = kp_source.shape[:2]
kp_driving_new = kp_driving.clone()
delta = self.stitch(kp_source, kp_driving_new)
delta_exp = delta[..., :3*num_kp].reshape(bs, num_kp, 3) # 63 -> (21, 3)
delta_tx_ty = delta[..., 3*num_kp:3*num_kp+2].reshape(bs, 1, 2) # the last 2
kp_driving_new += delta_exp
kp_driving_new[..., :2] += delta_tx_ty
return kp_driving_new
Read it slowly. The first 63 outputs are a per-keypoint 3D delta. Each of the 21 points gets its own small x, y, z correction. Those get added to every keypoint. The last two outputs are a single global translation in x and y, added to all keypoints at once. So the network can both reshape the face locally and slide the whole face as one piece to meet the shoulders. That global tx, ty is the literal "stop floating" knob.
The correction is small by design. Note that the network has an initialize_weights_to_zero method that zeros every weight and bias. Trained from a zero start, the natural resting state of the delta is "change nothing," and the network only learns to add a nudge where a nudge is needed. A correction network, not a generator.
Eye and lip retargeting: control the blink and the mouth
The other two instances solve a different problem. When you drive one person's face with another's, the eyes and lips do not always transfer cleanly. A person with small eyes driving a person with large eyes leaves the large eyes half open when they should be shut. Lips that should close stay slightly parted. The driving motion is in the right direction but the wrong amount.
Retargeting fixes the amount. The trick: instead of feeding the network a pose, you feed it a target. You tell it "I want the eyes this open" or "I want the lips this closed," and it returns the keypoint delta that gets you there.
That target is a ratio computed straight from facial landmarks, in utils/retargeting_utils.py:
def calculate_distance_ratio(lmk, idx1, idx2, idx3, idx4, eps=1e-6):
return (np.linalg.norm(lmk[:, idx1] - lmk[:, idx2], axis=1, keepdims=True) /
(np.linalg.norm(lmk[:, idx3] - lmk[:, idx4], axis=1, keepdims=True) + eps))
def calc_eye_close_ratio(lmk, target_eye_ratio=None):
lefteye_close_ratio = calculate_distance_ratio(lmk, 6, 18, 0, 12)
righteye_close_ratio = calculate_distance_ratio(lmk, 30, 42, 24, 36)
...
def calc_lip_close_ratio(lmk):
return calculate_distance_ratio(lmk, 90, 102, 48, 66)
A close ratio is vertical opening over horizontal width. For an eye: distance between the top and bottom lids divided by the eye's corner-to-corner width. A wide open eye has a high ratio, a shut eye a low one. Dividing by width makes it scale-free, so it works whether the face is big or small in the frame. The eps = 1e-6 is there so a degenerate landmark set never divides by zero.
Now the sizes click into place. The lip network takes input_size 65: that is 63 for the source keypoints plus a lip ratio of 2 numbers. From the wrapper:
def retarget_lip(self, kp_source, lip_close_ratio):
# kp_source: BxNx3, lip_close_ratio: Bx2
feat_lip = concat_feat(kp_source, lip_close_ratio)
delta = self.stitching_retargeting_module['lip'](feat_lip)
return delta.reshape(-1, kp_source.shape[1], 3) # -> (B, 21, 3)
The eye network takes input_size 66: 63 keypoints plus an eye ratio of 3 numbers (left eye, right eye, and a target). Both output 63, a per-keypoint 3D delta that you add to the source keypoints. The eye network has the deeper stack, [256, 256, 128, 128, 64], because opening and closing eyes the right amount across identities is the harder regression. Same class, more layers.
The pattern is identical to stitching: feed in the current keypoints plus a small control signal, get back a keypoint delta. The only difference is what the control signal is. For stitching it is the driving pose. For retargeting it is a ratio you can dial yourself.
How it actually runs
These three calls compose. In the wrapper you can see the full chain when you control everything by hand:
eyes_delta = self.retarget_eye(x_s_user, combined_eye_ratio_tensor)
lip_delta = self.retarget_lip(x_s_user, combined_lip_ratio_tensor)
x_d_new = x_s_user + eyes_delta + lip_delta
x_d_new = self.stitching(x_s_user, x_d_new)
Deltas just add. You take the source keypoints, add the eye correction, add the lip correction, then run stitching last so the final pose is glued back to the body after you have changed the eyes and lips. Order matters: stitch after retargeting, not before, so the global tx, ty accounts for everything you moved.
Every call runs under torch.no_grad(). These are inference-only correction heads. They are also cheap. A 126-input MLP with three hidden layers is a rounding error next to the appearance extractor and the dense motion network, which is why LivePortrait can offer this control without a real frame-rate cost.
Gotchas worth knowing
A few things I learned the slow way.
Stitching is on by default, retargeting is not. In the config, flag_stitching defaults to True and the eye and lip retargeting flags default to False. The comments in argument_config.py flag eye and lip retargeting as "not recommend to be True, WIP." Stitching is the safe, always-want-it correction. Eye and lip retargeting are an expressive extra that can introduce jitter, which is why in the Wunjo path the per-frame deltas get run through a Kalman filter before they are applied. If your blink flickers frame to frame, that smoothing is the lever.
Stitching does not load by default for animals. The same comment notes you want stitching off for large head movement or for animal sources. The networks were trained on the human face keypoint layout, so the corrections only make sense there.
The whole module is optional. Look at the wrapper init: if checkpoint_S is missing, self.stitching_retargeting_module = None, and stitching() returns the driving keypoints untouched. The pipeline still runs. It just runs with the floating head. That is the cleanest way to see what these tiny networks buy you: delete them and watch the head detach.
 |
About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn. |
Wrap-up
The lesson I keep relearning: the artifact you cannot stand is often fixed by the smallest part of the system. Here the floating head, the puppet look, the half-open eyes, all come down to three ReLU MLPs that predict a few keypoint deltas and add them back. 126 numbers in, 65 out for stitching. 65 in, 63 out for lips. 66 in, 63 out for eyes. Everything else in the pipeline is huge by comparison, and none of it would look right without these.
If you want to read the code yourself, start at stitching_retargeting_network.py for the class, then live_portrait_wrapper.py for stitching, retarget_eye, and retarget_lip. The full repo is open source on GitHub.
References
- Guo, Zhang, Liu, et al. "LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control." 2024. arXiv:2407.03168. https://arxiv.org/abs/2407.03168
Top comments (0)