Hiding the seam: making a lip-synced mouth stop looking pasted on

Lip-sync models like Wav2Lip give you a small, correct mouth. Then you paste that mouth back onto the face and the illusion collapses. You can see the rectangle. The skin tone is off by a shade, the lighting does not match, and the jaw flickers between frames. The network did its job. The compositing gave it away.

The interesting code in a lip-sync pipeline is not the model. It is everything that happens after the model, in the few dozen lines that make a generated patch belong to a real face. This is a walkthrough of that part.

The eye goes straight to the mouth on a talking face, which is exactly why a sloppy paste there is so obvious. Photo: Unsplash.

Where the seam comes from

Wav2Lip (Prajwal et al., ACM Multimedia 2020) takes a masked face plus an audio chunk and predicts the lower face for that audio. You get back a crop the size of the mouth box. The model never saw the full frame, the exact skin tone of the cheeks, or the lighting on the chin. So when you drop the crop in:

ff = f.copy()
ff[y1:y2, x1:x2] = p   # p is the predicted mouth, f is the original frame

you get a hard edge where the crop meets the face, a color step across that edge, and detail that does not match the resolution of the rest of the head. Three separate problems. The fix is three separate tools, stacked.

Tool 1: restore the detail (GFPGAN)

The raw prediction is a little soft. Before blending anything, the patched frame goes through GFPGAN (Wang et al., CVPR 2021), a face restoration GAN, to bring the mouth detail up to the same sharpness as the surrounding skin:

_, _, restored_img = self.gfpgan.enhance(
    ff, has_aligned=False, only_center_face=True, paste_back=True
)

only_center_face=True keeps it from wasting work on faces in the background. Now we have a sharp version of the whole face, but pasting that whole face back would change the eyes and cheeks too. We only want its mouth.

Tool 2: take only the mouth, with a soft mask

This is the step that separates a clean result from a visibly fake one. Instead of a rectangle, use a face-parsing network to cut out only the mouth and lips, then feather the edge.

# select the mouth / lip classes from the face parser
mm = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 255, 255, 255, 0, 0, 0, 0, 0, 0]
mouth_mask = np.zeros_like(restored_img)
tmp_mask = self.face_enhancement.faceparser.process(restored_img[y1:y2, x1:x2], mm)[0]
tmp_mask_resized = cv2.resize(tmp_mask, (x2 - x1, y2 - y1))[:, :, np.newaxis] / 255.
mouth_mask[y1:y2, x1:x2] = tmp_mask_resized

That mm list is a class selector. The face parser labels every pixel (skin, eyes, nose, lips, and so on), and setting only a few entries to 255 keeps just the mouth region. The result is a mask shaped like an actual mouth, not a box. That shape is what lets the blend follow the lips instead of cutting across the cheek.

Tool 3: blend at every scale (Laplacian pyramid)

A single alpha blend along the mask edge still shows a line, because skin has detail at many scales and a hard cut mixes them all at once. The classic answer is a Laplacian pyramid blend (Burt and Adelson, 1983): split both images into frequency bands, blend each band with a correspondingly blurred mask, then add the bands back. Low frequencies merge over a wide area, high frequencies merge over a narrow one, and the seam disappears.

def laplacian_pyramid_blending_with_mask(A, B, m, num_levels=6):
    GA, GB, GM = A.copy(), B.copy(), m.copy()
    gpA, gpB, gpM = [GA], [GB], [GM]
    for _ in range(num_levels):                 # build Gaussian pyramids
        GA, GB, GM = cv2.pyrDown(GA), cv2.pyrDown(GB), cv2.pyrDown(GM)
        gpA.append(np.float32(GA)); gpB.append(np.float32(GB)); gpM.append(np.float32(GM))

    lpA, lpB, gpMr = [gpA[-1]], [gpB[-1]], [gpM[-1]]
    for i in range(num_levels, 0, -1):          # Laplacian = level minus upscaled next level
        lpA.append(np.subtract(gpA[i-1], cv2.pyrUp(gpA[i])))
        lpB.append(np.subtract(gpB[i-1], cv2.pyrUp(gpB[i])))
        gpMr.append(gpM[i-1])

    LS = [la * gm[:, :, None] + lb * (1 - gm[:, :, None])   # blend each band by the mask
          for la, lb, gm in zip(lpA, lpB, gpMr)]

    out = LS[0]                                  # reconstruct
    for i in range(1, num_levels):
        out = cv2.add(cv2.pyrUp(out), LS[i])
    return out

The pipeline runs this at num_levels=10 on 512x512 crops. Ten bands is more than a still photo needs, but video punishes any visible edge frame after frame, so it is worth the extra levels.

Tool 4: match the light (Poisson blending)

The pyramid blend fixes texture. It does not fix a global color or brightness difference between the patch and the face. For that the final step is Poisson blending (Pérez, Gangnet, Blake, SIGGRAPH 2003), which pastes by matching gradients instead of pixels. It keeps the detail of the mouth but reads the color and lighting from the surrounding face, so the patch takes on the skin tone around it.

f, _, _ = self.face_enhancement.process(pp, f, bbox=c, possion_blending=True)

After this, there is no patch. There is a face that happens to be saying something new.

The bonus: a running mean that kills jaw jitter

One more problem only shows up in motion. Face detection wobbles a pixel or two every frame, so the mouth box jumps, and a jumping box makes the chin vibrate even when the blend is perfect.

The fix is cheap. Keep a short history of recent mouth centers and reject any new detection that jumps too far from the running average:

center_current = np.array([(x1 + x2) / 2, (y1 + y2) / 2])
len_coords_mouth = 3      # history length
max_distance_mouth = 20   # pixels

if len(coords_mouth) < len_coords_mouth:
    coords_mouth.append(center_current)
    # ... do the full blend for this frame ...
else:
    distances = [calculate_distance(center_current, c) for c in coords_mouth]
    if np.mean(distances) < max_distance_mouth:
        # close enough: accept and blend
        ...
    coords_mouth.pop(0)               # slide the window
    coords_mouth.append(center_current)

Three frames of memory and a 20-pixel threshold. That is the entire anti-jitter system, and it removes most of the chin shake that makes cheap lip-sync look cheap.

How the model is even asked to invent a mouth

Worth one paragraph, because it explains why the blend has anything to blend. Before inference, the bottom half of the reference face is zeroed out and stacked with the unmasked face as extra channels:

img_masked = img_batch.copy()
img_masked[:, img_size // 2:] = 0
img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.

The model sees the top of the face for identity and pose, and a blank lower half it must fill from the audio. That is why the output is a plausible new mouth rather than a copy of the original one. It also deserves its own post, which is coming.

---

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

---

Takeaway

The model gives you a correct mouth. Making it a believable mouth is four steps of classical image work plus three frames of memory: restore detail, cut to a mouth-shaped soft mask, blend across scales with a Laplacian pyramid, match the light with Poisson, and stabilize the box over time. None of it is deep learning. All of it is the reason the result holds up.

The full function is in lip_sync/sync.py in the Wunjo Make repo. If you have been fighting a visible patch in your own pipeline, the mask shape and the Poisson step are where I would start.

References

• Prajwal, Mukhopadhyay, Namboodiri, Jawahar. "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild." ACM MM 2020. arXiv:2008.10010
• Wang, Li, Zhang, Shan. "Towards Real-World Blind Face Restoration with Generative Facial Prior." CVPR 2021. arXiv:2101.04061
• Pérez, Gangnet, Blake. "Poisson Image Editing." ACM SIGGRAPH 2003.
• Burt, Adelson. "The Laplacian Pyramid as a Compact Image Code." IEEE Trans. Communications, 1983.