DEV Community

Cover image for Why lip-sync blacks out the bottom half of the face before the model sees it
Wlad Radchenko
Wlad Radchenko

Posted on

Why lip-sync blacks out the bottom half of the face before the model sees it

#python #machinelearning #computervision #deeplearning

A lip-sync model has one job: draw a mouth that matches the audio. So the obvious input is a face and a sound. Feed in the picture, feed in the speech, get back a talking face.

That is not what the code feeds it. Before a single frame reaches the model, the bottom half of every face is set to zero. Pure black. The model never sees the real mouth it is supposed to reproduce.

The first time I read this in the data generator, I assumed it was a bug or some leftover debug code. It is neither. Blanking the lower face is the whole reason the model learns to lip-sync instead of learning to copy. This post walks through the few lines that build that input, in datagen, the generator that batches frames and audio for Wav2Lip.

Half-lit portrait
Give the model the lit half and ask it to invent the rest. Photo: Unsplash.

The setup: frames in, mel chunks in

datagen is a generator. It takes a list of frame file paths, a list of mel-spectrogram chunks (the audio, sliced into short pieces), the target face size, and a batch size. It yields batches the model can run.

def datagen(self, frame_files, mels, img_size, wav2lip_batch_size, crop=None):
    img_batch, mel_batch, frame_batch, coords_batch = [], [], [], []
    face_det_results = self.face_detect_with_alignment_crop(frame_files, crop)
Enter fullscreen mode Exit fullscreen mode

First it runs face detection once over all the frames, so it knows where every face sits. Then it walks the audio, not the video:

for i, m in enumerate(mels):
    idx = i % len(frame_files)
Enter fullscreen mode Exit fullscreen mode

That idx = i % len(frame_files) is small but it sets the rhythm of the whole loop. The driver is the audio. There is one mel chunk per output frame, and each chunk needs a face to draw on. The modulo just wraps the frame index back to the start when the audio runs longer than the video. Twenty seconds of speech over a five-second clip? The clip loops. You never run out of faces, because the frame list is treated as circular.

Be precise about which list leads. If you have more audio than video, you get more output frames than input frames and the source clip repeats. If you have more video than audio, the loop stops when the mel chunks stop, and the tail of the clip is unused. The mel list is the clock.

Crop the face, resize to a square

For each audio chunk, grab the matching frame and the detected face box:

frame_to_save = cv2.imread(frame_files[idx])
dets = face_det_results[idx] if idx < len(face_det_results) else face_det_results[i % len(face_det_results)]
coords = dets[0].get("bbox") if len(dets) > 0 and dets[0] is not None and isinstance(dets[0], dict) else (0, 0, 0, 0)
x1, y1, x2, y2 = map(int, coords)
Enter fullscreen mode Exit fullscreen mode

Clamp the box to the image, then either cut out the face and resize it to a square, or, if no face was found, resize the whole frame as a fallback:

if int(x2 - x1) == 0 and int(y2 - y1) == 0:
    img_batch.append(cv2.resize(frame_to_save, (img_size, img_size)))
else:
    face = frame_to_save[int(y1): int(y2), int(x1):int(x2)]
    img_batch.append(cv2.resize(face, (img_size, img_size)))

mel_batch.append(m)
frame_batch.append(frame_to_save)
coords_batch.append((y1, y2, x1, x2))
Enter fullscreen mode Exit fullscreen mode

Notice it keeps two parallel things. img_batch holds the resized face crops the model will work on. frame_batch holds the original full frames, untouched, and coords_batch holds where each face was. The full frame and the coordinates are not for the model. They are for later, when the predicted mouth has to be pasted back into the real picture. The model only ever sees the square crop.

The trick: blank the lower half, then stack the original on top

Here is the part that matters. Once a batch is full, it gets assembled:

if len(img_batch) >= wav2lip_batch_size:
    img_batch, mel_batch = np.asarray(img_batch), np.asarray(mel_batch)

    img_masked = img_batch.copy()
    img_masked[:, img_size // 2:] = 0

    img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.
    mel_batch = np.reshape(mel_batch, [len(mel_batch), mel_batch.shape[1], mel_batch.shape[2], 1])

    yield img_batch, mel_batch, frame_batch, coords_batch
    img_batch, mel_batch, frame_batch, coords_batch = [], [], [], []
Enter fullscreen mode Exit fullscreen mode

Three lines do the real work. Take them one at a time.

Copy the faces.

img_masked = img_batch.copy()
Enter fullscreen mode Exit fullscreen mode

We need two versions of the same faces, so we copy first. The original stays intact.

Black out the bottom half of the copy.

img_masked[:, img_size // 2:] = 0
Enter fullscreen mode Exit fullscreen mode

img_batch has shape (batch, height, width, channels). The slice [:, img_size//2:] is "every image, every row from the vertical middle downward." Setting it to 0 paints those rows black. The face is square and roughly centered, so the middle row falls around the nose. Everything below, the mouth, the jaw, the chin, goes dark. What survives is the top half: eyes, brow, hairline, head angle.

Think of it like a witness sketch. You give the artist the eyes and the shape of the head and say "the rest matched this voice, draw it." The model gets identity and pose from the top half, and it has to invent the mouth from the audio. It cannot cheat by reading the answer off the bottom of the picture, because the bottom of the picture is gone.

Stack the untouched face behind the masked one.

img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.
Enter fullscreen mode Exit fullscreen mode

This is the line people miss. axis=3 is the channel axis. A normal image has 3 channels (BGR). Concatenating the masked face and the original face along that axis makes a 6-channel image: channels 0 to 2 are the masked face (top half only), channels 3 to 5 are the full, unmasked face.

So the model gets both at once, pixel-aligned:

  • The masked half tells it where to draw and whose face this is.
  • The full face is a reference. It is the same person, in this lighting, at this resolution, with this skin tone, just from a different moment in the clip.

Why hand over the full face if the point was to hide the mouth? Because there is a difference between hiding the answer for this frame and hiding what this person looks like. The masked input removes the mouth for the audio chunk being generated. The reference shows the model the subject's appearance so the painted mouth comes out the right color and sharpness and clearly belongs to that face. Wav2Lip (Prajwal et al., ACM Multimedia 2020) is built around this: the reference is a different timestep than the masked frame, so the network cannot just copy pixels straight across. It has to actually paint a new mouth in the right style. Same person, wrong moment. Learn to paint, not to copy.

The whole thing is divided by 255. at the end to put pixels in [0, 1].

The mel chunk gets a trailing channel dimension so it reads as a single-channel image too:

mel_batch = np.reshape(mel_batch, [len(mel_batch), mel_batch.shape[1], mel_batch.shape[2], 1])
Enter fullscreen mode Exit fullscreen mode

Now both inputs are image-shaped tensors, and the batch is yielded.

The tail batch

The same masking and concatenation are repeated once more after the loop, for whatever is left over when the audio does not divide evenly into full batches:

if len(img_batch) > 0:
    img_batch, mel_batch = np.asarray(img_batch), np.asarray(mel_batch)
    img_masked = img_batch.copy()
    img_masked[:, img_size // 2:] = 0
    img_batch = np.concatenate((img_masked, img_batch), axis=3) / 255.
    ...
    yield img_batch, mel_batch, frame_batch, coords_batch
Enter fullscreen mode Exit fullscreen mode

Duplicated logic, easy to overlook if you only read the main loop. If you ever change the masking, change it in both places, or your last partial batch will be built differently from every full one.

Gotchas you will actually hit

A few things bit me, or would have:

  • Six channels, not three. If you write your own model loader or swap checkpoints, the first conv layer expects 6 input channels. Hand it a 3-channel image and the shapes will not match. The doubling happens here, not in the model.
  • The masked half is exactly half, by a hardcoded fraction. img_size // 2 assumes the face is roughly centered and upright in the square crop. If your face detector returns a loose or tilted box, the cut can land on the lip line or the nose. The face detection and the masking are coupled even though they live in different functions.
  • Channel order. Crops are read with cv2.imread, so they are BGR. Both halves of the 6-channel tensor are BGR. Keep that consistent with whatever the checkpoint was trained on.
  • Audio length decides output length. Because the loop iterates mels and wraps the frame index with %, the number of output frames equals the number of mel chunks, not the number of input frames. Short video, long audio: the clip loops. That is by design, but it surprises people who expect output length to match the source video.
  • The full frame is not model input. frame_batch and coords_batch ride along only so the predicted mouth can be placed back later. Do not feed them to the network.
Wlad Radchenko About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

Wrap-up

The interesting decision in lip-sync input is not "give the model a face and some audio." It is "give the model a face with the mouth deleted, plus a clean reference of the same person, and make the audio fill the gap." The masking forces the model to generate from sound. The reference keeps the result looking like the right person. Two faces stacked into six channels, one of them half-erased.

The pasting-back and seam-hiding is a separate problem with its own tricks, covered in another post. This one stops at the input, because the input is where the lip-sync actually gets taught.

Code: portable/src/visual_processing/lip_sync/sync.py in Wunjo Make, the datagen method.

References

Top comments (0)