DEV Community: Wlad Radchenko

Putting a processed face back into the frame without a visible rectangle

Wlad Radchenko — Tue, 08 Sep 2026 12:52:00 +0000

Most face pipelines do not work on the whole picture. They cut out the face, process the small crop, and then have to put it back. That last step sounds trivial. It is not. Paste the crop back with a plain assignment and you get a rectangle on the frame, a patch of skin a shade off from the cheeks around it, an edge you can trace with your finger.

This is a walk through paste_pic, the function that does the putting-back in Wunjo Make. It is short. The interesting part is two things: the coordinate bookkeeping that gets the crop to the exact pixel it came from, and one call, cv2.seamlessClone, that makes the paste read as part of the frame instead of a sticker on top of it.

To be clear up front, this is a different job from the lip-sync seam blend I wrote about earlier. That one blends only the mouth region with a Laplacian pyramid. This one pastes the whole processed face crop back into the full frame, and it leans on a single Poisson clone to do it.

The crop the model returns was lit by the model, not by the room. Pasting it back means matching the room. Photo: Unsplash.

The setup: a crop, and the note that says where it came from

Earlier in the pipeline, the face was detected and cropped out of the original image. paste_pic gets two things back: the processed frames (the crop, one per video frame) and crop_info, the receipt that records exactly where that crop was taken from.

if len(crop_info) != 3:
    print("you didn't crop the image")
    return
else:
    clx, cly, crx, cry = crop_info[1]
    oy1, oy2, ox1, ox2 = cly, cry, clx, crx

crop_info[1] is the box: left x, top y, right x, bottom y. The function renames it into the same up-down, left-right order it uses everywhere else (oy1, oy2 for the vertical span, ox1, ox2 for the horizontal). This is not busywork. Half of getting a paste right is keeping x and y straight, because OpenCV indexes arrays as [row, column], which is [y, x], the opposite of how you say coordinates out loud. Naming the four numbers once, clearly, is cheaper than debugging a transposed paste later.

The full original image is loaded separately, either as a still or as the first frame of a video:

if pic_path_type == "static":
    full_img = cv2.imread(pic_path)
else:
    # read the first frame of the source video
    full_img = frame

full_img is the canvas. The processed crops are what we paint onto it.

Resize the crop back to its real size

The model probably worked at a fixed resolution. So before the crop goes anywhere, it gets resized to the exact width and height of the box it was cut from:

p = cv2.resize(crop_frame.astype(np.uint8), (crx - clx, cry - cly))

crx - clx is the box width, cry - cly the box height. If you skip this and the processed crop is even a few pixels off the original box size, every later coordinate is wrong and the paste lands in the wrong place. Resize first, then place.

The mask is the whole crop

Here is the part that surprises people who expect a clever mask. The mask is a solid white rectangle the exact shape of the crop:

mask = 255 * np.ones(p.shape, p.dtype)

Every pixel is 255. That means "clone all of it." There is no feathering, no face-shaped cutout, no soft edge. The whole crop is fair game. The softness does not come from the mask. It comes from the clone method, which is the next line.

seamlessClone: paste the gradients, not the pixels

location = ((ox1 + ox2) // 2, (oy1 + oy2) // 2)
gen_img = cv2.seamlessClone(p, full_img, mask, location, cv2.NORMAL_CLONE)

location is the center of the box, because seamlessClone wants the center point where the patch should land, not the top-left corner. (ox1 + ox2) // 2 is the horizontal middle, (oy1 + oy2) // 2 the vertical middle. Get this wrong by treating it as a corner and the face lands half a face-width off.

Now the part that earns the function its name. A plain paste copies pixel values:

full_img[oy1:oy2, ox1:ox2] = p   # what we are NOT doing

That copies the crop's absolute colors straight in. If the crop is a touch brighter or cooler than the surrounding skin, you see the step at the border, hard and obvious.

cv2.seamlessClone with cv2.NORMAL_CLONE does something different. It is OpenCV's implementation of Poisson image editing (Pérez, Gangnet, Blake, SIGGRAPH 2003). Instead of copying the crop's pixel values, it copies the crop's gradients, the change from each pixel to its neighbor, and then solves for the actual pixel values so that the colors at the border match the surrounding frame exactly.

The everyday version: imagine you have a photo print and you want to drop a smaller print into the middle of it. A plain paste is taping the small print on top, edges and all. Poisson cloning is more like repainting the patch by hand, keeping every brushstroke and detail from the small print, but mixing the paint to match the colors already on the big print right at the seam. You keep the content of the patch. You inherit the lighting and color of the background. The border has nowhere to show up, because the two sides agree at the boundary by construction.

That is why this beats the plain overwrite. The face the model produced was lit by whatever the model imagined. The frame it goes back into was lit by the real scene. The Poisson clone lets the processed face keep its identity and detail while taking on the brightness and color of the actual frame around it. No rectangle, because there is no color step at the edge to draw one.

Two output modes, one clone

The function supports two preprocess modes, and they only differ in what they save, not in how they clone:

if preprocess == "resize":
    frame_w = ox2 - ox1
    frame_h = oy2 - oy1
else:
    frame_h = full_img.shape[0]
    frame_w = full_img.shape[1]

In crop mode (the else), the output frame is the full original size, so you get the whole scene with the new face cloned in. In resize mode, the output is just the box size, and after cloning the function trims back down to the box:

gen_img = cv2.seamlessClone(p, full_img, mask, location, cv2.NORMAL_CLONE)
if preprocess == "resize":
    gen_img = gen_img[oy1:oy2, ox1:ox2]

Note the order. Even in resize mode it clones onto the full image first, then crops the result. That matters, because Poisson cloning reads the colors of the surrounding region to set the boundary. If you cloned onto a bare box with no margin around it, there would be nothing to match against and the whole point would be lost. Clone wide, crop after.

Each finished frame is written out, and at the end the audio is muxed back on:

out_tmp.write(gen_img)
# ...
new_video_name = VideoManipulation.save_video_with_audio(tmp_path, new_audio_path, video_save_dir)

The progress bar is even labelled seamlessClone:, which tells you where the per-frame cost actually goes.

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

Gotchas worth knowing

A few things I would warn anyone about before they reach for seamlessClone:

It needs margin. The center location plus the mask must sit fully inside the destination, with room around it. Push a crop to the very edge of the frame and OpenCV will throw, because the clone needs surrounding pixels to read. The center-point math here keeps the box where it was detected, which is normally well inside the frame.
The center, not the corner. location is the center of the destination box. This is the single most common mistake with this call.
A full-white mask clones everything. That is the right choice here because the crop is already just the face region from the detector. If your crop contains background you do not want, the mask is where you would carve it out.
It is not free. Poisson cloning solves a system over the masked region every frame. On a long video that adds up, which is why the loop has its own progress bar.

Wrap-up

paste_pic is maybe forty lines and most of them are file plumbing. The two ideas that matter are small. Keep the crop coordinates honest so the face lands exactly where it was taken from, and use cv2.seamlessClone so it lands without a color step at the edge. The first is bookkeeping. The second is a 2003 paper doing the heavy lifting in one function call.

The full function is in visual_processing/utils/paste_pic.py in the Wunjo Make repo. If you are pasting processed faces back into frames and fighting a visible box, swapping a plain assignment for seamlessClone is the first thing I would try.

References

Pérez, Gangnet, Blake. "Poisson Image Editing." ACM SIGGRAPH 2003. (The method cv2.seamlessClone implements.) PDF

From a click to a mask: how point prompts drive SAM for object selection

Wlad Radchenko — Wed, 02 Sep 2026 12:51:00 +0000

You click once on a cat in a photo and a clean mask snaps to its outline. Click again on the background and the mask shrinks away from a spot you did not want. It feels instant, like the model is re-running the whole thing on every click. It is not. The expensive part ran once, when the image loaded. Every click after that is cheap.

This is a walkthrough of how that works in the Segment Anything code inside Wunjo Make. I will follow the real prompt path: encode the image once with set_image, then send each click as a tiny prompt to the decoder. The model behind it is SAM (Kirillov et al., ICCV 2023), and the split between a heavy image encoder and a light prompt decoder is the whole reason interactive selection feels live.

The heavy work happens once. After that, each click is a quick lookup against what the model already knows about the picture. Photo: Unsplash.

The one expensive step, done once

When you load an image, the predictor runs it through the image encoder and keeps the result:

@torch.no_grad()
def set_torch_image(self, transformed_image, original_image_size):
    self.reset_image()
    self.original_size = original_image_size
    self.input_size = tuple(transformed_image.shape[-2:])
    input_image = self.model.preprocess(transformed_image)
    self.features = self.model.image_encoder(input_image)
    self.is_image_set = True

That self.model.image_encoder(input_image) call is the costly one. It is a vision transformer chewing through a 1024x1024 image and producing a feature grid. The wrapper in Wunjo Make caches the output so you only pay for it once per image:

@staticmethod
def get_embedding(predictor, img):
    predictor.set_image(img)
    return predictor.get_image_embedding().cpu().numpy()

The embedding is small and fixed. For SAM it is a 1 x 256 x 64 x 64 tensor, a 64x64 grid where each cell carries a 256-dimensional description of that patch of the picture. Once you have it, you never touch the encoder again for that image. Every click reads against this grid.

If you try to predict before this runs, the code stops you cold:

if not self.is_image_set:
    raise RuntimeError("An image must be set with .set_image(...) before mask prediction.")

So the rule is simple. Set the image once. Then click as much as you want.

A click is two numbers and a label

In the browser the user clicks. The wrapper's draw_mask turns those clicks into two arrays the model understands: where the points are, and what each point means.

for point in point_list:
    # scale the click from canvas pixels to original image pixels
    scaleFactorX = originalWidth / canvasWidth
    scaleFactorY = originalHeight / canvasHeight
    newX = int(point['x'] * scaleFactorX)
    newY = int(point['y'] * scaleFactorY)
    input_point.append([newX, newY])

    # color tells us if this is a "keep" or "drop" click
    if point.get("color") == "lightblue" or point.get("color") == 1:
        input_label.append(1)   # foreground: include this
    elif point.get("color") == "red" or point.get("color") == 0:
        input_label.append(0)   # background: exclude this

That label is the whole interaction model. A label of 1 is a foreground click, meaning "this is part of the thing I want." A label of 0 is a background click, meaning "this is not part of it, push the mask away from here." Blue clicks add, red clicks subtract. The user paints intent with two colors, and the model figures out the rest.

The first click is deliberately ambiguous. One blue dot on a person could mean the shirt, the torso, or the whole body. So you click again. Each extra click narrows it down. Background clicks are how you carve away the bits SAM grabbed that you did not want.

A box is just two more points with special labels

SAM treats a bounding box as two corner points, but with their own labels so the model knows they are corners and not regular clicks:

if box is not None:
    onnx_box_coords = box.reshape(2, 2)
    onnx_box_labels = np.array([2, 3])   # 2 = top-left corner, 3 = bottom-right corner
    onnx_coord = np.concatenate([input_point, onnx_box_coords], axis=0)[None, :, :]
    onnx_label = np.concatenate([input_label, onnx_box_labels], axis=0)[None, :].astype(np.float32)

So the label vocabulary is small and fixed: 1 foreground point, 0 background point, 2 box top-left, 3 box bottom-right. You can mix them. A box to rough out the region plus a couple of foreground and background clicks to clean up the edges is a strong combination.

When there is no box, the code still appends one filler point with label -1:

else:
    onnx_coord = np.concatenate([input_point, np.array([[0.0, 0.0]])], axis=0)[None, :, :]
    onnx_label = np.concatenate([input_label, np.array([-1])], axis=0)[None, :].astype(np.float32)

That -1 is a padding label, a "not a point" marker. It exists so the input tensor has a consistent shape even when you only sent clicks. Inside the prompt encoder, those padded slots get zeroed out and replaced with a dedicated embedding so they contribute nothing to the result. More on that next.

Where the labels turn into something the model can use

The clicks are still just coordinates and integers. The prompt encoder is where they become vectors the decoder can read. This is the heart of it:

def _embed_points(self, points, labels, pad):
    points = points + 0.5  # shift to the center of the pixel
    if pad:
        padding_point = torch.zeros((points.shape[0], 1, 2), device=points.device)
        padding_label = -torch.ones((labels.shape[0], 1), device=labels.device)
        points = torch.cat([points, padding_point], dim=1)
        labels = torch.cat([labels, padding_label], dim=1)
    point_embedding = self.pe_layer.forward_with_coords(points, self.input_image_size)
    point_embedding[labels == -1] = 0.0
    point_embedding[labels == -1] += self.not_a_point_embed.weight
    point_embedding[labels == 0]  += self.point_embeddings[0].weight
    point_embedding[labels == 1]  += self.point_embeddings[1].weight
    return point_embedding

Read it from the bottom. Each point first gets a position embedding from forward_with_coords, which encodes where the click landed. Then the code adds a learned vector that encodes what kind of click it was. Label 0 gets one learned vector, label 1 gets a different one, label -1 (the padding) gets wiped to zero and given the not_a_point vector so it carries no location signal. Boxes get the same treatment with their own two learned vectors:

def _embed_boxes(self, boxes):
    boxes = boxes + 0.5
    coords = boxes.reshape(-1, 2, 2)
    corner_embedding = self.pe_layer.forward_with_coords(coords, self.input_image_size)
    corner_embedding[:, 0, :] += self.point_embeddings[2].weight  # top-left
    corner_embedding[:, 1, :] += self.point_embeddings[3].weight  # bottom-right
    return corner_embedding

There are exactly four learned point vectors, set up once in the constructor:

self.num_point_embeddings = 4  # pos/neg point + 2 box corners
point_embeddings = [nn.Embedding(1, embed_dim) for i in range(self.num_point_embeddings)]

Four small vectors. Foreground, background, box corner one, box corner two. That is the entire prompt language. Position says where, the learned vector says what role, and the two added together is what the decoder reads. The output of all this is a handful of tiny vectors, the "sparse embeddings." That is the cheap thing that flows into the decoder on every click.

The cheap step that runs on every click

Now the payoff. With the image embedding already cached, predicting a mask is just the prompt encoder plus the mask decoder:

@torch.no_grad()
def predict_torch(self, point_coords, point_labels, boxes, mask_input, multimask_output, return_logits=False):
    points = (point_coords, point_labels) if point_coords is not None else None

    # cheap: turn the clicks into a few vectors
    sparse_embeddings, dense_embeddings = self.model.prompt_encoder(
        points=points, boxes=boxes, masks=mask_input,
    )

    # cheap: the decoder reads the cached image grid against those vectors
    low_res_masks, iou_predictions = self.model.mask_decoder(
        image_embeddings=self.features,                 # cached from set_image
        image_pe=self.model.prompt_encoder.get_dense_pe(),
        sparse_prompt_embeddings=sparse_embeddings,     # from your clicks
        dense_prompt_embeddings=dense_embeddings,
        multimask_output=multimask_output,
    )
    masks = self.model.postprocess_masks(low_res_masks, self.input_size, self.original_size)
    if not return_logits:
        masks = masks > self.model.mask_threshold
    return masks, iou_predictions, low_res_masks

Notice image_embeddings=self.features. The decoder does not re-encode anything. It reads the grid the encoder produced once and lets the prompt vectors point at the part you mean. The decoder is small, the prompts are tiny, so this runs in a few milliseconds even on modest hardware. That is the entire reason clicking feels live. The cost of a click is the cost of the decoder, not the encoder.

Three masks, with scores, so ambiguity is solvable

One click is genuinely ambiguous, so by default SAM returns three candidate masks instead of guessing one:

multimask_output: bool = True

The docstring in the code spells out why:

If true, the model will return three masks. For ambiguous input prompts (such as a single click), this will often produce better masks than a single prediction. If only a single mask is needed, the model's predicted quality score can be used to select the best mask.

Each candidate comes with a predicted quality score, the iou_predictions. So you can show all three and let the user pick, or auto-pick the highest-scoring one. The return tuple is exactly that:

masks_np = masks[0].detach().cpu().numpy()                 # the masks
iou_predictions_np = iou_predictions[0].detach().cpu().numpy()  # one score per mask
low_res_masks_np = low_res_masks[0].detach().cpu().numpy()      # 256x256 logits to reuse

Once your prompts are unambiguous, say a box plus a few clicks, you can flip multimask_output to False and get a single cleaner mask. The code's own guidance: ambiguous input wants three, well-specified input wants one.

Gotchas I would warn you about

A few things that bite people working with this path:

Set the image once, not per click. The easy mistake is calling set_image inside your click handler. That re-runs the encoder every time and throws away the entire speed advantage. Encode on load, predict on click. The wrapper's draw_mask does call set_image because it is built around per-frame masking for video; if you are masking one still image across many clicks, hoist the embedding out and reuse it.
Coordinates are in original image pixels, not canvas pixels. The frontend canvas is usually a scaled-down preview. If you skip the scaleFactorX/scaleFactorY step, your clicks land in the wrong place and the mask makes no sense. Always rescale before sending.
Points are (X, Y), and the labels array must match. Point coords are [x, y], and there must be exactly one label per point. point_labels must be supplied if point_coords is supplied is an assertion in the code, not a suggestion.
The -1 padding point is not optional with the ONNX path. It keeps the input shape consistent when there is no box. Drop it and the exported model gets the wrong tensor shape.
The returned low_res_masks is reusable. Those 256x256 logits can be fed back as mask_input on the next click to refine the same object. It is the cheapest way to iterate, and it is what makes "add one more click" feel additive rather than a fresh start.

Takeaway

The mechanism is a clean split. The image encoder is heavy and runs once, producing a small cached feature grid. Each click is turned into a few tiny vectors: a position plus one of four learned role vectors (foreground, background, or a box corner), with a padding marker for empty slots. The decoder reads the cached grid against those vectors and returns candidate masks with quality scores, all in a few milliseconds. Encode once, decode per click. That asymmetry is why selecting an object by clicking feels instant.

The prompt path lives in visual_processing/segmentation/segment_anything/ in the Wunjo Make repo: predictor.py for set_image and predict, modeling/prompt_encoder.py for how points and boxes become embeddings, and segment_anything/detect.py for the click-to-label wrapper. If you build your own interactive selection, the one rule that matters is: cache the embedding, then let the clicks do the cheap work.

References

Kirillov et al. "Segment Anything." ICCV 2023. arXiv:2304.02643

Two enhancers, one frame: routing faces to GFPGAN and the rest to Real-ESRGAN

Wlad Radchenko — Tue, 01 Sep 2026 11:50:00 +0000

Run a general upscaler on a portrait and the face comes back plastic. Skin loses its pores, eyes go glassy, the whole face reads as smoothed-over wax. Run a face restorer on the same photo and the face looks great, but point it at the wall behind the person and it either ignores the wall or smears it, because it never learned what a wall is. Neither model is broken. They were trained for different jobs.

The fix in Wunjo Make is to not ask one model to do both. The enhancement code detects faces, restores each face on its own, and leaves the rest of the frame to a general upscaler. This is a walkthrough of how that split is wired, from the content_enhancer entry point down into GFPGANer.enhance, where the actual face-versus-background routing happens.

I am not covering Real-ESRGAN's tiling here. That has its own article. This one is only about which pixels go to which model and how they get glued back together.

A face and the room behind it are two different restoration problems in the same picture. Photo: Unsplash.

The entry point is a method switch, not magic

content_enhancer takes a method string and picks a restorer. It is a plain branch:

def content_enhancer(media_path, save_folder, method='gfpgan', device='cpu', fps=30, progress_callback=None):
    ...
    if method == 'gfpgan':
        from .gfpganer import GFPGANer
        restorer = GFPGANer(
            model_path=model_path,
            root_dir=local_model_path,
            upscale=1,
            arch="clean",
            channel_multiplier=2,
            bg_upsampler=None,
            device=device
        )
    elif method == 'realesrgan':
        from .realesrgan import RealESRGANer
        ...
        restorer = RealESRGANer(scale=4, model_path=models_path, ...)

So at the top level the choice is per-run: you either run the face path (gfpgan) or the general path (realesrgan / animesgan) on the whole media. The interesting routing, the part where one frame gets carved into "face" and "not face," lives one level down, inside the GFPGAN path.

Notice bg_upsampler=None in the GFPGAN branch. That flag is the seam between the two models, and I will come back to it. Notice also upscale=1. The face path here is restoration, not enlargement: it cleans the face up at the same size.

How a frame gets enhanced

The loop is the same for image or video. For each frame, GFPGAN gets the whole frame and three flags:

_, _, output = restorer.enhance(frame, has_aligned=False, only_center_face=False, paste_back=True)

Read those flags in plain English:

has_aligned=False: the input is a normal photo, not a pre-cropped face. So GFPGAN has to go find the faces itself.
only_center_face=False: restore every face it finds, not just the biggest one in the middle.
paste_back=True: after restoring the faces, put them back into the original frame, do not just hand back loose crops.

Those three flags are the routing decision. Everything else is GFPGANer.enhance carrying it out.

Step 1. Find the faces

The first thing enhance does, when the input is not already an aligned crop, is detect faces and align them:

self.face_helper.read_image(img)
self.face_helper.get_face_landmarks_5(only_center_face=only_center_face, eye_dist_threshold=5)
self.face_helper.align_warp_face()

face_helper is a FaceRestoreHelper, set up in the constructor with a detector:

self.face_helper = FaceRestoreHelper(
    upscale,
    face_size=512,
    crop_ratio=(1, 1),
    det_model='retinaface_resnet50',
    save_ext='png',
    use_parse=True,
    device=self.device,
    model_rootpath=root_dir)

This is where "what is a face" gets decided. retinaface_resnet50 is RetinaFace, a face detector (Deng et al., 2019). It returns face boxes and five landmarks per face: two eyes, nose, two mouth corners. Those five points are enough to align each face into a canonical pose. The eye_dist_threshold=5 throws away detections where the two eyes are fewer than five pixels apart, which filters out tiny false positives that are too small to restore usefully.

align_warp_face then warps each detected face into a fixed 512x512 frame using those landmarks, so every face GFPGAN sees is upright and the same size, regardless of how it sat in the original photo. That alignment is not a nicety. It is a requirement, and it is the reason this model cannot touch the background, which I will get to.

After this step the frame has been split. The faces are now a list of 512x512 aligned crops in face_helper.cropped_faces. The background is still sitting in the original image, untouched.

Step 2. Restore each face crop

Now GFPGAN runs, but only on the crops:

for cropped_face in self.face_helper.cropped_faces:
    cropped_face_t = img2tensor(cropped_face / 255., bgr2rgb=True, float32=True)
    normalize(cropped_face_t, (0.5, 0.5, 0.5), (0.5, 0.5, 0.5), inplace=True)
    cropped_face_t = cropped_face_t.unsqueeze(0).to(self.device)

    output = self.gfpgan(cropped_face_t, return_rgb=False, weight=weight)[0]
    restored_face = tensor2img(output.squeeze(0), rgb2bgr=True, min_max=(-1, 1))
    restored_face = restored_face.astype('uint8')
    self.face_helper.add_restored_face(restored_face)

One forward pass per face. The model output is a clean 512x512 face. GFPGAN is built on a generative facial prior, a face-generating network (StyleGAN2) baked into the architecture, so it does not just sharpen what is there. It can rebuild a believable eye or a believable mouth from a blurry one, because it knows what faces look like (Wang et al., CVPR 2021). That is exactly why you want it on faces and nowhere else.

The model call is wrapped in a try/except RuntimeError. If a single face crop fails inference, the code keeps the original crop instead of crashing the whole frame:

except RuntimeError as error:
    print(f'\tFailed inference for GFPGAN: {error}.')
    restored_face = cropped_face

Small thing, but it means one bad face does not kill a 2000-frame video.

Step 3. Paste the faces back, leave the background alone

This is the recombination, and it is where bg_upsampler finally matters:

if not has_aligned and paste_back:
    if self.bg_upsampler is not None:
        # the background goes to Real-ESRGAN
        bg_img = self.bg_upsampler.enhance(img, outscale=self.upscale)[0]
    else:
        bg_img = None

    self.face_helper.get_inverse_affine(None)
    restored_img = self.face_helper.paste_faces_to_input_image(upsample_img=bg_img)

Read it as two decisions.

First, the background. If you handed the restorer a bg_upsampler (a Real-ESRGAN instance), the whole frame goes through that general upscaler to produce bg_img. If you passed None, as content_enhancer does, the background is just the original frame. This is the literal "the rest goes to Real-ESRGAN" wire. It is one if. The face restorer never touches the background itself; it either delegates the background to a different model or leaves it as is.

Second, the faces. get_inverse_affine computes the reverse of each face's alignment warp. Remember, every face was warped into an upright 512x512 box. To put a restored face back, you have to undo that exact warp so the face lands where it came from, at the right size and angle. paste_faces_to_input_image then composites each restored face onto bg_img along that inverse transform. Because the helper was built with use_parse=True, it uses a face-parsing mask at the paste step, so the restored crop blends along the actual face outline (hair, jaw, neck) instead of pasting a hard 512x512 square with visible edges.

The output is one frame: GFPGAN's faces sitting inside Real-ESRGAN's background (or the plain background), seamed together along a parsing mask.

Why one model genuinely cannot do both

This is the whole reason the split exists, so it is worth being concrete.

A face restorer like GFPGAN only works on aligned face crops. Its prior is a face generator. Feed it a brick wall and it has no concept to draw from; it was never trained on walls, foliage, text, or fabric. At best it leaves them alone, at worst it hallucinates face-like structure into textures that are not faces. That is why the code crops faces out and feeds GFPGAN those crops only. The 512x512 alignment is not optional dressing, it is the input contract the model was trained against.

A general upscaler like Real-ESRGAN is the opposite. It was trained on a broad mix of real-world image degradation, so it handles walls, hair, fabric, and grass well (Wang et al., ICCV 2021 Workshops). But it has no face prior. On skin it tends to over-smooth, because its job is to remove noise and compression, and a face under that treatment loses the fine texture that makes it read as a real face. Plastic skin.

So you route. Faces to the model with the face prior, everything else to the model with the general prior, and stitch. Each model only sees the kind of pixels it was trained on.

Gotchas I would warn you about

A few things that bite when you wire this up:

bg_upsampler=None means the background is not upscaled at all. In content_enhancer the GFPGAN path passes None, so faces get restored but the background stays at the original resolution. If you want sharper background too, pass a Real-ESRGAN instance as bg_upsampler. With None, faces improve and the rest is unchanged.
only_center_face=False restores every face. On a crowd shot that is many forward passes, one per detected face, so cost scales with the number of faces, not the frame size. If you only care about the subject, only_center_face=True restores just the largest central face and skips the rest.
No face, no change. If RetinaFace finds nothing, there are no crops, the loop does nothing, and paste_faces_to_input_image returns the background as is. The GFPGAN path is a no-op on a landscape with no people. If a face is there but not getting fixed, suspect the detector or the eye_dist_threshold=5 filter before the restorer.
Alignment quality is upstream of restoration quality. Everything rests on those five landmarks being right. A bad warp gives GFPGAN a distorted face and the restored result, pasted back through the inverse warp, can look subtly off. If output faces look wrong in shape, look at detection, not the GAN.
upscale=1 here is restoration, not enlargement. The constructor is called with upscale=1, so this path cleans faces at the same size. Do not expect a bigger image out of the GFPGAN method; that is the Real-ESRGAN method's job.

Takeaway

The face-versus-background split is three moves. Detect and align every face into a 512x512 crop with RetinaFace. Run GFPGAN on those crops only, because its face prior would hallucinate on anything that is not a face. Paste the restored faces back through the inverse alignment warp, onto a background that is either left alone or handed to Real-ESRGAN, blended along a face-parsing mask so there is no square seam.

The reason it is two models and not one is plain: a face restorer makes skin look real but cannot draw a wall, and a general upscaler makes a wall look sharp but turns skin to plastic. Give each model only the pixels it was trained on.

The code is content_enhancer in visual_processing/enhancement/face_enhancer.py and GFPGANer.enhance in visual_processing/enhancement/gfpganer.py in the Wunjo Make repo. The one line that wires the two models together is bg_upsampler in the paste-back step.

References

Wang, Li, Zhang, Shan. "Towards Real-World Blind Face Restoration with Generative Facial Prior." CVPR 2021. arXiv:2101.04061 (GFPGAN)
Wang, Xie, Dong, Shan. "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data." ICCV 2021 Workshops. arXiv:2107.10833
Deng, Guo, Zhou, Yu, Kotsia, Zafeiriou. "RetinaFace: Single-stage Dense Face Localisation in the Wild." 2019. arXiv:1905.00641

Keep the same face across frames with a tiny embedding memory

Wlad Radchenko — Wed, 26 Aug 2026 11:47:00 +0000

You click a face in frame 1. Two people are on screen. By frame 40 the tool has quietly swapped to the other person, because for a few frames they leaned closer to the camera and got bigger. Anyone who has built face tracking on real video has hit this. The naive fix everyone tries first is "pick the biggest face" or "pick the most centered face", and it fails the moment a second face exists.

This post walks through the part of Wunjo Make that fixes it: a small per-identity memory of face embeddings that says "this is the person you picked, keep following them" even when someone else is bigger or more centered. The code lives in portable/src/visual_processing/face_detection/recognition.py, in the FaceRecognition class.

The everyday version first

Think about recognizing a friend in a crowd. You do not pick whoever is closest or tallest. You remember their face and scan for a match. If someone walks between you, you do not suddenly decide the stranger is your friend.

That is the whole idea here. We keep a running memory of what the chosen face looks like, as numbers, and each new frame we ask "which of these detections looks most like the one I remember?" before we worry about where it is on screen.

The numbers come from a face recognition network. InsightFace's buffalo_l pack runs an ArcFace model that turns a face crop into a 512-dimension vector, an embedding (Deng et al., CVPR 2019; arXiv:1801.07698). Two crops of the same person land near each other in that 512-D space. Two different people land far apart. The detector that finds the boxes in the first place is RetinaFace (Deng et al., 2019; arXiv:1905.00641). We do not need the math of either to use them. We need one operation: distance between two embeddings.

The memory

Here is the state the class holds:

class FaceRecognition(FaceAnalysis):
    def __init__(self, name='buffalo_l', root='~/.insightface', allowed_modules=None, **kwargs):
        super().__init__(name=name, root=root, allowed_modules=allowed_modules, **kwargs)
        self.id = 0
        self.window = {}           # per-id list of past detections
        self.window_size = 50      # warm-up length / last n distances
        self.max_window_size = 250 # cap on stored embeddings in RAM
        self.running_average = {}
        self.max_rate_of_change = 0.05

window is a dict keyed by a face id, so you can track more than one target at once, each with its own memory. Every entry stored for an id is small:

def update_memory(self, face):
    self.set_face_id()
    center = ((face.bbox[0] + face.bbox[2]) / 2, (face.bbox[1] + face.bbox[3]) / 2)
    self.window[self.id].append({
        'embedding': face.normed_embedding,
        'center': center,
        'gender': face.gender,
    })
    if len(self.window[self.id]) > self.max_window_size:
        self.window[self.id].pop(0)

Note normed_embedding, not the raw embedding. ArcFace embeddings are L2-normalized, so they sit on a unit sphere. That matters for the distance step, coming up. The list is a sliding window: once it passes 250 entries it drops the oldest. So the memory is recent. If a person's lighting drifts over a long shot, the memory drifts with them instead of clinging to frame 1 forever.

Distance is just L2

The "looks like" test is one line:

@staticmethod
def calculate_distance(embedding1, embedding2):
    return np.linalg.norm(embedding1 - embedding2)

Straight L2 (Euclidean) distance. Because both vectors are already normalized, this distance is bounded and behaves consistently across face sizes and lighting. Small distance, same person. Larger distance, probably someone else.

For each detected face in the current frame, we do not compare against one remembered embedding. We compare against every embedding in the window and take the mean:

face.mean_distance = np.mean([
    self.calculate_distance(face.normed_embedding, known_face["embedding"])
    for known_face in self.window[self.id]
]) if len(self.window[self.id]) > 0 else float('inf')

Averaging over the window is the quiet trick. A single bad frame, motion blur, a half-turn of the head, gives a noisy embedding. If you compared against only the last frame, one ugly frame could throw the match off. Averaging across the recent window smooths that out. One bad embedding in 50 barely moves the mean.

The sort that ignores "biggest"

This is the heart of it. Selection happens in sort_by_direction, in the distance-from-embedding branch:

if direction == 'distance-from-embedding':
    return sorted(
        filter(lambda face: face.mean_distance < threshold * face.dynamic_threshold, faces),
        key=lambda face: (((face['bbox'][2]+face['bbox'][0])/2 - face_center[0])**2
                        + ((face['bbox'][3]+face['bbox'][1])/2 - face_center[1])**2)**0.5
    )

Read it in two moves. First the filter: throw away any face whose mean embedding distance is too high. That is the identity gate. A stranger who happens to be huge and dead-center never makes it past this line, because their embedding does not match the memory. Then the sorted: among the faces that did match, pick by plain pixel distance from where the person was last seen. Spatial position is the tie-breaker, not the decision.

That ordering is the fix. "Biggest face" and "most centered face" use geometry as the decision. Here geometry only breaks ties between faces that already passed the identity check.

The threshold is not a constant

threshold * face.dynamic_threshold is the gate. Two pieces. dynamic_threshold depends on how big the face is in the frame:

@staticmethod
def get_dynamic_threshold(face, height):
    face_height = (face.bbox[3] - face.bbox[1]) / height
    if face_height < 0.1:
        return 1.05
    if face_height < 0.2:
        return 1.2
    if face_height < 0.4:
        return 1.3
    elif face_height < 0.8:
        return 1.35
    elif face_height < 0.9:
        return 1.4
    else:
        return 1.45

Why scale by size? A tiny, far-away face gives a noisier, less reliable embedding, so we keep the gate tight (1.05) and demand a close match. A large, clear face is trustworthy, so we loosen the gate (up to 1.45) and forgive more variation, because a big face turning its head can legitimately drift further in embedding space while still being the same person. The constant threshold is the per-id running_average, which I will get to.

Multiple faces in one frame, the exact situation where "pick the biggest" jumps to the wrong person. Photo: Unsplash

The running average and why it is clamped

The memory has no idea what a "matching" distance looks like on your specific video. A studio close-up and a shaky handheld clip have different baseline distances for the same person. So the gate self-calibrates. Once the window has warmed up past window_size - 1 entries, it seeds a per-id running_average:

if len(self.window[self.id]) > self.window_size - 1:
    if self.running_average[self.id] is None:
        self.running_average[self.id] = max([
            self.calculate_distance(ret[0].normed_embedding, known_face["embedding"])
            for known_face in self.window[self.id]
        ])
    else:
        change = max([
            self.calculate_distance(ret[0].normed_embedding, known_face["embedding"])
            for known_face in self.window[self.id]
        ]) - self.running_average[self.id]
        change = np.clip(change, -self.max_rate_of_change, self.max_rate_of_change)
        self.running_average[self.id] += change

The seed is the worst (max) match distance inside the current window, a rough "this is how far the same person can drift." After that, each frame nudges the average toward the new worst-match distance, but the nudge is clamped to ±0.05.

That clamp is the safety rail. Imagine the memory got fooled for one frame and locked onto the wrong face. Without the clamp, that wrong face would yank the running average toward its own distances in a single step, and the gate would re-tune itself around the impostor. With the clamp, the gate can only move 0.05 per frame. One bad frame cannot rewrite the baseline. Recovery is possible because the correct face keeps matching at low distance while the average can only crawl.

Warm-up: position first, identity later

There is a chicken-and-egg problem. On frame 1 the memory is empty, so embedding distance is meaningless. The code handles this by falling back to pure position during warm-up:

ret = sort_by_direction(
    ret,
    'distance-from-retarget-face'
        if direction == 'distance-from-embedding' and len(self.window[self.id]) < self.window_size
        else direction,
    face_center, face_gender,
    self.running_average[self.id] if self.running_average[self.id] else "inf"
)

Until the window holds window_size (50) entries, distance-from-embedding silently degrades to distance-from-retarget-face, which is just Euclidean distance from the click point with no identity gate. And on the very first frame, the seed is the face whose box actually contains the click point:

if len(self.window[self.id]) == 0 and face_center[0] and face_center[1]:
    for face in ret:
        x1, y1, x2, y2 = face.bbox
        if x1 <= face_center[0] <= x2 and y1 <= face_center[1] <= y2:
            ret = [face]
            break
    else:
        ret = []

So the flow is: you click a person, the box under your click seeds the memory, the next ~50 frames are tracked by position while the embedding window fills, and from then on identity drives selection and position is only the tie-breaker.

Gotchas you will actually hit

A few things that cost real debugging time:

Feed RGB, consistently. The class is built on InsightFace; mixing BGR and RGB between detection and the embedding model gives embeddings that drift for no visible reason. Stay in one color space end to end.
Call clear() between clips. window and running_average persist on the instance. Reuse the object across two unrelated videos and the second video starts with the first video's memory. clear() resets both.
Set self.id per target. All the state is keyed by self.id. Track two people by passing id= per call so each gets its own window; forget it and both collapse into id 0.
Gender filtering is off in this version. There is a commented-out variant of the filter that also matched on face.gender. It is disabled because of a gender-detection bug, so do not count on gender as part of the gate today. The active filter is distance-only.
The first 50 frames are not identity-aware. During warm-up the tracker can still be lured by a closer face, because it is running on position alone. If your subject is not the nearest face to the click in those early frames, seed more carefully.

Wrap-up

The mechanism is small. A sliding window of normalized ArcFace embeddings per identity, a mean L2 distance as the match score, a size-aware gate that self-calibrates with a clamped running average, and position used only to break ties among faces that already passed the identity check. That combination is why the tracker stays on the person you picked instead of hopping to whoever is biggest.

Read the full file here: recognition.py in Wunjo Make. It is open source, so you can paste the class into a notebook, feed it frames, and watch the window fill.

References

Deng, Guo, Xue, Zafeiriou. "ArcFace: Additive Angular Margin Loss for Deep Face Recognition." CVPR 2019. arXiv:1801.07698. https://arxiv.org/abs/1801.07698 (InsightFace / buffalo_l recognition.)
Deng, Guo, Zhou, Yu, Kotsia, Zafeiriou. "RetinaFace: Single-stage Dense Face Localisation in the Wild." 2019. arXiv:1905.00641. https://arxiv.org/abs/1905.00641

Upscaling a huge image on a small GPU: tiled super-resolution without seams

Wlad Radchenko — Tue, 25 Aug 2026 13:46:00 +0000

You point Real-ESRGAN at a 6000x4000 photo, ask for 4x, and CUDA dies with out of memory. The model is fine. Your GPU just cannot hold a 24000x16000 tensor plus the activations to produce it. An 8GB card does not stand a chance, and even a 24GB card falls over once the input gets big enough.

The fix is to never feed the whole image to the model at once. Cut it into tiles, upscale each tile on its own, then glue the results back together. The trick is gluing them without leaving visible lines where the tiles meet. This is a walkthrough of the exact tiling code in Wunjo Make's RealESRGANer, which is the standard Real-ESRGAN inference helper (Wang et al., ICCV 2021 Workshops).

Tiling a picture is the same idea as tiling a wall, except the hard part is hiding the grout. Photo: Unsplash.

The two knobs that matter

Two numbers control everything here:

self.tile_size = tile      # e.g. 256: each tile is 256x256 input pixels. 0 = no tiling
self.tile_pad  = tile_pad  # default 10: extra pixels read around each tile

tile_size is how big each chunk is in input pixels. Smaller tiles fit on smaller cards but mean more chunks and more overhead. tile_pad is the overlap, the halo of extra pixels read around each tile so the model has real neighbors to look at near the tile edge. Default is 10. Keep that number in mind, it is the whole reason seams do not show.

If tile_size is 0, the model runs on the full image and you are back to the OOM problem. Everything below assumes tile_size > 0.

Step 1. Lay out the grid

Given the padded input image, work out how many tiles fit across and down:

batch, channel, height, width = self.img.shape
output_height = height * self.scale
output_width  = width  * self.scale

# start with a black canvas at the final, upscaled size
self.output = self.img.new_zeros(output_shape)

tiles_x = math.ceil(width  / self.tile_size)
tiles_y = math.ceil(height / self.tile_size)

Two things to notice. The output is allocated once, at full upscaled resolution, as a black image. That is the only big tensor that lives on the device for the whole run, and it is just storage, not activations, so it is cheap compared to running the model on the full frame. Each tile then writes its piece into this canvas.

The ceil means the last column and last row can be partial. A 600-pixel-wide image with tile_size=256 gives ceil(600/256) = 3 columns: 256, 256, and a runt of 88. The code handles that with a min clamp, which is the next part.

Step 2. Cut one tile, with its halo

For each tile in the grid, the code defines two rectangles. The core is the part this tile is responsible for. The padded rectangle is what actually gets read and fed to the model.

ofs_x = x * self.tile_size
ofs_y = y * self.tile_size

# core area: this tile's own region, clamped at the image edge
input_start_x = ofs_x
input_end_x   = min(ofs_x + self.tile_size, width)
input_start_y = ofs_y
input_end_y   = min(ofs_y + self.tile_size, height)

# padded area: core plus a tile_pad halo, clamped at the image edge
input_start_x_pad = max(input_start_x - self.tile_pad, 0)
input_end_x_pad   = min(input_end_x   + self.tile_pad, width)
input_start_y_pad = max(input_start_y - self.tile_pad, 0)
input_end_y_pad   = min(input_end_y   + self.tile_pad, width)  # note: clamps to width

The max(..., 0) and min(..., width) are what keep the halo inside the image. A tile in the middle gets a full 10-pixel halo on all four sides. A tile against the left edge gets no halo on the left, because there is nothing there to read. The halo is real image data borrowed from the neighboring tile, not invented pixels.

That distinction is the entire point. A super-resolution network looks at a neighborhood around each pixel to decide what to draw. A pixel sitting one row inside a tile boundary needs to see the rows on the other side of that boundary. If you fed it a bare tile, that pixel would see the tile's edge and the model would hallucinate a border there. By reading 10 extra pixels of the actual neighbor, the model gets honest context right up to the boundary.

Then the padded tile, and only the padded tile, goes through the model:

input_tile = self.img[:, :, input_start_y_pad:input_end_y_pad,
                            input_start_x_pad:input_end_x_pad]
with torch.no_grad():
    output_tile = self.model(input_tile)

This is the small tensor. A 256x256 tile with a 10-pixel halo is 276x276, and at 4x its output is 1104x1104. That fits on a modest GPU even though the full output would not.

Step 3. Stitch the core back, throw the halo away

Now the careful part. The model upscaled the padded tile, halo included. But the halo pixels were also upscaled inside the neighboring tile, where they were the core. If you wrote the halo too, neighboring tiles would overwrite each other's edges and you would get exactly the seams you were trying to avoid.

So the code computes where the core sits inside the upscaled padded tile, and copies only that:

# where this tile's core lands in the final output (input coords times scale)
output_start_x = input_start_x * self.scale
output_end_x   = input_end_x   * self.scale
output_start_y = input_start_y * self.scale
output_end_y   = input_end_y   * self.scale

# where the core sits inside the upscaled padded tile
output_start_x_tile = (input_start_x - input_start_x_pad) * self.scale
output_end_x_tile   = output_start_x_tile + input_tile_width  * self.scale
output_start_y_tile = (input_start_y - input_start_y_pad) * self.scale
output_end_y_tile   = output_start_y_tile + input_tile_height * self.scale

# write only the core into the canvas
self.output[:, :, output_start_y:output_end_y, output_start_x:output_end_x] = \
    output_tile[:, :, output_start_y_tile:output_end_y_tile,
                      output_start_x_tile:output_end_x_tile]

Read (input_start_x - input_start_x_pad) as "how many halo pixels are on the left of this tile." For an interior tile that is 10, so the slice skips the first 10*scale columns of the upscaled tile. For a left-edge tile it is 0, so nothing is skipped, which is correct because there was no left halo. Everything is multiplied by scale to move from input coordinates to output coordinates.

The result: each output pixel is written exactly once, by the tile that owns it, and every owned pixel was upscaled with real context around it. The tiles meet edge to edge with no overlap in the final image, but because each side of the boundary was computed with knowledge of the other side, the two halves line up.

A note on what "overlap" means here

This is worth being precise about, because the word overlap gets used two ways.

In this code, tiles overlap on the way in (the halo) but not on the way out. The padded read regions of two neighbors share tile_pad columns of input. The written core regions do not share anything. There is no alpha blend or feathering across the boundary. The seam is prevented by giving each side correct context, not by averaging the two sides together.

That is different from a feathered-overlap approach, where tiles overlap in the output and you cross-fade them. Both work. The halo-and-crop method here is simpler and avoids any softening from the blend, at the cost of needing the model to be consistent across the boundary, which a convolutional upscaler with enough halo generally is. With tile_pad=10, ten pixels of context is plenty for Real-ESRGAN's receptive field on its own scale.

Step 4. The whole-image padding around the edges

There is a second, smaller padding layer that is easy to miss. Before any tiling, pre_process pads the entire image:

if self.pre_pad != 0:
    self.img = F.pad(self.img, (0, self.pre_pad, 0, self.pre_pad), 'reflect')

This pre_pad (default 10) handles the outer border of the whole image, where there is no neighbor to borrow from. It mirrors the edge pixels outward with reflect so the model does not see a hard frame at the very edge of the picture. There is also a mod_scale pad that rounds the dimensions up to a multiple the network needs (mod 2 when scale==2, mod 4 when scale==1).

Both pads are cropped back off at the end, scaled up by the upscale factor:

# remove the mod pad, then the pre pad, both at output scale
self.output = self.output[:, :, 0:h - self.mod_pad_h * self.scale,
                                0:w - self.mod_pad_w * self.scale]
self.output = self.output[:, :, 0:h - self.pre_pad   * self.scale,
                                0:w - self.pre_pad   * self.scale]

So you get back exactly input_size * scale, no leftover border.

Gotchas I would warn you about

A few things that bite people:

Pick tile_size to fit VRAM, not to be tidy. Memory scales with tile area, not perimeter. Going from 512 to 256 roughly quarters the per-tile activation memory. If you OOM, halve the tile size before touching anything else.
tile_pad is in input pixels, halo cost is per tile. A bigger halo means safer seams but more redundant compute, since every halo pixel is upscaled twice (once as halo, once as someone's core). The default 10 is a good balance, do not crank it to 100.
The OOM is swallowed. The model call is wrapped in try/except RuntimeError that just prints the error. If a single tile is still too big, you get a black square in the output instead of a crash, which can be confusing. If you see a black tile, your tile is still too large for the card.
Smaller tiles are slower. More tiles means more kernel launches and more halo overhead. There is a real time-versus-memory trade here. On a roomy GPU, use the biggest tile that fits.
Edge tiles are smaller. The last row and column are partial, and the halo is asymmetric there. The code already handles this with the min/max clamps, but if you ever rewrite this loop, that is the case that breaks first.

Takeaway

Tiled super-resolution is three moves. Allocate the full output once as a black canvas. For each tile, read a core region plus a tile_pad halo of real neighbor pixels, upscale that padded tile, then write back only the core and discard the halo. The halo gives the model honest context at every boundary, the crop makes sure each output pixel is owned by exactly one tile, and the two together mean the tiles line up without a blend.

That is how an 8GB card upscales a photo whose full output would need ten times that. The model never sees more than one tile at a time.

The full method is tile_process in visual_processing/enhancement/realesrgan.py in the Wunjo Make repo. If you are hitting OOM on large inputs, start by setting a tile_size and leave tile_pad at 10.

References

Wang, Xie, Dong, Shan. "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data." ICCV 2021 Workshops. arXiv:2107.10833

A referee for lip-sync: scoring whether audio and video match with two embedding towers

Wlad Radchenko — Wed, 19 Aug 2026 12:48:00 +0000

You have a talking-head video. The audio and the lips are close, but not exact. Maybe the sound runs three frames ahead. Maybe two people are on screen and you need to know which mouth the voice belongs to. How do you measure that automatically, without a human watching every clip?

That is the problem SyncNet solves, and the open-source Wunjo Make repo ships two versions of it. I want to walk through the actual code: how the network is built, how audio turns into the thing the network eats, and how the codebase turns the network's output into a usable offset and a confidence number.

Plain version first. You build two small networks. One looks at the lips. One listens to the sound. Each one squeezes its input down to a short vector. If the lips and the sound belong together, the two vectors land close. If they do not, the vectors land far apart. Distance is the whole game.

Two towers, never mixed

Here is the first version, SyncNet_color in portable/src/visual_processing/lip_sync/syncnet.py. It is the Wav2Lip-style net. The shape of the very first layer tells you almost everything:

self.face_encoder = nn.Sequential(
    Conv2d(15, 32, kernel_size=(7, 7), stride=1, padding=3),
    ...
)

self.audio_encoder = nn.Sequential(
    Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
    ...
)

The face tower starts at 15 input channels. The audio tower starts at 1. Those two numbers are the design.

Why 15? Because a single still frame cannot tell you anything about motion, and lip-sync is motion. So the face tower does not take one frame. It takes five frames, stacked. Five RGB frames is 5 times 3, which is 15 channels. The network sees a short burst of mouth movement in one shot, not a single freeze-frame.

The audio tower takes 1 channel because a mel spectrogram is a single grayscale image: time on one axis, frequency on the other, loudness as brightness. One channel is all it needs.

The two towers never share weights and never touch inside the network. They run in parallel, on different inputs, and only meet at the very end as two vectors. That is the contrastive idea: learn a face space and a sound space that line up only for matching pairs.

The trick at the bottom: normalize, then compare

The forward pass is short, and the last three lines are the part that matters:

def forward(self, audio_sequences, face_sequences):
    face_embedding = self.face_encoder(face_sequences)
    audio_embedding = self.audio_encoder(audio_sequences)

    audio_embedding = audio_embedding.view(audio_embedding.size(0), -1)
    face_embedding = face_embedding.view(face_embedding.size(0), -1)

    audio_embedding = F.normalize(audio_embedding, p=2, dim=1)
    face_embedding = F.normalize(face_embedding, p=2, dim=1)

    return audio_embedding, face_embedding

Both embeddings get L2-normalized. That means every vector is pushed onto a unit sphere: only its direction survives, its length is thrown away. Once both vectors have length one, Euclidean distance and cosine similarity carry the same information. A small distance means the directions agree, which means the lips and the sound match.

This is the part people skip when they reimplement these nets and then wonder why training is unstable. If you compare raw embeddings, a loud frame or a bright frame can pump up the vector length and swamp the signal you actually care about. Normalizing first removes magnitude from the equation. You are left comparing shape, not volume.

The forward pass returns the two vectors. It does not return a score. The caller decides what distance means.

Lining up sound and picture is a matching problem, not a generation problem. Photo: Unsplash

Feeding the audio tower: wav to mel

The audio tower wants a mel window, and portable/src/visual_processing/lip_sync/wav2mel.py builds it. The interesting part is the chunking, where audio time gets cut to match video time:

def chunk_mel(self, mel, fps, mel_step_size):
    mel_chunks = []
    mel_idx_multiplier = 80./fps
    i = 0
    while True:
        start_idx = int(i * mel_idx_multiplier)
        if start_idx + mel_step_size > len(mel[0]):
            mel_chunks.append(mel[:, len(mel[0]) - mel_step_size:])
            break
        mel_chunks.append(mel[:, start_idx : start_idx + mel_step_size])
        i += 1
    return mel_chunks

mel_step_size is 16. So each chunk is a fixed 16-column slice of the spectrogram, which is the window the audio tower is built to swallow.

The number 80./fps is the bridge between two clocks. The mel is computed at 16000 Hz with a 200-sample hop (see hparams.py), which works out to 80 mel columns per second of audio. Video runs at fps frames per second. So 80./fps is "how many mel columns advance per video frame." At 25 fps that is 3.2 columns per frame. The loop steps one video frame at a time and grabs the matching audio window each time, so chunk number i lines up with video frame number i. The last chunk clamps to the end of the array so you never run off the edge.

There is also a small, honest gotcha baked into the code:

def check_for_nan(self, mel):
    if np.isnan(mel.reshape(-1)).sum() > 0:
        raise ValueError('Mel contains nan! Using a TTS voice? '
                         'Add a small epsilon noise to the wav file and try again')

Dead-silent synthetic audio can produce a log of zero, which is a NaN, which poisons the whole spectrogram. The fix in the message is blunt and it works: add a hair of noise so the log has something to bite on.

Where the scoring actually happens

Now the honest part, because the code does not lie about it. SyncNet_color is exported from the module, but at inference time it is not wired up to produce a sync score. The syncnet_wt you see in hparams.py is a training-time loss weight for the lip-sync generator, not a runtime judge. So if you are looking for the place that actually scores audio against video in this repo, it is the other SyncNet.

That second version lives in portable/src/visual_processing/face_detection/sync_net/, and it is the classic Chung and Zisserman SyncNet (ACCV 2016). Same two-tower idea, different build: a 3D-conv face tower and an MFCC audio tower, with separate fully connected heads forward_lip and forward_aud in model.py. This one is alive and doing work.

The work is the offset search, in instance.py:

def calc_pdist(feat1, feat2, vshift=10):
    win_size = vshift * 2 + 1
    feat2p = torch.nn.functional.pad(feat2, (0, 0, vshift, vshift))
    dists = []
    for i in range(0, len(feat1)):
        dists.append(
            torch.nn.functional.pairwise_distance(
                feat1[[i], :].repeat(win_size, 1),
                feat2p[i:i + win_size, :]))
    return dists

Read it as a sliding test. For each video frame's face vector, it does not just compare against the audio vector at the same instant. It compares against a window of audio vectors, from vshift frames early to vshift frames late. That gives you a distance for every possible small shift between the two streams.

Then evaluate collapses that into an answer:

mdist = torch.mean(torch.stack(dists_list, 1), 1)
minval, minidx = torch.min(mdist, 0)

offset = vshift - minidx
conf = torch.median(mdist) - minval

Average the distances across all frames for each shift. The shift with the smallest average distance is the true audio-video offset, because that is where the lips and the sound agree the most. The confidence is the gap between the typical distance and the best distance. A big gap means one shift clearly won, so you can trust it. A tiny gap means every shift looked about the same, so the audio probably is not driving this face at all.

Picking the right face from a crowd

The payoff is in recognition.py, in FaceRecognitionSpeaker. When a video has several faces, the code crops each face track, runs syncnet.evaluate on each one, and keeps the per-frame confidence:

offset, conf, dist = syncnet.evaluate(tmp_dir, video_file=fname)
...
fconf = np.median(mean_dists) - fdist
fconfm = signal.medfilt(fconf, kernel_size=9)
...
if conf > threshold:
    if scene_dict is None or scene_dict.get("conf") is None or scene_dict.get("conf") < conf:
        scenes_dict[offset_idx] = {"x": x, "y": y, "idx": idx, "conf": conf}

For each frame the code keeps the track with the highest sync confidence, as long as it clears threshold (default 3.0). That is active-speaker detection built entirely on distance: the mouth whose movements match the soundtrack best is the one talking. The medfilt with a kernel of 9 smooths the per-frame confidence so a couple of noisy frames cannot flip the decision.

So the same plain idea drives both versions. Two towers, two vectors, compare by distance. The second version just adds the sliding window so distance gives you an offset and a trust score, not only a yes or no.

Things that will trip you up

A few practical notes if you go poking at this code.

The two SyncNets are not interchangeable. SyncNet_color eats 15-channel stacked RGB and a mel window. SyncNetInstance eats a 5-frame 3D-conv volume and an MFCC slice. Different preprocessing, different checkpoints. Do not feed one's input to the other.

The audio clock is fixed at 16000 Hz and the offset search assumes 25 fps video. The evaluate function normalizes input to 25 fps first for exactly this reason. If your fps drifts, your offset frame count drifts with it.

The confidence is relative, not absolute. It is median distance minus minimum distance, so it tells you how clearly one shift beat the rest for this clip. It is not a percentage you can compare across different videos without care.

If you want to read it all yourself, the model is in portable/src/visual_processing/lip_sync/syncnet.py, the audio prep in wav2mel.py, and the live scoring path in face_detection/sync_net/instance.py and face_detection/recognition.py.

References

Chung, Zisserman. "Out of Time: Automated Lip Sync in the Wild." ACCV 2016 Workshops. The original two-tower SyncNet and the offset-by-distance idea. https://www.robots.ox.ac.uk/~vgg/publications/2016/Chung16a/
Prajwal, Mukhopadhyay, Namboodiri, Jawahar. "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild." ACM Multimedia 2020. arXiv:2008.10010. The Wav2Lip line that SyncNet_color belongs to. https://arxiv.org/abs/2008.10010

Keeping the same face across generations: how reference photos enter an OmniGen prompt

Wlad Radchenko — Tue, 18 Aug 2026 11:45:00 +0000

You want to generate a new picture of a specific person. Not a generic face, that exact person, the one in your three reference photos. You write a prompt like "the woman in <|image_1|> standing in a kitchen" and you expect the face to survive. Half the time it does not. You get someone who is sort of in the neighborhood, a cousin maybe.

The interesting part of identity preservation in a model like OmniGen is not the diffusion. It is the few dozen lines that decide where your reference photos go inside the prompt and how the model is told "these pixels are the person, the rest is instructions." That happens in the processor, before a single denoising step runs. This is a walkthrough of that code path in the Wunjo Make repo.

OmniGen (Xiao et al., 2024) is a single transformer that takes text and images mixed together in one sequence, instead of a separate text encoder feeding a separate U-Net. That design is exactly why the conditioning code is worth reading: text tokens and image features live in the same stream, and the splice between them is where identity is kept or lost.

Several reference photos and one prompt all become one flat sequence of tokens. Photo: Unsplash.

The prompt is a string with image holes in it

You reference images by tag. The prompt literally contains <|image_1|>, <|image_2|>, and so on. The first job is to split the text at those tags and tokenize the text between them. From processor.py:

pattern = r"<\|image_\d+\|>"
prompt_chunks = [self.text_tokenizer(chunk).input_ids for chunk in re.split(pattern, text)]

for i in range(1, len(prompt_chunks)):
    if prompt_chunks[i][0] == 1:
        prompt_chunks[i] = prompt_chunks[i][1:]

re.split on the tag pattern gives you the text pieces in order, with the tags removed. Each piece is tokenized on its own. That second loop is a small but real detail: the tokenizer prepends a beginning-of-sequence token (id 1) to every chunk, and you only want one at the very start of the whole prompt. So every chunk after the first gets its leading 1 stripped. Miss this and you sprinkle BOS tokens through the middle of your sequence, which is not what the model was trained on.

Tags carry an order, and the order is enforced

Next the code reads the tags back out and turns them into integer ids:

image_tags = re.findall(pattern, text)
image_ids = [int(s.split("|")[1].split("_")[-1]) for s in image_tags]

unique_image_ids = sorted(list(set(image_ids)))
assert unique_image_ids == list(range(1, len(unique_image_ids)+1)), \
    f"image_ids must start from 1, and must be continuous int, ... cannot be {unique_image_ids}"
assert len(unique_image_ids) == len(input_images), \
    f"total images must be the same as the number of image tags, ..."

input_images = [input_images[x-1] for x in image_ids]

Two assertions do real work here. The ids must be [1, 2, 3, ...] with no gaps, and the number of distinct tags must equal the number of images you passed. If you write <|image_1|> and <|image_3|> but skip 2, it stops. If you pass two photos but only tagged one, it stops. This is the kind of guard you want, because the failure it prevents is silent: the model would happily generate from a misaligned set and you would spend an hour wondering why the face is wrong.

That last line is the one to sit with. input_images = [input_images[x-1] for x in image_ids] reorders your actual image tensors to match the order the tags appear in the text. The position in the sentence, not the position in your list, is what counts. A tag can repeat too, and the same photo gets placed at each spot it is named.

Reserving the right number of slots for each image

Now the sequence gets assembled. Text token ids and image placeholders are concatenated into one flat list:

all_input_ids = []
img_inx = []
idx = 0
for i in range(len(prompt_chunks)):
    all_input_ids.extend(prompt_chunks[i])
    if i != len(prompt_chunks) - 1:
        start_inx = len(all_input_ids)
        size = input_images[i].size(-2) * input_images[i].size(-1) // 16 // 16
        img_inx.append([start_inx, start_inx + size])
        all_input_ids.extend([0] * size)

return {"input_ids": all_input_ids, "pixel_values": input_images, "image_sizes": img_inx}

Walk it. After each text chunk (except the last) there is an image, so the code computes how many tokens that image will occupy and pushes that many placeholder 0s into the sequence. The count is H * W // 16 // 16. An image is encoded by the VAE down by a factor of 8, then split into patches of size 2 by the patch embedder, so each side shrinks by 16 total and one image becomes (H/16) * (W/16) tokens. A 1024 by 1024 photo is 64 * 64 = 4096 tokens. That is a lot of sequence per image, which is also why three references cost real memory.

The 0s are pure reservation. They are placeholders, not the image. What matters is img_inx, the list of [start, end] spans recording exactly where each image sits in the flat sequence. Those spans are the address book the model uses later to drop the real image features into place.

The placeholder spans become image features

The actual splice happens in model.py, inside forward. The input ids go through the text embedding table as usual, then the recorded spans are overwritten with VAE-encoded, patch-embedded image latents:

condition_embeds = self.llm.embed_tokens(input_ids).clone()
input_img_inx = 0
for b_inx in input_image_sizes.keys():
    for start_inx, end_inx in input_image_sizes[b_inx]:
        condition_embeds[b_inx, start_inx:end_inx] = input_latents[input_img_inx]
        input_img_inx += 1

This is the whole trick. The [0] placeholders were embedded to some throwaway vectors; this loop replaces those vectors, span by span, with the features of the actual reference photos. After it runs, condition_embeds is one sequence where some positions are text and some positions are "here is exactly what this person's face looks like, at this patch." Then time embedding and the noisy latent are concatenated on, and the whole thing goes through the Phi-3 transformer backbone (Abdin et al., 2024) that OmniGen builds on. Text and image are processed by the same attention layers, in the same sequence. That shared attention is how a token of the prompt can look directly at a patch of the reference face.

One more thing: image tokens see each other fully

There is a detail in the collator that matters for identity. The base attention mask is causal (each token sees only earlier tokens), but for the image spans that is wrong; a patch in the top-left of a face should be able to see a patch in the bottom-right. So the collator opens up full attention inside each image block:

def adjust_attention_for_input_images(self, attention_mask, image_sizes):
    for b_inx in image_sizes.keys():
        for start_inx, end_inx in image_sizes[b_inx]:
            attention_mask[b_inx][start_inx:end_inx, start_inx:end_inx] = 1
    return attention_mask

Inside a reference image, every patch attends to every other patch. Between images and text, the causal order still holds. So each photo is read as a coherent whole, not as a left-to-right scan, which is what you want when the thing you are trying to preserve is a face.

Gotchas you will actually hit

A few things that bite in practice, all visible in the code above:

Tag your images, do not just pass them. If your instruction does not contain <|image_1|>, the assertion path is skipped and the photos are never spliced in. Passing images to the pipeline is not enough; the tag is the anchor.
Number from 1, no gaps. [1, 2, 3], never [0, 1] or [1, 3]. The assertion is strict on purpose.
Reference resolution is sequence length. Each 1024 image is 4096 tokens before your prompt and the output canvas are even counted. If you are running out of memory with three references, that is why. Smaller reference crops cost fewer tokens.
Order is semantic. Because images are reordered to tag order, "<|image_1|> wearing the hat from <|image_2|>" is not the same prompt as swapping the two tags. Be deliberate about which photo is which number.

Takeaway

Identity preservation across generations is not magic in the sampler. It is bookkeeping in the processor: split the prompt at image tags, tokenize the text between, reserve H*W//16//16 placeholder slots per image, record the span of each, then overwrite those spans with real image features so text and face share one attention stream. The model can only keep the same person if the same person's pixels are sitting in the sequence, in known positions, with full attention inside each photo.

The code is in visual_generation/generation/omnigen/processor.py and model.py in the Wunjo Make repo. If your reference-based results drift, start by printing image_sizes and confirming the spans line up with the images you think you passed.

References

Xiao, Wu, Yang, et al. "OmniGen: Unified Image Generation." 2024. arXiv:2409.11340
Abdin et al. "Phi-3 Technical Report." 2024. arXiv:2404.14219

Picking video keyframes with scene cuts plus a fixed stride

Wlad Radchenko — Wed, 12 Aug 2026 12:48:00 +0000

You have a 90-second clip and you need to pick a handful of frames out of it. Not random frames. Frames that, between them, describe the whole clip well enough that a downstream model can fill in the rest. This shows up everywhere in video tooling: you stylize a few frames and warp the look across the gaps, you run an expensive diffusion pass on keyframes only, you build a thumbnail strip, you sample frames for a vision model. The whole job hangs on which frames you keep.

The lazy answer is "every tenth frame." It is one line and it is wrong in a specific, annoying way. The fancier answer is "frames where the shot cuts." That is better and still wrong, in the opposite direction. The function I want to walk through picks both, and the reason it picks both is the interesting part.

The code is SceneDetect.detecting in portable/src/visual_generation/inference.py in Wunjo Make. It is short. Around 30 lines do the whole thing.

Why "every Nth frame" is dumb

Say you take every tenth frame. The video has a hard cut at frame 47, the camera jumps from a face to a wide landscape. Your stride lands you on frames 40 and 50. Frame 40 is the face, frame 50 is the landscape, and the cut between them is invisible to you. Now whatever you do downstream has to bridge frame 40 to frame 50 as if they were the same continuous shot. They are not. A motion-based fill will try to warp a face into a mountain and produce mush. A thumbnail strip will skip the cut entirely and make two scenes look like one.

A fixed stride has no idea where the content actually changes. It samples the timeline, not the footage. On a slow, single-shot clip it oversamples, ten near-identical frames where two would do. On a fast-cut montage it straddles every cut and never lands cleanly on the start of a new shot. The stride is blind.

Why "scene cuts only" is too sparse

So detect the cuts. PySceneDetect does this well, and the function uses it:

from scenedetect import detect, ContentDetector

scene_list = detect(source_file, ContentDetector(threshold=threshold))

ContentDetector works on content difference between consecutive frames. It converts each frame to HSV, measures how much hue, saturation and value changed frame to frame, and when that change spikes past a threshold it calls a cut. There is no neural network here and no training. It is a hand-tuned heuristic, and that is fine, because shot boundaries are mostly a sudden jump in pixels and HSV captures that jump cheaply. The default threshold here is 50.0. Lower it and the detector gets twitchy, calling cuts on fast pans and lighting changes. Raise it and it misses soft cuts. Fifty is a sane middle for general footage.

Now you have a list of scenes. The problem: a scene can be ten seconds long. If you keep only the first frame of each scene, a ten-second continuous shot of someone walking across a room gives you one keyframe. Everything that happens inside that shot, the person crossing from left to right, is gone. For a thumbnail that might be fine. For anything that has to reconstruct the in-between frames, one keyframe per shot is far too little. The motion inside the shot has no anchors.

Scene cuts tell you where the content changes hard. They say nothing about the slower drift inside a shot.

Combining both

Here is the whole selection block:

if len(scene_list) > 0:
    for i, scene in enumerate(scene_list):
        start_frame = scene[0].get_frames()
        end_frame = scene[1].get_frames()
        if interval:
            scene_frames.extend(range(start_frame, end_frame, interval))
            scene_frames.append(end_frame - 1)
        else:
            scene_frames.extend([start_frame, end_frame - 1])
    else:
        scene_frames.append(total_frames - 1)
else:
    if interval:
        scene_frames = list(range(0, total_frames - 1, interval))
        scene_frames.append(total_frames - 1)
    else:
        scene_frames = [0, total_frames - 1]

Read it from the top. For each detected scene you have a start_frame and an end_frame. The key two lines are these:

scene_frames.extend(range(start_frame, end_frame, interval))
scene_frames.append(end_frame - 1)

The first line walks the scene from its start to its end in steps of interval (default 10), and adds every one of those frames. So inside the shot you get the start, then start+10, then start+20, and so on. That is the fixed stride, but now it is reset at the start of every scene. It cannot straddle a cut, because each scene gets its own fresh stride that begins exactly on the cut. The stride does the sampling inside the shot; scene detection makes sure the stride never crosses a boundary.

The second line appends end_frame - 1, the last frame of the scene. This matters because range(start, end, interval) rarely lands on the final frame. If a scene runs from frame 100 to frame 137 with a stride of 10, range gives you 100, 110, 120, 130, and stops. Frame 136, the last real frame before the cut, never gets picked. But that last frame is the most important one to keep, because it is the frame right before the content changes. It is what a downstream fill needs to hand off cleanly to the next shot. So the code explicitly adds it. end_frame - 1, not end_frame, because the scene's end_frame is exclusive: it is the first frame of the next scene, not the last frame of this one.

Put the two ideas together and you get the property you actually want. Keyframes are dense enough inside each shot to capture motion, and they always sit cleanly on shot boundaries, never across them. Scene detection gives the structure, the stride gives the density, and the explicit last-frame append seals each scene so nothing falls through the gap at the cut.

A keyframe per cut is too coarse; every Nth frame straddles the cuts. Striding inside each detected scene, plus the scene's last frame, gives both. Photo: Unsplash

The fallback when nothing is detected

There is a second branch, and skipping it is how you ship a crash. Some inputs produce no scenes at all. A single static shot, a very short clip, a screen recording with no real cuts. PySceneDetect returns an empty list, and if your code only handles the "scenes found" path, you get zero keyframes and a confusing downstream failure.

The else covers it:

else:
    if interval:
        scene_frames = list(range(0, total_frames - 1, interval))
        scene_frames.append(total_frames - 1)
    else:
        scene_frames = [0, total_frames - 1]

No scenes means the whole video is treated as one big scene. Stride across the entire thing, then append the true last frame. It degrades to plain "every Nth frame" exactly when there are no cuts to respect, which is the one case where plain stride is the right call. The combined approach and the dumb approach converge when the footage has no structure to detect.

Dedup before you read the disk

The stride and the last-frame append can collide. A short scene where end_frame - 1 happens to be a multiple of the stride from the start will have that frame in the list twice. Reading and writing the same frame twice is wasted I/O. One line handles it:

for i in sorted(set(scene_frames)):
    if i <= total_frames - 1:
        cap.set(cv2.CAP_PROP_POS_FRAMES, i)
        ret, frame = cap.read()
        if ret:
            file_path = os.path.join(local_save_dir, f"{i}.jpg")
            cv2.imwrite(file_path, frame)

sorted(set(scene_frames)) does two jobs in one expression. set removes the duplicates so each frame is read once. sorted puts them back in ascending order so the seek moves forward through the file instead of jumping around. Seeking forward is much cheaper than seeking backward with cv2.VideoCapture, so reading frames in order is not just tidy, it is faster.

The i <= total_frames - 1 guard is a seatbelt. total_frames from cv2.CAP_PROP_FRAME_COUNT is not always exact, some containers report a frame or two more than you can actually decode, so the guard keeps a bad count from asking for a frame that does not exist.

Gotchas you will actually hit

The frame numbers in the output are one-based. Look at how each kept frame is recorded: the saved entry uses "frame": i + 1. The file on disk is named by the zero-based index i (f"{i}.jpg"), but the metadata reports i + 1. If you consume this output and assume the frame field matches the .jpg filename, you are off by one. The filename is the seek index, the frame field is the human-facing count.

end_frame is exclusive. The single most common bug when you reimplement this is appending end_frame instead of end_frame - 1 and writing a frame that belongs to the next scene, or running off the end of the file. PySceneDetect's scene end is the boundary, not the last frame.

The threshold is footage-dependent. threshold=50.0 is a default, not a law. Animation, screen recordings, and high-contrast edited content cut differently from handheld video. If you get too many keyframes, the detector is over-firing; raise the threshold. Too few, lower it. The stride is the other dial: a wide interval saves frames but misses fast motion inside a shot, a narrow one captures motion but costs you frames downstream.

interval=0 is a real mode. Passing a falsy interval switches every branch to "endpoints only," start and last frame of each scene, or [0, total_frames - 1] for the no-scene case. That is the right setting when you only want shot boundaries and will handle the in-between frames yourself.

Wrap-up

Keyframe selection looks trivial until you try to use the keyframes for something. A fixed stride samples the clock and ignores the footage, so it straddles every cut. Scene detection finds the cuts but leaves long shots almost unsampled. The fix is to use scene detection for structure and a per-scene stride for density, then explicitly keep the last frame of each scene so the handoff at every cut is clean. Add a fallback that treats a cutless video as one scene, dedup with sorted(set(...)) so you read each frame once and in order, and guard against an over-reported frame count.

Thirty lines, no model weights, and the output is a frame list you can actually build on. The code is SceneDetect.detecting in visual_generation/inference.py in the Wunjo Make repo. If your own keyframe picker chokes on cuts, this is the shape to copy.

References

PySceneDetect, ContentDetector: detects shot changes from frame-to-frame HSV content difference. Library, no associated paper. https://www.scenedetect.com

Auto-highlights with a small VLM or how the model votes yes or no on every 10-second cut

Wlad Radchenko — Tue, 11 Aug 2026 11:42:00 +0000

You point a tool at a 40-minute gameplay video and ask for a one-minute highlight reel. What should it actually do?

The naive answer is "find the exciting parts." But a model has no idea what exciting means for your video until you tell it. A goal in football, a punchline in a podcast, and a jump scare in a let's-play are all highlights, and none of them look alike. So the real problem is two problems stacked on top of each other. First, figure out what counts as a highlight for this specific video. Then go through the video and decide, piece by piece, whether each piece is one of those.

That is exactly the shape of the code in portable/src/visual_generation/highlights/processing.py in Wunjo Make. It uses a small vision-language model called SmolVLM2 and, for the audio path, OpenAI's Whisper. What I like about it is that the "scoring" is not a regression head or a learned ranker. It is the model answering the word yes. Let me walk through how that works and where the sharp edges are.

The whole pipeline in one breath

Here is the visual path, top to bottom, from generate_visual_highlight_moments:

duration = self.get_video_duration_seconds(video_path)
description = self.process_video_analyser(video_path)
highlight_response = self.process_highlight(description, highlight_description)
self.cut_scenes(video_path, save_dir, segment_length)   # 10s chunks via ffmpeg
# ...shuffle segment indices, chunk them...
if self.process_video_segment(segment_path, highlight_response):
    kept_segments.append((start_time, end_time))

Four model calls, in order:

Describe the video. Ask the VLM what kind of video this is.
Invent the highlight menu. Feed that description back and ask for a list of dramatic moments that would make good highlights for this type of video.
Cut into segments. Use ffmpeg to split the file into 10-second clips. No model here.
Vote per segment. For each clip, show it to the VLM with the menu and ask: does this clip contain one of these? Keep the ones where the answer is yes.

The model wrapper, and the one line that parses the answer

class SmolVLM2:
    def __init__(self, model_path: str = "HuggingFaceTB/SmolVLM2-2.2B-Instruct", device: str = "cuda", download_root: str = None):
        self.processor = AutoProcessor.from_pretrained(model_path)
        self.model = AutoModelForImageTextToText.from_pretrained(
            model_path,
            torch_dtype=torch.bfloat16,
            device_map=self.device,
            _attn_implementation="flash_attention_2" if FLASH_ATTN_2_AVAILABLE else None
        )

SmolVLM2 is a small multimodal model, the 2.2B-Instruct checkpoint here. Small matters: this runs per segment in a loop, so a 70B model would be unusable on a desktop. It loads in bfloat16 and turns on FlashAttention 2 only if the import succeeded at the top of the file, otherwise it passes None and lets transformers fall back.

The generate method is where the interface to the rest of the pipeline lives:

def generate(self, messages: list, max_new_tokens: int = 64):
    inputs = self.processor.apply_chat_template(
        messages[:8192],
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(self.model.device, dtype=torch.bfloat16)
    generated_ids = self.model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.7)
    generated_output = self.processor.decode(generated_ids[0], skip_special_tokens=True).lower().split("assistant: ")[1]
    return generated_output

Three things to notice here, because they all affect the highlight decision.

do_sample=True, temperature=0.7. The voting is not deterministic. The same clip with the same prompt can come back yes once and no the next run. For a highlight reel that is fine, even pleasant. You get variety. For a unit test it is a headache.

max_new_tokens=64 for the per-segment call. The model is not asked to write an essay, it is asked for a quick verdict. Sixty-four tokens is enough for "yes" plus a one-line justification and nothing more.

.lower().split("assistant: ")[1]. This is the cheap, fragile glue that turns chat output into a usable string. It lowercases everything (so the later "yes" in response check is case-insensitive) and keeps only the text after the literal assistant: marker. If the decoded output ever does not contain that exact marker, this line raises an IndexError. Worth knowing if you swap the checkpoint.

The model writes its own rubric

This is my favorite part. Before scoring anything, the code asks the model to enumerate what a highlight even is for this video. Look at the system messages:

SYSTEM_MESSAGES = {
    "highlight_editor": "You are a highlight editor. List archetypal dramatic moments that would make compelling highlights if they appear in the video. Each moment should be specific enough to be recognizable but generic enough to potentially exist in other videos of this type.",
    "highlight_assistant": "You are a helpful visual-language assistant ... Highlights should be rare and important events in the video in question.",
    "highlight_analyzer": "You are a video highlight analyzer. Your role is to identify moments that have high dramatic value ... Be categorical and choose only the brightest moments. ..."
}

And the prompt that builds the menu:

@staticmethod
def prompt_highlight(description: str, highlight_description: str = None, behaviour: int = None):
    prompts = {
        1: "List potential highlight moments to look for in this video:",
        2: "List dramatic moments that would make compelling highlights if they appear in the video. ...:"
    }
    behaviour = 1 if highlight_description is None and behaviour is None else 2
    highlight_description = prompts[behaviour] if highlight_description is None else highlight_description
    return [
        {"role": "system", "content": [{"type": "text", "text": SYSTEM_MESSAGES["highlight_editor" if behaviour == 1 else "highlight_assistant"]}]},
        {"role": "user", "content": [{"type": "text", "text": f"""Here is a description of a video:\n\n{description}\n\n{highlight_description}"""}]}
    ]

The behaviour switch decides the personality. If the caller passed nothing, behaviour is 1 and the model plays "highlight editor", listing archetypal moments. If the caller passed their own highlight_description (say, "show me every time someone laughs"), behaviour becomes 2 and the system role switches to "highlight assistant". The wording in those two system prompts is doing real work: "specific enough to be recognizable but generic enough to potentially exist in other videos of this type" is what keeps the menu from being too tied to one exact frame.

There is one more wrinkle in process_highlight:

def process_highlight(self, description: str, highlight_description: str = None) -> str:
    if highlight_description is not None:
        highlight_description = f"List potential {highlight_description.lower()} to look for in this video:"
    highlight_response_messages = self.prompt_highlight(description, highlight_description)
    highlight_response = self.generate(highlight_response_messages, max_new_tokens=256)
    return highlight_response

When you give your own description, it is wrapped into "List potential {your text} to look for in this video:". So a user request like "funny reactions" becomes "List potential funny reactions to look for in this video:". The output of this call, up to 256 tokens, is the rubric. It is plain text. It gets pasted verbatim into every per-segment prompt later.

The yes/no vote, and why scoring is just `"yes" in response`

Here is the prompt that scores one clip:

@staticmethod
def prompt_video_segment(video_segment_path: str, highlight_types: str):
    return [
        {"role": "system", "content": [{"type": "text", "text": SYSTEM_MESSAGES["highlight_analyzer"]}]},
        {"role": "user", "content": [
            {"type": "video", "path": video_segment_path},
            {"type": "text", "text": f"""Given these highlight examples:\n{highlight_types}\n\nDoes this video contain a moment that matches the core action of one of the highlights? Answer with:\n'yes' or 'no'\nIf yes, justify it"""}]}
    ]

And the scoring:

def process_video_segment(self, video_segment_path: str, highlight_types: str) -> bool:
    messages = self.prompt_video_segment(video_segment_path, highlight_types)
    response = self.generate(messages, max_new_tokens=64)
    print(f"Segment response {response}")
    return "yes" in response

That is the entire scorer. No score between 0 and 1, no ranking, no threshold you tune. The clip is shown to the model alongside the rubric, the model is told to answer yes or no and justify a yes, and the code does a substring check for yes on the lowercased text. It is a vote, not a rank.

This is a real design choice with real consequences. The upside: it is dead simple and it leans on the model's own judgment, which is the whole point of using a VLM. The downside is the substring check. The word "yes" appearing anywhere in 64 tokens flips the clip to "keep". A justification like "there is no clear yes-or-no moment here" contains yes and would be kept. In practice the instruction to lead with yes or no makes this rare, but it is the failure mode to watch if you fork this.

Stopping early: shuffle, chunk, budget

You do not want to score all 240 clips of a 40-minute video. You want roughly one minute of highlights and then stop. Here is how the loop manages that:

duration_limit_sec = duration_limit_min * 60
duration_highlight = 0

indices = list(range(len(segments_path)))
random.shuffle(indices)

chunk_size = 10
chunks = [indices[i:i + chunk_size] for i in range(0, len(indices), chunk_size)]

for num, chunk in enumerate(chunks):
    for i in chunk:
        segment_path = segments_path[i]
        start_time = float(i * segment_length)
        end_time = min(float(i * segment_length + segment_length), duration)
        if duration_highlight + segment_length > duration_limit_sec:
            break
        if self.process_video_segment(segment_path, highlight_response):
            kept_segments.append((start_time, end_time))
            duration_highlight += segment_length
    else:
        continue   # limit not reached, process next chunk
    break          # limit reached, stop

Three deliberate moves here.

The indices are shuffled before scoring. If you scanned in order and your budget filled up after the first six clips, your highlight reel would be the first sixty seconds of the video, every time. Shuffling spreads the sampled clips across the whole timeline, so a one-minute reel can pull a moment from the start, the middle, and the end.

The duration_highlight counter is the budget. Every kept clip adds segment_length (ten seconds). The check if duration_highlight + segment_length > duration_limit_sec: break stops adding once the next clip would overflow the requested length. So duration_limit_min=1 gives you up to six ten-second clips.

The for/else/break is the early exit. This is one of Python's least-used features. The else on a for loop runs only if the loop finished without hitting break. So: if a chunk completes without filling the budget, else: continue moves to the next chunk. If the budget filled mid-chunk and we breaked out of the inner loop, the else is skipped, and the outer break ends everything. Chunking by ten just bounds how often the budget is checked.

One last line that matters:

kept_segments.sort(key=lambda x: x[0])

We scored in shuffled order, so the kept clips are out of order. This sorts them back by start time, so the reel plays chronologically. Without it your highlights would jump around in time.

The audio twin: Whisper instead of frames

generate_text_highlight_moments is the same machine with a different sensor. Instead of looking at frames, it reads what was said. The transcription comes from the Whisper class:

class Whisper:
    def __init__(self, download_root: str, model_id: str = "large-v3-turbo", device: str = "cuda"):
        self.model = whisper.load_model(model_id, device=device, download_root=download_root)

    def __call__(self, audio_path, language=None, task="transcribe"):
        audio = self.load_audio(audio_path)
        output = whisper.transcribe(self.model, audio, detect_disfluencies=True, vad=True, language=language, task=task)
        return output

It uses large-v3-turbo with vad=True (voice activity detection, so silence is dropped) and detect_disfluencies=True (it marks ums and hesitations). The transcript gets grouped into roughly 30-second blocks by get_timestamp, each block carrying start, end, and text. Those blocks are the segments now. The scorer is process_text_segment, which asks the same yes/no question about a chunk of transcript text, and the same "yes" in response vote decides it. Same shuffle, same budget, same early exit, same final sort.

There is also a small cleanup helper worth a mention, because small VLMs love to repeat themselves:

@staticmethod
def strip_repeats(text, min_words=2, max_words=6):
    pattern = re.compile(r'\b((?:\w+\s*){' + str(min_words) + ',' + str(max_words) + r'})\1+', flags=re.IGNORECASE)
    return pattern.sub(r'\1', text)

It collapses a phrase of 2 to 6 words that repeats back-to-back into a single copy. The text path runs the description through this before re-summarizing. It is a band-aid for a known habit of small models, and it is honest about being one.

A highlight reel is just the clips that survived the vote, stitched back in time order. Photo: Unsplash

Stitching the survivors

Once you have kept_segments, the cut is plain ffmpeg. concatenate_scenes builds a filter_complex that trims and concatenates each kept range:

for i, (start_sec, end_sec) in enumerate(scene_times):
    filter_complex_parts.append(f"[0:v]trim=start={start_sec}:end={end_sec},setpts=PTS-STARTPTS[v{i}];")
    filter_complex_parts.append(f"[0:a]atrim=start={start_sec}:end={end_sec},asetpts=PTS-STARTPTS[a{i}];")
    concat_inputs.append(f"[v{i}][a{i}]")

concat_filter = f"{''.join(concat_inputs)}concat=n={len(scene_times)}:v=1:a=1[outv][outa]"

setpts=PTS-STARTPTS resets the timestamps on each trimmed piece so they line up at zero before concatenation. Audio gets the same treatment with asetpts. The output is re-encoded to H.264 with AAC audio. There is a separate concatenate_audio for audio-only output. Both honor a DEBUG env var: when DEBUG=True the command runs through os.system so you see ffmpeg's output, otherwise it runs silently through subprocess.run.

Gotchas if you run this yourself

A few things that will bite you, all visible in the code:

Sampling makes it non-reproducible. do_sample=True, temperature=0.7 means two runs on the same video give different reels. Set do_sample=False in generate if you need repeatability while debugging.
The "yes" in response substring is greedy. Any clip whose 64-token answer contains the letters yes is kept. If you change the checkpoint or the prompt, check that the model still leads with a clean yes/no.
The split("assistant: ")[1] will throw on the wrong format. A different chat template that does not emit that exact lowercase marker breaks generate with an IndexError. This is the first thing to check when swapping models.
cut_scenes re-reads the directory with os.listdir. Segments are matched by index against segments_path, and start_time is computed as i * segment_length. That assumes ffmpeg produced evenly sized chunks in sorted order. The last chunk is shorter, which is why end_time is clamped with min(..., duration).
The budget is coarse. It counts in whole segment_length units (10s), not in actual kept seconds, so a one-minute request gives "up to six clips", not "exactly sixty seconds".

That is the whole selection engine. No trained ranker, no score threshold, just a small model that writes its own rubric, then votes yes or no on each ten-second slice while a budget and a shuffle keep the reel short and spread out. The simplicity is the feature. You can read every decision the system makes, because every decision is a word the model said.

References

Marafioti et al. "SmolVLM: Redefining Small and Efficient Multimodal Models." 2025. arXiv:2504.05299. https://arxiv.org/abs/2504.05299
Radford, Kim, Xu, Brockman, McLeavey, Sutskever. "Robust Speech Recognition via Large-Scale Weak Supervision." 2022. arXiv:2212.04356 (Whisper). https://arxiv.org/abs/2212.04356
Source file: portable/src/visual_generation/highlights/processing.py in Wunjo Make.

Growing an image past its border without a visible seam

Wlad Radchenko — Wed, 05 Aug 2026 13:38:00 +0000

You have a photo and you want more of it. More sky above the head, more room to the left of the subject, a wider frame than the camera gave you. Outpainting is the name for inventing those new pixels. The model part is well covered everywhere. The part nobody talks about is the few dozen lines that run before the model, and that part is where the seam is won or lost.

If you get the prep wrong, the new region meets the old photo at a hard line. The colors step. The texture changes mid-wall. You can point at exactly where the real photo ended. This post is a walkthrough of the prep code in Wunjo Make that stops that from happening. It lives in one file, visual_generation/generation/outpaint.py, and it is all NumPy and OpenCV. No network in sight.

Outpainting is asking for canvas that was never photographed. The trick is making the new canvas agree with the old one at the join. Photo: Unsplash.

The shape of the problem

Say you want to extend an image to the left. You need a bigger canvas, the old pixels parked on the right side of it, and an empty strip on the left for the new content. Three things have to be true at the end:

The known pixels survive untouched in the middle and far side.
The model is told, precisely, which strip is allowed to change.
The old edge does not become a wall the model paints up against.

The whole function is built around those three. Here are the defaults it runs with:

def process_image(image, fill_color=(0, 0, 0), mask_offset=50, blur_radius=500,
                  expand_pixels=256, direction="left",
                  inpaint_mask_color=50, max_size=1024):

expand_pixels=256 is how much new canvas you grow. mask_offset=50 is the overlap that hides the seam, and it is the most important number in the file. blur_radius=500 feathers the mask edge. max_size=1024 is the hard cap on the output dimension. Keep those four in mind.

Step 1. Size the new canvas, and crop if it would blow the cap

First it computes the new size by adding expand_pixels on the axis you are growing:

new_height = height + (expand_pixels if direction in ["top", "bottom"] else 0)
new_width  = width  + (expand_pixels if direction in ["left", "right"] else 0)

Then the part people forget. If growing the image would push it past max_size, it does not let the canvas grow without limit. It crops the same number of pixels off the opposite side first, so the output stays at max_size:

if new_width > max_size:
    if direction == "left":
        image = image[:, :max_size]            # drop the far-right columns
    elif direction == "right":
        image = image[:, new_width - max_size:] # drop the far-left columns
    new_width = max_size

This is a practical decision, not a cosmetic one. The model downstream has a size it is happy with. Letting the canvas creep past 1024 on every pass would either run out of memory or force a resize that softens the whole image. So the canvas slides: you gain 256 new pixels on one edge and lose the equivalent off the other, holding the frame at a fixed size.

Step 2. Three buffers

After the size is settled, it builds three arrays the same size as the new canvas:

new_image    = np.full((new_height, new_width, 3), fill_color, dtype=np.uint8)  # the canvas
mask         = np.full_like(new_image, 255, dtype=np.uint8)  # soft guidance mask
inpaint_mask = np.full_like(new_image, 0,   dtype=np.uint8)  # hard fill region

mask         = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
inpaint_mask = cv2.cvtColor(inpaint_mask, cv2.COLOR_BGR2GRAY)

Two different masks, and the difference matters.

mask is the soft one that the generation step uses. It starts all white (255), and white here means "keep this." inpaint_mask is the hard one, a quick scratch buffer used a moment later to pre-fill the empty strip. It starts all black (0).

Step 3. Place the pixels and carve the strips

This is the heart of it. For the left direction:

new_image[:, expand_pixels:] = image[:, : max_size - expand_pixels]
mask[:, : expand_pixels + mask_offset] = inpaint_mask_color   # 50
inpaint_mask[:, :expand_pixels] = 255

Line one drops the real pixels into the canvas, shifted right by expand_pixels so the left strip is left empty for the new content.

Line three marks the hard fill region. Exactly the expand_pixels columns on the left, set to 255. That is the genuinely empty part, the 256 columns that were never photographed.

Line two is the trick. It paints the soft mask to a low value (inpaint_mask_color, which is 50) across expand_pixels + mask_offset columns. Not 256. 306. It reaches 50 pixels past the new region, into the real photo.

That 50-pixel overlap is the whole game. If the editable region stopped exactly at the old border, the model would have to match the real photo perfectly at a single column, and it never will, so you would see a line. By letting the model also repaint a thin band of real pixels right at the join, you give it room to ramp the new content into the old. It blends across the edge instead of butting up against it.

Note the mask value is 50, not 0. The region is not marked "fully replace" but "mostly free," a gray instruction rather than a hard switch. Softer guidance near the seam, which is exactly what you want there.

Step 4. Father the mask edge

A sharp transition in the mask still prints a sharp transition in the result. So the mask gets a big Gaussian blur:

if blur_radius % 2 == 0:
    blur_radius += 1                 # OpenCV needs an odd kernel
mask = cv2.GaussianBlur(mask, (blur_radius, blur_radius), 0)

blur_radius=500 becomes a 501x501 kernel. That is enormous, and on purpose. It smears the boundary between the kept region and the editable region across hundreds of pixels, so there is no single row or column where guidance flips. The model gets a gradient of freedom, most free in the new strip, gently constrained as it moves back into real territory. The odd-number guard is there because OpenCV rejects an even kernel size.

Step 5. Pre-fill the empty strip with Telea

Here is the last and most underrated step. The empty strip is still flat fill_color (black, by default). Handing a model a black bar to start from is a bad init. So before returning, it fills that strip with a cheap classical inpaint:

_, mask_np = cv2.threshold(inpaint_mask, 128, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
inpaint = cv2.inpaint(new_image, mask_np, 3, cv2.INPAINT_TELEA)

cv2.inpaint with INPAINT_TELEA (the Fast Marching Method) walks inward from the border of the hole and paints each new pixel from its neighbors. The 3 is the inpaint radius, how far it looks for those neighbors. It is not smart. It will not invent a tree or a window. What it does is bleed the edge colors of the real photo into the empty strip, so the strip starts as a soft smear of the right colors instead of a black void.

Why bother, if a generator is about to repaint it anyway? Because the generator does better work when it starts from a plausible color field than from flat black. The smear carries the lighting and palette of the old photo into the new region before generation even begins. The black-bar init pulls the result toward dark, muddy borders. The Telea smear pulls it toward the photo it is extending.

The function returns two things: inpaint, the canvas with real pixels plus a softly pre-filled strip, and mask, the feathered soft mask. That pair is the init image and the mask you hand to an inpainting or diffusion model. A conditioning setup like this is the same idea behind ControlNet (Zhang, Rao, Agrawala, ICCV 2023): you do not just ask the model to generate, you hand it a structured starting point and a map of where it is allowed to work.

Doing it on both sides: the centering pass

decode_image is the wrapper that turns a single landscape-or-portrait image into a fixed canvas (default 576x1024). It figures out which axis is short, then calls process_image for both sides of that axis and stitches them:

expand_pixels_half_x = int((max_width - width) / 2)
expand_pixels_half_y = int((max_height - height) / 2)

if expand_pixels_half_x > 0:
    expand_pixels_half = expand_pixels_half_x
    directions = ["left", "right"]
else:
    expand_pixels_half = expand_pixels_half_y
    directions = ["top", "bottom"]

Splitting the needed growth in half and growing both edges keeps the original subject centered, instead of shoving it to one side. merge_images then pastes the two halves back together into one canvas.

The wrapper also returns its own combined binary mask, and it pulls in by padding=20 on the long axis:

mask_combined[pad_y_start:pad_y_end, pad_x_start:pad_x_end] = 0

That 20-pixel inset is the same instinct as the 50-pixel offset upstream: do not trust the boundary exactly, let the editable zone eat slightly into the known region so the join has somewhere to blend.

Gotchas you will actually hit

A few things that bit me, in case you build something similar.

The mask polarity is easy to get backwards. In this code white (255) means keep and the low value (50) means free to change. If your downstream model expects the opposite, you will outpaint the wrong region and destroy the photo. Check the convention before anything else.

The 50-pixel overlap is tunable, but do not drop it to zero. Zero overlap brings the hard seam right back. Too large, and the model starts rewriting parts of the photo you wanted kept. Fifty pixels on a 1024 canvas is a reasonable middle.

The Gaussian kernel must be odd. The code guards for this with the % 2 check, and if you change blur_radius, keep that guard or OpenCV will throw.

The opposite-side crop in step 1 is silent. If you feed in an image that is already near max_size and keep extending, you are quietly losing content off the far edge each pass. That is correct for a fixed-canvas product, but surprising if you expected pure growth.

Takeaway

The model invents the new pixels. Everything that makes them join cleanly happens in classical code before the model runs: pad the canvas, place the known pixels, mark the empty strip, overlap the editable mask 50 pixels into the real photo so it can blend across the old edge, feather that mask with a 501-pixel blur, and pre-fill the empty strip with a Telea smear so the model starts from the right colors instead of black. No generation in any of it.

The full file is visual_generation/generation/outpaint.py in the Wunjo Make repo. If you are fighting a visible seam in your own outpainting, the mask_offset overlap and the Telea pre-fill are the two places I would look first.

References

Zhang, Rao, Agrawala. "Adding Conditional Control to Text-to-Image Diffusion Models." ICCV 2023. arXiv:2302.05543
Telea. "An Image Inpainting Technique Based on the Fast Marching Method." Journal of Graphics Tools, 2004. (The INPAINT_TELEA method in OpenCV.)

The file is called encryption.py. It is actually an invisible watermark!

Wlad Radchenko — Tue, 04 Aug 2026 11:37:00 +0000

There is a file in my app called encryption.py. The class is EncryptionEncoder. The methods on the video manager are encrypted() and decrypted(). Every label says crypto.

It is not crypto. There is no cipher in that file, no key, no AES, no Fernet. What it actually does is hide a short text string inside the pixels of a frame so you cannot see it but a decoder can read it back. That is a watermark. The naming is just unfortunate, and I am going to use the honest word for the rest of this post.

I want to walk through the code, because the mechanism is small and clever, and then be straight about what a thing like this can and cannot do when it ships inside a fully local app that the user owns.

The name on the file says encryption. The code says watermark. Photo: Unsplash.

What problem this solves

The app generates synthetic media. Face swaps, lip-sync, retargeted portraits. At some point you want to answer one question about a file: did this come out of my tool, or is it an untouched original?

The honest way to answer that without a visible logo is to embed a marker the eye does not notice and a program can find. So when the app finishes a video, it writes a hidden string into a corner of each frame. Later, an analysis step reads the corners and checks for that string.

Here is the read side, from inference.py:

is_original = VideoManipulation.decrypted(source_file)
if is_original:
    lprint(msg="Content is original.", ...)
else:
    lprint(msg="Content is Wunjo fake.", ...)

So the whole point is a yes/no flag: is this our synthetic output. The marker string in the code is literally name="fake". That is the payload it stamps and the payload it looks for.

How the marker gets into a frame

The payload is just bytes. EncryptionEncoder turns a UTF-8 string into a flat list of bits:

seq = np.array([n for n in content], dtype=np.uint8)
self._encryptions = list(np.unpackbits(seq))   # e.g. b"fake" -> 32 bits
self._wmLen = len(self._encryptions)

Four ASCII characters give 32 bits. That 32 matters later, the decoder is hard-coded to expect 32-bit payloads.

The default embedding method in the app is dwtDctSvd. Three transforms stacked, and each one earns its place. Walk it from the outside in.

Step one, color space. Convert the patch from BGR to YUV and only touch the Y (luma) and U channels:

yuv = cv2.cvtColor(bgr, cv2.COLOR_BGR2YUV)
for channel in range(2):   # only Y and U
    ...

Luma and chroma are where you can nudge values without the change jumping out at a viewer. The V channel is left alone.

Step two, wavelet. Run a single-level Haar dwt2 on the channel. This splits the image into a coarse approximation ca1 and three detail bands:

ca1, (h1, v1, d1) = pywt.dwt2(yuv[:row//4*4, :col//4*4, channel], 'haar')

Everything gets embedded into ca1, the coarse band. Coarse coefficients survive compression and resizing far better than fine detail, which is the first reason this watermark is hard to wipe by accident.

Step three, the actual bit. Chop ca1 into 4x4 blocks. For each block, take a DCT, then an SVD, and quantize the top singular value:

def diffuse_dct_svd(self, block, wmBit, scale):
    u, s, v = np.linalg.svd(cv2.dct(block))
    s[0] = (s[0] // scale + 0.25 + 0.5 * wmBit) * scale
    return cv2.idct(np.dot(u, np.dot(np.diag(s), v)))

That one line is the trick. Read it slowly.

scale is 36. s[0] is the largest singular value of the block, the bulk of its energy. s[0] // scale snaps it down to a multiple of 36. Then you add back a fixed offset that depends only on the bit you want to store: 0.25 * scale for a 0, 0.75 * scale for a 1. So after this, every block sits at either "a quarter of the way into its 36-wide bin" or "three quarters of the way in," and which one it is encodes the bit.

Decoding just asks where in the bin the value landed:

score = int((s[0] % scale) > scale * 0.5)

Below the halfway mark reads 0, above reads 1. The 0.25 and 0.75 placement is deliberate, it parks each bit as far from the decision boundary as possible so JPEG noise has to be large to flip it.

The part that makes it survive: repeat and average

A 32-bit payload does not need many blocks. A 256x256 patch gives you hundreds of 4x4 blocks. So the encoder writes the same 32 bits over and over:

wmBit = self._encryptions[(num % self._wmLen)]   # cycle through the 32 bits

Block 0 holds bit 0, block 32 holds bit 0 again, and so on. Every bit is stamped dozens of times across the patch.

The decoder collects all the readings for each bit position and takes the mean before thresholding:

scores = [[] for i in range(self._wmLen)]
# ... append a score per block to scores[num % wmLen] ...
avgScores = list(map(lambda l: np.array(l).mean(), scores))
bits = (np.array(avgScores) * 255 > 127)

This is the cheap, effective version of error correction. Compression might flip a handful of blocks, but the average over a hundred copies of the same bit barely moves. That is why the marker survives a save-and-reopen.

Where it gets written, and the skip check

The app does not watermark the whole frame. It picks a 256x256 box, usually a corner or the center, and stamps only that:

positions = [
    (0, box_size, 0, box_size),                       # top-left
    (0, box_size, int(width - box_size), width),      # top-right
    (int(height - box_size), height, 0, box_size),    # bottom-left
    (int(height - box_size), height, int(width - box_size), width),
    # ...center, plus some (None, None, None, None) no-op slots
]
y1, y2, x1, x2 = random.sample(positions, 1)[0]

Note the None slots. On video, some frames get no watermark at all, picked at random. That spreads the marker across frames instead of branding every single one, which keeps any per-frame artifact from accumulating into something visible.

Before stamping, the writer decodes first and skips if the marker is already there:

decoded_text = decoder.decode(bgr, 'dwtDctSvd').decode('utf-8')
if decoded_text != name and width >= box_size and height >= box_size:
    # encode

So re-running the tool on its own output does not double-stamp. And the 256x256 minimum is enforced in the encoder too, it raises if r*c < 256*256. Below that you do not have enough blocks for the repeat-and-average to be reliable.

On the read side, decrypted() checks the four corners of each frame and returns False (not original) the moment any corner decodes to the marker. One hit is enough.

Gotchas you will actually hit

A few things bit me, and will bit you if you reuse this code:

The payload length is wired to 32 bits. EncryptionDecoder('bytes', 32) everywhere. Your marker string has to be exactly four bytes, or you change the length in both encoder and decoder. set_by_ipv4 even asserts self._wmLen == 32.
Lossless transcode matters. The video is reassembled with ffmpeg ... -c:v copy, a stream copy, no re-encode. If you instead re-encode at a low bitrate, you are testing how much the watermark survives, and the answer depends on scale. A bigger scale survives harder compression but distorts the patch more.
The rivaGan method needs ONNX models. The third method loads encoder.onnx and decoder.onnx from the module directory and even runs chmod/icacls on them to fix file permissions. The app does not use this path by default; dwtDctSvd is what videoio.py calls.
There is dead code. infer_dct_svd has a return score followed by an unreachable if score >= 0.5. Do not be confused by it, the function returns the int and exits.

The honest part: this is obfuscation, not protection

Now the uncomfortable bit. This ships inside a fully local desktop app. Everything is on the user's machine: the encoder, the decoder, and the marker string fake, all in plain Python in an open-source repo.

So be clear about what that means:

There is no secret. Anyone who reads the repo knows the algorithm, the scale, and the payload. They can read the marker, strip it, or forge it onto an untouched file.
It is not DRM. It does not stop anyone from using, copying, or editing the output. It does not lock anything.
It is not a cipher, despite the filename. A cipher needs a key the attacker does not have. A local app has no place to hide a key from its own user, so this scheme does not try.

What it actually buys you is provenance under cooperative conditions. If a file passes through normal handling, a re-save, a crop that leaves a corner intact, a platform transcode, the marker is likely still readable, and the app can honestly say "this came out of me." That is genuinely useful for a tool that generates synthetic media and wants to flag its own output. It is a fragile-but-invisible label, not a lock.

The threat it addresses is accidental loss of provenance, not a motivated adversary. Knowing which threat you are defending against is the whole game. Calling it encryption oversells it. Calling it a watermark, and being clear that the watermark is removable by anyone who cares, is the truth.

Wrap-up

The mechanism is one quantized singular value per 4x4 block, the same 32 bits repeated across a 256x256 patch, decoded by averaging. Small, hard to wipe by ordinary handling, and completely open. If you want to read the whole thing, it is portable/src/visual_processing/utils/encryption.py in the repo, with the call site in videoio.py.

If you take one thing away: name your modules for what they do. A watermark called encryption.py will make every future reader, including you, reason about a threat model that the code never had.

LivePortrait without the puppet look: stitching and retargeting in a 126-number MLP

Wlad Radchenko — Wed, 29 Jul 2026 12:36:00 +0000

You animate a still portrait with LivePortrait. The face moves, the mouth talks, the eyes blink. And then you notice it: the head floats. It drifts a little off the neck, the jaw slides past the shoulders, and the whole thing reads like a paper mask on a stick. The puppet look.

I hit this in Wunjo Make while wiring up the portrait animation path. The fix is not a bigger generator or a fancier warp. It is three small multi-layer perceptrons that predict tiny corrections to the face keypoints. One keeps the head attached to the body. One controls the eyes. One controls the lips. They are so small you could miss them in the repo, and they are the reason the output stops looking like a puppet.

This post walks through what those networks take in, what they output, and how a handful of numbers gets added back to the keypoints to fix the artifact. All of it from the actual code in the repo.

The input is one still photo. Everything after is keypoints and deltas. Photo: Unsplash

The one network class, used three ways

Here is the whole network definition. It lives in portable/src/visual_processing/portrait_animation/modules/stitching_retargeting_network.py:

class StitchingRetargetingNetwork(nn.Module):
    def __init__(self, input_size, hidden_sizes, output_size):
        super(StitchingRetargetingNetwork, self).__init__()
        layers = []
        for i in range(len(hidden_sizes)):
            if i == 0:
                layers.append(nn.Linear(input_size, hidden_sizes[i]))
            else:
                layers.append(nn.Linear(hidden_sizes[i - 1], hidden_sizes[i]))
            layers.append(nn.ReLU(inplace=True))
        layers.append(nn.Linear(hidden_sizes[-1], output_size))
        self.mlp = nn.Sequential(*layers)

    def forward(self, x):
        return self.mlp(x)

That is it. Linear, ReLU, Linear, ReLU, Linear. No convolutions, no attention, no recurrence. A plain stack of fully connected layers. The interesting part is that this same class is instantiated three times with different sizes, and each instance solves a different problem.

The sizes come straight from the model config in live_portrait_wrapper.py:

'stitching_retargeting_module_params': {
    'stitching': {
        'input_size': 126, 'hidden_sizes': [128, 128, 64], 'output_size': 65
    },
    'lip': {
        'input_size': 65, 'hidden_sizes': [128, 128, 64], 'output_size': 63
    },
    'eye': {
        'input_size': 66, 'hidden_sizes': [256, 256, 128, 128, 64], 'output_size': 63
    }
}

Those numbers are the whole story. To read them you need one fact: LivePortrait represents a face as 21 implicit 3D keypoints. The config sets num_kp: 21. Twenty one points, each with an x, y, z. So one full set of keypoints is 21 * 3 = 63 numbers. Hold onto 63 and 21, because every size below is built from them.

Stitching: keep the head on the shoulders

Think about what causes the floating head. The animation pipeline takes the source keypoints (the still photo) and the driving keypoints (the motion you want), warps the source feature volume from one to the other, and decodes an image. The warp is good at the face. It does not know or care that the face has to connect to a neck and shoulders that are not moving. So the face can shift as a block and detach from the body.

Stitching is a learned correction that nudges the driving keypoints so the face lines up with where the body already is. Plain version: before we render, we ask a small network "given where the face started and where you want it to go, what small shift puts it back on the shoulders?" Then we add that shift.

The input is 126 numbers. That is 63 for the source keypoints plus 63 for the driving keypoints, flattened and concatenated. Here is the concat, from utils/helper.py:

def concat_feat(kp_source: torch.Tensor, kp_driving: torch.Tensor) -> torch.Tensor:
    # kp_source: (bs, k, 3), kp_driving: (bs, k, 3) -> Return: (bs, 2k*3)
    feat = torch.cat([kp_source.view(bs_src, -1), kp_driving.view(bs_dri, -1)], dim=1)
    return feat

So the stitching MLP sees both poses at once: where the head is anchored, and where the animation wants it. Now the output, 65 numbers. Not 63. The extra two matter. Here is how the wrapper unpacks them in stitching():

def stitching(self, kp_source, kp_driving):
    bs, num_kp = kp_source.shape[:2]
    kp_driving_new = kp_driving.clone()
    delta = self.stitch(kp_source, kp_driving_new)

    delta_exp = delta[..., :3*num_kp].reshape(bs, num_kp, 3)     # 63 -> (21, 3)
    delta_tx_ty = delta[..., 3*num_kp:3*num_kp+2].reshape(bs, 1, 2)  # the last 2

    kp_driving_new += delta_exp
    kp_driving_new[..., :2] += delta_tx_ty
    return kp_driving_new

Read it slowly. The first 63 outputs are a per-keypoint 3D delta. Each of the 21 points gets its own small x, y, z correction. Those get added to every keypoint. The last two outputs are a single global translation in x and y, added to all keypoints at once. So the network can both reshape the face locally and slide the whole face as one piece to meet the shoulders. That global tx, ty is the literal "stop floating" knob.

The correction is small by design. Note that the network has an initialize_weights_to_zero method that zeros every weight and bias. Trained from a zero start, the natural resting state of the delta is "change nothing," and the network only learns to add a nudge where a nudge is needed. A correction network, not a generator.

Eye and lip retargeting: control the blink and the mouth

The other two instances solve a different problem. When you drive one person's face with another's, the eyes and lips do not always transfer cleanly. A person with small eyes driving a person with large eyes leaves the large eyes half open when they should be shut. Lips that should close stay slightly parted. The driving motion is in the right direction but the wrong amount.

Retargeting fixes the amount. The trick: instead of feeding the network a pose, you feed it a target. You tell it "I want the eyes this open" or "I want the lips this closed," and it returns the keypoint delta that gets you there.

That target is a ratio computed straight from facial landmarks, in utils/retargeting_utils.py:

def calculate_distance_ratio(lmk, idx1, idx2, idx3, idx4, eps=1e-6):
    return (np.linalg.norm(lmk[:, idx1] - lmk[:, idx2], axis=1, keepdims=True) /
            (np.linalg.norm(lmk[:, idx3] - lmk[:, idx4], axis=1, keepdims=True) + eps))

def calc_eye_close_ratio(lmk, target_eye_ratio=None):
    lefteye_close_ratio  = calculate_distance_ratio(lmk, 6, 18, 0, 12)
    righteye_close_ratio = calculate_distance_ratio(lmk, 30, 42, 24, 36)
    ...

def calc_lip_close_ratio(lmk):
    return calculate_distance_ratio(lmk, 90, 102, 48, 66)

A close ratio is vertical opening over horizontal width. For an eye: distance between the top and bottom lids divided by the eye's corner-to-corner width. A wide open eye has a high ratio, a shut eye a low one. Dividing by width makes it scale-free, so it works whether the face is big or small in the frame. The eps = 1e-6 is there so a degenerate landmark set never divides by zero.

Now the sizes click into place. The lip network takes input_size 65: that is 63 for the source keypoints plus a lip ratio of 2 numbers. From the wrapper:

def retarget_lip(self, kp_source, lip_close_ratio):
    # kp_source: BxNx3, lip_close_ratio: Bx2
    feat_lip = concat_feat(kp_source, lip_close_ratio)
    delta = self.stitching_retargeting_module['lip'](feat_lip)
    return delta.reshape(-1, kp_source.shape[1], 3)   # -> (B, 21, 3)

The eye network takes input_size 66: 63 keypoints plus an eye ratio of 3 numbers (left eye, right eye, and a target). Both output 63, a per-keypoint 3D delta that you add to the source keypoints. The eye network has the deeper stack, [256, 256, 128, 128, 64], because opening and closing eyes the right amount across identities is the harder regression. Same class, more layers.

The pattern is identical to stitching: feed in the current keypoints plus a small control signal, get back a keypoint delta. The only difference is what the control signal is. For stitching it is the driving pose. For retargeting it is a ratio you can dial yourself.

How it actually runs

These three calls compose. In the wrapper you can see the full chain when you control everything by hand:

eyes_delta = self.retarget_eye(x_s_user, combined_eye_ratio_tensor)
lip_delta  = self.retarget_lip(x_s_user, combined_lip_ratio_tensor)
x_d_new    = x_s_user + eyes_delta + lip_delta
x_d_new    = self.stitching(x_s_user, x_d_new)

Deltas just add. You take the source keypoints, add the eye correction, add the lip correction, then run stitching last so the final pose is glued back to the body after you have changed the eyes and lips. Order matters: stitch after retargeting, not before, so the global tx, ty accounts for everything you moved.

Every call runs under torch.no_grad(). These are inference-only correction heads. They are also cheap. A 126-input MLP with three hidden layers is a rounding error next to the appearance extractor and the dense motion network, which is why LivePortrait can offer this control without a real frame-rate cost.

Gotchas worth knowing

A few things I learned the slow way.

Stitching is on by default, retargeting is not. In the config, flag_stitching defaults to True and the eye and lip retargeting flags default to False. The comments in argument_config.py flag eye and lip retargeting as "not recommend to be True, WIP." Stitching is the safe, always-want-it correction. Eye and lip retargeting are an expressive extra that can introduce jitter, which is why in the Wunjo path the per-frame deltas get run through a Kalman filter before they are applied. If your blink flickers frame to frame, that smoothing is the lever.

Stitching does not load by default for animals. The same comment notes you want stitching off for large head movement or for animal sources. The networks were trained on the human face keypoint layout, so the corrections only make sense there.

The whole module is optional. Look at the wrapper init: if checkpoint_S is missing, self.stitching_retargeting_module = None, and stitching() returns the driving keypoints untouched. The pipeline still runs. It just runs with the floating head. That is the cleanest way to see what these tiny networks buy you: delete them and watch the head detach.

Wrap-up

The lesson I keep relearning: the artifact you cannot stand is often fixed by the smallest part of the system. Here the floating head, the puppet look, the half-open eyes, all come down to three ReLU MLPs that predict a few keypoint deltas and add them back. 126 numbers in, 65 out for stitching. 65 in, 63 out for lips. 66 in, 63 out for eyes. Everything else in the pipeline is huge by comparison, and none of it would look right without these.

If you want to read the code yourself, start at stitching_retargeting_network.py for the class, then live_portrait_wrapper.py for stitching, retarget_eye, and retarget_lip. The full repo is open source on GitHub.

References

Guo, Zhang, Liu, et al. "LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control." 2024. arXiv:2407.03168. https://arxiv.org/abs/2407.03168