DEV Community: Wlad Radchenko

Keep the same face across frames with a tiny embedding memory

Wlad Radchenko — Wed, 26 Aug 2026 11:47:00 +0000

You click a face in frame 1. Two people are on screen. By frame 40 the tool has quietly swapped to the other person, because for a few frames they leaned closer to the camera and got bigger. Anyone who has built face tracking on real video has hit this. The naive fix everyone tries first is "pick the biggest face" or "pick the most centered face", and it fails the moment a second face exists.

This post walks through the part of Wunjo Make that fixes it: a small per-identity memory of face embeddings that says "this is the person you picked, keep following them" even when someone else is bigger or more centered. The code lives in portable/src/visual_processing/face_detection/recognition.py, in the FaceRecognition class.

The everyday version first

Think about recognizing a friend in a crowd. You do not pick whoever is closest or tallest. You remember their face and scan for a match. If someone walks between you, you do not suddenly decide the stranger is your friend.

That is the whole idea here. We keep a running memory of what the chosen face looks like, as numbers, and each new frame we ask "which of these detections looks most like the one I remember?" before we worry about where it is on screen.

The numbers come from a face recognition network. InsightFace's buffalo_l pack runs an ArcFace model that turns a face crop into a 512-dimension vector, an embedding (Deng et al., CVPR 2019; arXiv:1801.07698). Two crops of the same person land near each other in that 512-D space. Two different people land far apart. The detector that finds the boxes in the first place is RetinaFace (Deng et al., 2019; arXiv:1905.00641). We do not need the math of either to use them. We need one operation: distance between two embeddings.

The memory

Here is the state the class holds:

class FaceRecognition(FaceAnalysis):
    def __init__(self, name='buffalo_l', root='~/.insightface', allowed_modules=None, **kwargs):
        super().__init__(name=name, root=root, allowed_modules=allowed_modules, **kwargs)
        self.id = 0
        self.window = {}           # per-id list of past detections
        self.window_size = 50      # warm-up length / last n distances
        self.max_window_size = 250 # cap on stored embeddings in RAM
        self.running_average = {}
        self.max_rate_of_change = 0.05

window is a dict keyed by a face id, so you can track more than one target at once, each with its own memory. Every entry stored for an id is small:

def update_memory(self, face):
    self.set_face_id()
    center = ((face.bbox[0] + face.bbox[2]) / 2, (face.bbox[1] + face.bbox[3]) / 2)
    self.window[self.id].append({
        'embedding': face.normed_embedding,
        'center': center,
        'gender': face.gender,
    })
    if len(self.window[self.id]) > self.max_window_size:
        self.window[self.id].pop(0)

Note normed_embedding, not the raw embedding. ArcFace embeddings are L2-normalized, so they sit on a unit sphere. That matters for the distance step, coming up. The list is a sliding window: once it passes 250 entries it drops the oldest. So the memory is recent. If a person's lighting drifts over a long shot, the memory drifts with them instead of clinging to frame 1 forever.

Distance is just L2

The "looks like" test is one line:

@staticmethod
def calculate_distance(embedding1, embedding2):
    return np.linalg.norm(embedding1 - embedding2)

Straight L2 (Euclidean) distance. Because both vectors are already normalized, this distance is bounded and behaves consistently across face sizes and lighting. Small distance, same person. Larger distance, probably someone else.

For each detected face in the current frame, we do not compare against one remembered embedding. We compare against every embedding in the window and take the mean:

face.mean_distance = np.mean([
    self.calculate_distance(face.normed_embedding, known_face["embedding"])
    for known_face in self.window[self.id]
]) if len(self.window[self.id]) > 0 else float('inf')

Averaging over the window is the quiet trick. A single bad frame, motion blur, a half-turn of the head, gives a noisy embedding. If you compared against only the last frame, one ugly frame could throw the match off. Averaging across the recent window smooths that out. One bad embedding in 50 barely moves the mean.

The sort that ignores "biggest"

This is the heart of it. Selection happens in sort_by_direction, in the distance-from-embedding branch:

if direction == 'distance-from-embedding':
    return sorted(
        filter(lambda face: face.mean_distance < threshold * face.dynamic_threshold, faces),
        key=lambda face: (((face['bbox'][2]+face['bbox'][0])/2 - face_center[0])**2
                        + ((face['bbox'][3]+face['bbox'][1])/2 - face_center[1])**2)**0.5
    )

Read it in two moves. First the filter: throw away any face whose mean embedding distance is too high. That is the identity gate. A stranger who happens to be huge and dead-center never makes it past this line, because their embedding does not match the memory. Then the sorted: among the faces that did match, pick by plain pixel distance from where the person was last seen. Spatial position is the tie-breaker, not the decision.

That ordering is the fix. "Biggest face" and "most centered face" use geometry as the decision. Here geometry only breaks ties between faces that already passed the identity check.

The threshold is not a constant

threshold * face.dynamic_threshold is the gate. Two pieces. dynamic_threshold depends on how big the face is in the frame:

@staticmethod
def get_dynamic_threshold(face, height):
    face_height = (face.bbox[3] - face.bbox[1]) / height
    if face_height < 0.1:
        return 1.05
    if face_height < 0.2:
        return 1.2
    if face_height < 0.4:
        return 1.3
    elif face_height < 0.8:
        return 1.35
    elif face_height < 0.9:
        return 1.4
    else:
        return 1.45

Why scale by size? A tiny, far-away face gives a noisier, less reliable embedding, so we keep the gate tight (1.05) and demand a close match. A large, clear face is trustworthy, so we loosen the gate (up to 1.45) and forgive more variation, because a big face turning its head can legitimately drift further in embedding space while still being the same person. The constant threshold is the per-id running_average, which I will get to.

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

Multiple faces in one frame, the exact situation where "pick the biggest" jumps to the wrong person. Photo: Unsplash

The running average and why it is clamped

The memory has no idea what a "matching" distance looks like on your specific video. A studio close-up and a shaky handheld clip have different baseline distances for the same person. So the gate self-calibrates. Once the window has warmed up past window_size - 1 entries, it seeds a per-id running_average:

if len(self.window[self.id]) > self.window_size - 1:
    if self.running_average[self.id] is None:
        self.running_average[self.id] = max([
            self.calculate_distance(ret[0].normed_embedding, known_face["embedding"])
            for known_face in self.window[self.id]
        ])
    else:
        change = max([
            self.calculate_distance(ret[0].normed_embedding, known_face["embedding"])
            for known_face in self.window[self.id]
        ]) - self.running_average[self.id]
        change = np.clip(change, -self.max_rate_of_change, self.max_rate_of_change)
        self.running_average[self.id] += change

The seed is the worst (max) match distance inside the current window, a rough "this is how far the same person can drift." After that, each frame nudges the average toward the new worst-match distance, but the nudge is clamped to ±0.05.

That clamp is the safety rail. Imagine the memory got fooled for one frame and locked onto the wrong face. Without the clamp, that wrong face would yank the running average toward its own distances in a single step, and the gate would re-tune itself around the impostor. With the clamp, the gate can only move 0.05 per frame. One bad frame cannot rewrite the baseline. Recovery is possible because the correct face keeps matching at low distance while the average can only crawl.

Warm-up: position first, identity later

There is a chicken-and-egg problem. On frame 1 the memory is empty, so embedding distance is meaningless. The code handles this by falling back to pure position during warm-up:

ret = sort_by_direction(
    ret,
    'distance-from-retarget-face'
        if direction == 'distance-from-embedding' and len(self.window[self.id]) < self.window_size
        else direction,
    face_center, face_gender,
    self.running_average[self.id] if self.running_average[self.id] else "inf"
)

Until the window holds window_size (50) entries, distance-from-embedding silently degrades to distance-from-retarget-face, which is just Euclidean distance from the click point with no identity gate. And on the very first frame, the seed is the face whose box actually contains the click point:

if len(self.window[self.id]) == 0 and face_center[0] and face_center[1]:
    for face in ret:
        x1, y1, x2, y2 = face.bbox
        if x1 <= face_center[0] <= x2 and y1 <= face_center[1] <= y2:
            ret = [face]
            break
    else:
        ret = []

So the flow is: you click a person, the box under your click seeds the memory, the next ~50 frames are tracked by position while the embedding window fills, and from then on identity drives selection and position is only the tie-breaker.

Gotchas you will actually hit

A few things that cost real debugging time:

Feed RGB, consistently. The class is built on InsightFace; mixing BGR and RGB between detection and the embedding model gives embeddings that drift for no visible reason. Stay in one color space end to end.
Call clear() between clips. window and running_average persist on the instance. Reuse the object across two unrelated videos and the second video starts with the first video's memory. clear() resets both.
Set self.id per target. All the state is keyed by self.id. Track two people by passing id= per call so each gets its own window; forget it and both collapse into id 0.
Gender filtering is off in this version. There is a commented-out variant of the filter that also matched on face.gender. It is disabled because of a gender-detection bug, so do not count on gender as part of the gate today. The active filter is distance-only.
The first 50 frames are not identity-aware. During warm-up the tracker can still be lured by a closer face, because it is running on position alone. If your subject is not the nearest face to the click in those early frames, seed more carefully.

Wrap-up

The mechanism is small. A sliding window of normalized ArcFace embeddings per identity, a mean L2 distance as the match score, a size-aware gate that self-calibrates with a clamped running average, and position used only to break ties among faces that already passed the identity check. That combination is why the tracker stays on the person you picked instead of hopping to whoever is biggest.

Read the full file here: recognition.py in Wunjo Make. It is open source, so you can paste the class into a notebook, feed it frames, and watch the window fill.

References

Deng, Guo, Xue, Zafeiriou. "ArcFace: Additive Angular Margin Loss for Deep Face Recognition." CVPR 2019. arXiv:1801.07698. https://arxiv.org/abs/1801.07698 (InsightFace / buffalo_l recognition.)
Deng, Guo, Zhou, Yu, Kotsia, Zafeiriou. "RetinaFace: Single-stage Dense Face Localisation in the Wild." 2019. arXiv:1905.00641. https://arxiv.org/abs/1905.00641

Upscaling a huge image on a small GPU: tiled super-resolution without seams

Wlad Radchenko — Tue, 25 Aug 2026 13:46:00 +0000

You point Real-ESRGAN at a 6000x4000 photo, ask for 4x, and CUDA dies with out of memory. The model is fine. Your GPU just cannot hold a 24000x16000 tensor plus the activations to produce it. An 8GB card does not stand a chance, and even a 24GB card falls over once the input gets big enough.

The fix is to never feed the whole image to the model at once. Cut it into tiles, upscale each tile on its own, then glue the results back together. The trick is gluing them without leaving visible lines where the tiles meet. This is a walkthrough of the exact tiling code in Wunjo Make's RealESRGANer, which is the standard Real-ESRGAN inference helper (Wang et al., ICCV 2021 Workshops).

Tiling a picture is the same idea as tiling a wall, except the hard part is hiding the grout. Photo: Unsplash.

The two knobs that matter

Two numbers control everything here:

self.tile_size = tile      # e.g. 256: each tile is 256x256 input pixels. 0 = no tiling
self.tile_pad  = tile_pad  # default 10: extra pixels read around each tile

tile_size is how big each chunk is in input pixels. Smaller tiles fit on smaller cards but mean more chunks and more overhead. tile_pad is the overlap, the halo of extra pixels read around each tile so the model has real neighbors to look at near the tile edge. Default is 10. Keep that number in mind, it is the whole reason seams do not show.

If tile_size is 0, the model runs on the full image and you are back to the OOM problem. Everything below assumes tile_size > 0.

Step 1. Lay out the grid

Given the padded input image, work out how many tiles fit across and down:

batch, channel, height, width = self.img.shape
output_height = height * self.scale
output_width  = width  * self.scale

# start with a black canvas at the final, upscaled size
self.output = self.img.new_zeros(output_shape)

tiles_x = math.ceil(width  / self.tile_size)
tiles_y = math.ceil(height / self.tile_size)

Two things to notice. The output is allocated once, at full upscaled resolution, as a black image. That is the only big tensor that lives on the device for the whole run, and it is just storage, not activations, so it is cheap compared to running the model on the full frame. Each tile then writes its piece into this canvas.

The ceil means the last column and last row can be partial. A 600-pixel-wide image with tile_size=256 gives ceil(600/256) = 3 columns: 256, 256, and a runt of 88. The code handles that with a min clamp, which is the next part.

Step 2. Cut one tile, with its halo

For each tile in the grid, the code defines two rectangles. The core is the part this tile is responsible for. The padded rectangle is what actually gets read and fed to the model.

ofs_x = x * self.tile_size
ofs_y = y * self.tile_size

# core area: this tile's own region, clamped at the image edge
input_start_x = ofs_x
input_end_x   = min(ofs_x + self.tile_size, width)
input_start_y = ofs_y
input_end_y   = min(ofs_y + self.tile_size, height)

# padded area: core plus a tile_pad halo, clamped at the image edge
input_start_x_pad = max(input_start_x - self.tile_pad, 0)
input_end_x_pad   = min(input_end_x   + self.tile_pad, width)
input_start_y_pad = max(input_start_y - self.tile_pad, 0)
input_end_y_pad   = min(input_end_y   + self.tile_pad, width)  # note: clamps to width

The max(..., 0) and min(..., width) are what keep the halo inside the image. A tile in the middle gets a full 10-pixel halo on all four sides. A tile against the left edge gets no halo on the left, because there is nothing there to read. The halo is real image data borrowed from the neighboring tile, not invented pixels.

That distinction is the entire point. A super-resolution network looks at a neighborhood around each pixel to decide what to draw. A pixel sitting one row inside a tile boundary needs to see the rows on the other side of that boundary. If you fed it a bare tile, that pixel would see the tile's edge and the model would hallucinate a border there. By reading 10 extra pixels of the actual neighbor, the model gets honest context right up to the boundary.

Then the padded tile, and only the padded tile, goes through the model:

input_tile = self.img[:, :, input_start_y_pad:input_end_y_pad,
                            input_start_x_pad:input_end_x_pad]
with torch.no_grad():
    output_tile = self.model(input_tile)

This is the small tensor. A 256x256 tile with a 10-pixel halo is 276x276, and at 4x its output is 1104x1104. That fits on a modest GPU even though the full output would not.

Step 3. Stitch the core back, throw the halo away

Now the careful part. The model upscaled the padded tile, halo included. But the halo pixels were also upscaled inside the neighboring tile, where they were the core. If you wrote the halo too, neighboring tiles would overwrite each other's edges and you would get exactly the seams you were trying to avoid.

So the code computes where the core sits inside the upscaled padded tile, and copies only that:

# where this tile's core lands in the final output (input coords times scale)
output_start_x = input_start_x * self.scale
output_end_x   = input_end_x   * self.scale
output_start_y = input_start_y * self.scale
output_end_y   = input_end_y   * self.scale

# where the core sits inside the upscaled padded tile
output_start_x_tile = (input_start_x - input_start_x_pad) * self.scale
output_end_x_tile   = output_start_x_tile + input_tile_width  * self.scale
output_start_y_tile = (input_start_y - input_start_y_pad) * self.scale
output_end_y_tile   = output_start_y_tile + input_tile_height * self.scale

# write only the core into the canvas
self.output[:, :, output_start_y:output_end_y, output_start_x:output_end_x] = \
    output_tile[:, :, output_start_y_tile:output_end_y_tile,
                      output_start_x_tile:output_end_x_tile]

Read (input_start_x - input_start_x_pad) as "how many halo pixels are on the left of this tile." For an interior tile that is 10, so the slice skips the first 10*scale columns of the upscaled tile. For a left-edge tile it is 0, so nothing is skipped, which is correct because there was no left halo. Everything is multiplied by scale to move from input coordinates to output coordinates.

The result: each output pixel is written exactly once, by the tile that owns it, and every owned pixel was upscaled with real context around it. The tiles meet edge to edge with no overlap in the final image, but because each side of the boundary was computed with knowledge of the other side, the two halves line up.

A note on what "overlap" means here

This is worth being precise about, because the word overlap gets used two ways.

In this code, tiles overlap on the way in (the halo) but not on the way out. The padded read regions of two neighbors share tile_pad columns of input. The written core regions do not share anything. There is no alpha blend or feathering across the boundary. The seam is prevented by giving each side correct context, not by averaging the two sides together.

That is different from a feathered-overlap approach, where tiles overlap in the output and you cross-fade them. Both work. The halo-and-crop method here is simpler and avoids any softening from the blend, at the cost of needing the model to be consistent across the boundary, which a convolutional upscaler with enough halo generally is. With tile_pad=10, ten pixels of context is plenty for Real-ESRGAN's receptive field on its own scale.

Step 4. The whole-image padding around the edges

There is a second, smaller padding layer that is easy to miss. Before any tiling, pre_process pads the entire image:

if self.pre_pad != 0:
    self.img = F.pad(self.img, (0, self.pre_pad, 0, self.pre_pad), 'reflect')

This pre_pad (default 10) handles the outer border of the whole image, where there is no neighbor to borrow from. It mirrors the edge pixels outward with reflect so the model does not see a hard frame at the very edge of the picture. There is also a mod_scale pad that rounds the dimensions up to a multiple the network needs (mod 2 when scale==2, mod 4 when scale==1).

Both pads are cropped back off at the end, scaled up by the upscale factor:

# remove the mod pad, then the pre pad, both at output scale
self.output = self.output[:, :, 0:h - self.mod_pad_h * self.scale,
                                0:w - self.mod_pad_w * self.scale]
self.output = self.output[:, :, 0:h - self.pre_pad   * self.scale,
                                0:w - self.pre_pad   * self.scale]

So you get back exactly input_size * scale, no leftover border.

Gotchas I would warn you about

A few things that bite people:

Pick tile_size to fit VRAM, not to be tidy. Memory scales with tile area, not perimeter. Going from 512 to 256 roughly quarters the per-tile activation memory. If you OOM, halve the tile size before touching anything else.
tile_pad is in input pixels, halo cost is per tile. A bigger halo means safer seams but more redundant compute, since every halo pixel is upscaled twice (once as halo, once as someone's core). The default 10 is a good balance, do not crank it to 100.
The OOM is swallowed. The model call is wrapped in try/except RuntimeError that just prints the error. If a single tile is still too big, you get a black square in the output instead of a crash, which can be confusing. If you see a black tile, your tile is still too large for the card.
Smaller tiles are slower. More tiles means more kernel launches and more halo overhead. There is a real time-versus-memory trade here. On a roomy GPU, use the biggest tile that fits.
Edge tiles are smaller. The last row and column are partial, and the halo is asymmetric there. The code already handles this with the min/max clamps, but if you ever rewrite this loop, that is the case that breaks first.

Takeaway

Tiled super-resolution is three moves. Allocate the full output once as a black canvas. For each tile, read a core region plus a tile_pad halo of real neighbor pixels, upscale that padded tile, then write back only the core and discard the halo. The halo gives the model honest context at every boundary, the crop makes sure each output pixel is owned by exactly one tile, and the two together mean the tiles line up without a blend.

That is how an 8GB card upscales a photo whose full output would need ten times that. The model never sees more than one tile at a time.

The full method is tile_process in visual_processing/enhancement/realesrgan.py in the Wunjo Make repo. If you are hitting OOM on large inputs, start by setting a tile_size and leave tile_pad at 10.

References

Wang, Xie, Dong, Shan. "Real-ESRGAN: Training Real-World Blind Super-Resolution with Pure Synthetic Data." ICCV 2021 Workshops. arXiv:2107.10833

A referee for lip-sync: scoring whether audio and video match with two embedding towers

Wlad Radchenko — Wed, 19 Aug 2026 12:48:00 +0000

You have a talking-head video. The audio and the lips are close, but not exact. Maybe the sound runs three frames ahead. Maybe two people are on screen and you need to know which mouth the voice belongs to. How do you measure that automatically, without a human watching every clip?

That is the problem SyncNet solves, and the open-source Wunjo Make repo ships two versions of it. I want to walk through the actual code: how the network is built, how audio turns into the thing the network eats, and how the codebase turns the network's output into a usable offset and a confidence number.

Plain version first. You build two small networks. One looks at the lips. One listens to the sound. Each one squeezes its input down to a short vector. If the lips and the sound belong together, the two vectors land close. If they do not, the vectors land far apart. Distance is the whole game.

Two towers, never mixed

Here is the first version, SyncNet_color in portable/src/visual_processing/lip_sync/syncnet.py. It is the Wav2Lip-style net. The shape of the very first layer tells you almost everything:

self.face_encoder = nn.Sequential(
    Conv2d(15, 32, kernel_size=(7, 7), stride=1, padding=3),
    ...
)

self.audio_encoder = nn.Sequential(
    Conv2d(1, 32, kernel_size=3, stride=1, padding=1),
    ...
)

The face tower starts at 15 input channels. The audio tower starts at 1. Those two numbers are the design.

Why 15? Because a single still frame cannot tell you anything about motion, and lip-sync is motion. So the face tower does not take one frame. It takes five frames, stacked. Five RGB frames is 5 times 3, which is 15 channels. The network sees a short burst of mouth movement in one shot, not a single freeze-frame.

The audio tower takes 1 channel because a mel spectrogram is a single grayscale image: time on one axis, frequency on the other, loudness as brightness. One channel is all it needs.

The two towers never share weights and never touch inside the network. They run in parallel, on different inputs, and only meet at the very end as two vectors. That is the contrastive idea: learn a face space and a sound space that line up only for matching pairs.

The trick at the bottom: normalize, then compare

The forward pass is short, and the last three lines are the part that matters:

def forward(self, audio_sequences, face_sequences):
    face_embedding = self.face_encoder(face_sequences)
    audio_embedding = self.audio_encoder(audio_sequences)

    audio_embedding = audio_embedding.view(audio_embedding.size(0), -1)
    face_embedding = face_embedding.view(face_embedding.size(0), -1)

    audio_embedding = F.normalize(audio_embedding, p=2, dim=1)
    face_embedding = F.normalize(face_embedding, p=2, dim=1)

    return audio_embedding, face_embedding

Both embeddings get L2-normalized. That means every vector is pushed onto a unit sphere: only its direction survives, its length is thrown away. Once both vectors have length one, Euclidean distance and cosine similarity carry the same information. A small distance means the directions agree, which means the lips and the sound match.

This is the part people skip when they reimplement these nets and then wonder why training is unstable. If you compare raw embeddings, a loud frame or a bright frame can pump up the vector length and swamp the signal you actually care about. Normalizing first removes magnitude from the equation. You are left comparing shape, not volume.

The forward pass returns the two vectors. It does not return a score. The caller decides what distance means.

Lining up sound and picture is a matching problem, not a generation problem. Photo: Unsplash

Feeding the audio tower: wav to mel

The audio tower wants a mel window, and portable/src/visual_processing/lip_sync/wav2mel.py builds it. The interesting part is the chunking, where audio time gets cut to match video time:

def chunk_mel(self, mel, fps, mel_step_size):
    mel_chunks = []
    mel_idx_multiplier = 80./fps
    i = 0
    while True:
        start_idx = int(i * mel_idx_multiplier)
        if start_idx + mel_step_size > len(mel[0]):
            mel_chunks.append(mel[:, len(mel[0]) - mel_step_size:])
            break
        mel_chunks.append(mel[:, start_idx : start_idx + mel_step_size])
        i += 1
    return mel_chunks

mel_step_size is 16. So each chunk is a fixed 16-column slice of the spectrogram, which is the window the audio tower is built to swallow.

The number 80./fps is the bridge between two clocks. The mel is computed at 16000 Hz with a 200-sample hop (see hparams.py), which works out to 80 mel columns per second of audio. Video runs at fps frames per second. So 80./fps is "how many mel columns advance per video frame." At 25 fps that is 3.2 columns per frame. The loop steps one video frame at a time and grabs the matching audio window each time, so chunk number i lines up with video frame number i. The last chunk clamps to the end of the array so you never run off the edge.

There is also a small, honest gotcha baked into the code:

def check_for_nan(self, mel):
    if np.isnan(mel.reshape(-1)).sum() > 0:
        raise ValueError('Mel contains nan! Using a TTS voice? '
                         'Add a small epsilon noise to the wav file and try again')

Dead-silent synthetic audio can produce a log of zero, which is a NaN, which poisons the whole spectrogram. The fix in the message is blunt and it works: add a hair of noise so the log has something to bite on.

Where the scoring actually happens

Now the honest part, because the code does not lie about it. SyncNet_color is exported from the module, but at inference time it is not wired up to produce a sync score. The syncnet_wt you see in hparams.py is a training-time loss weight for the lip-sync generator, not a runtime judge. So if you are looking for the place that actually scores audio against video in this repo, it is the other SyncNet.

That second version lives in portable/src/visual_processing/face_detection/sync_net/, and it is the classic Chung and Zisserman SyncNet (ACCV 2016). Same two-tower idea, different build: a 3D-conv face tower and an MFCC audio tower, with separate fully connected heads forward_lip and forward_aud in model.py. This one is alive and doing work.

The work is the offset search, in instance.py:

def calc_pdist(feat1, feat2, vshift=10):
    win_size = vshift * 2 + 1
    feat2p = torch.nn.functional.pad(feat2, (0, 0, vshift, vshift))
    dists = []
    for i in range(0, len(feat1)):
        dists.append(
            torch.nn.functional.pairwise_distance(
                feat1[[i], :].repeat(win_size, 1),
                feat2p[i:i + win_size, :]))
    return dists

Read it as a sliding test. For each video frame's face vector, it does not just compare against the audio vector at the same instant. It compares against a window of audio vectors, from vshift frames early to vshift frames late. That gives you a distance for every possible small shift between the two streams.

Then evaluate collapses that into an answer:

mdist = torch.mean(torch.stack(dists_list, 1), 1)
minval, minidx = torch.min(mdist, 0)

offset = vshift - minidx
conf = torch.median(mdist) - minval

Average the distances across all frames for each shift. The shift with the smallest average distance is the true audio-video offset, because that is where the lips and the sound agree the most. The confidence is the gap between the typical distance and the best distance. A big gap means one shift clearly won, so you can trust it. A tiny gap means every shift looked about the same, so the audio probably is not driving this face at all.

Picking the right face from a crowd

The payoff is in recognition.py, in FaceRecognitionSpeaker. When a video has several faces, the code crops each face track, runs syncnet.evaluate on each one, and keeps the per-frame confidence:

offset, conf, dist = syncnet.evaluate(tmp_dir, video_file=fname)
...
fconf = np.median(mean_dists) - fdist
fconfm = signal.medfilt(fconf, kernel_size=9)
...
if conf > threshold:
    if scene_dict is None or scene_dict.get("conf") is None or scene_dict.get("conf") < conf:
        scenes_dict[offset_idx] = {"x": x, "y": y, "idx": idx, "conf": conf}

For each frame the code keeps the track with the highest sync confidence, as long as it clears threshold (default 3.0). That is active-speaker detection built entirely on distance: the mouth whose movements match the soundtrack best is the one talking. The medfilt with a kernel of 9 smooths the per-frame confidence so a couple of noisy frames cannot flip the decision.

So the same plain idea drives both versions. Two towers, two vectors, compare by distance. The second version just adds the sliding window so distance gives you an offset and a trust score, not only a yes or no.

Things that will trip you up

A few practical notes if you go poking at this code.

The two SyncNets are not interchangeable. SyncNet_color eats 15-channel stacked RGB and a mel window. SyncNetInstance eats a 5-frame 3D-conv volume and an MFCC slice. Different preprocessing, different checkpoints. Do not feed one's input to the other.

The audio clock is fixed at 16000 Hz and the offset search assumes 25 fps video. The evaluate function normalizes input to 25 fps first for exactly this reason. If your fps drifts, your offset frame count drifts with it.

The confidence is relative, not absolute. It is median distance minus minimum distance, so it tells you how clearly one shift beat the rest for this clip. It is not a percentage you can compare across different videos without care.

If you want to read it all yourself, the model is in portable/src/visual_processing/lip_sync/syncnet.py, the audio prep in wav2mel.py, and the live scoring path in face_detection/sync_net/instance.py and face_detection/recognition.py.

References

Chung, Zisserman. "Out of Time: Automated Lip Sync in the Wild." ACCV 2016 Workshops. The original two-tower SyncNet and the offset-by-distance idea. https://www.robots.ox.ac.uk/~vgg/publications/2016/Chung16a/
Prajwal, Mukhopadhyay, Namboodiri, Jawahar. "A Lip Sync Expert Is All You Need for Speech to Lip Generation In the Wild." ACM Multimedia 2020. arXiv:2008.10010. The Wav2Lip line that SyncNet_color belongs to. https://arxiv.org/abs/2008.10010

Keeping the same face across generations: how reference photos enter an OmniGen prompt

Wlad Radchenko — Tue, 18 Aug 2026 11:45:00 +0000

You want to generate a new picture of a specific person. Not a generic face, that exact person, the one in your three reference photos. You write a prompt like "the woman in <|image_1|> standing in a kitchen" and you expect the face to survive. Half the time it does not. You get someone who is sort of in the neighborhood, a cousin maybe.

The interesting part of identity preservation in a model like OmniGen is not the diffusion. It is the few dozen lines that decide where your reference photos go inside the prompt and how the model is told "these pixels are the person, the rest is instructions." That happens in the processor, before a single denoising step runs. This is a walkthrough of that code path in the Wunjo Make repo.

OmniGen (Xiao et al., 2024) is a single transformer that takes text and images mixed together in one sequence, instead of a separate text encoder feeding a separate U-Net. That design is exactly why the conditioning code is worth reading: text tokens and image features live in the same stream, and the splice between them is where identity is kept or lost.

Several reference photos and one prompt all become one flat sequence of tokens. Photo: Unsplash.

The prompt is a string with image holes in it

You reference images by tag. The prompt literally contains <|image_1|>, <|image_2|>, and so on. The first job is to split the text at those tags and tokenize the text between them. From processor.py:

pattern = r"<\|image_\d+\|>"
prompt_chunks = [self.text_tokenizer(chunk).input_ids for chunk in re.split(pattern, text)]

for i in range(1, len(prompt_chunks)):
    if prompt_chunks[i][0] == 1:
        prompt_chunks[i] = prompt_chunks[i][1:]

re.split on the tag pattern gives you the text pieces in order, with the tags removed. Each piece is tokenized on its own. That second loop is a small but real detail: the tokenizer prepends a beginning-of-sequence token (id 1) to every chunk, and you only want one at the very start of the whole prompt. So every chunk after the first gets its leading 1 stripped. Miss this and you sprinkle BOS tokens through the middle of your sequence, which is not what the model was trained on.

Tags carry an order, and the order is enforced

Next the code reads the tags back out and turns them into integer ids:

image_tags = re.findall(pattern, text)
image_ids = [int(s.split("|")[1].split("_")[-1]) for s in image_tags]

unique_image_ids = sorted(list(set(image_ids)))
assert unique_image_ids == list(range(1, len(unique_image_ids)+1)), \
    f"image_ids must start from 1, and must be continuous int, ... cannot be {unique_image_ids}"
assert len(unique_image_ids) == len(input_images), \
    f"total images must be the same as the number of image tags, ..."

input_images = [input_images[x-1] for x in image_ids]

Two assertions do real work here. The ids must be [1, 2, 3, ...] with no gaps, and the number of distinct tags must equal the number of images you passed. If you write <|image_1|> and <|image_3|> but skip 2, it stops. If you pass two photos but only tagged one, it stops. This is the kind of guard you want, because the failure it prevents is silent: the model would happily generate from a misaligned set and you would spend an hour wondering why the face is wrong.

That last line is the one to sit with. input_images = [input_images[x-1] for x in image_ids] reorders your actual image tensors to match the order the tags appear in the text. The position in the sentence, not the position in your list, is what counts. A tag can repeat too, and the same photo gets placed at each spot it is named.

Reserving the right number of slots for each image

Now the sequence gets assembled. Text token ids and image placeholders are concatenated into one flat list:

all_input_ids = []
img_inx = []
idx = 0
for i in range(len(prompt_chunks)):
    all_input_ids.extend(prompt_chunks[i])
    if i != len(prompt_chunks) - 1:
        start_inx = len(all_input_ids)
        size = input_images[i].size(-2) * input_images[i].size(-1) // 16 // 16
        img_inx.append([start_inx, start_inx + size])
        all_input_ids.extend([0] * size)

return {"input_ids": all_input_ids, "pixel_values": input_images, "image_sizes": img_inx}

Walk it. After each text chunk (except the last) there is an image, so the code computes how many tokens that image will occupy and pushes that many placeholder 0s into the sequence. The count is H * W // 16 // 16. An image is encoded by the VAE down by a factor of 8, then split into patches of size 2 by the patch embedder, so each side shrinks by 16 total and one image becomes (H/16) * (W/16) tokens. A 1024 by 1024 photo is 64 * 64 = 4096 tokens. That is a lot of sequence per image, which is also why three references cost real memory.

The 0s are pure reservation. They are placeholders, not the image. What matters is img_inx, the list of [start, end] spans recording exactly where each image sits in the flat sequence. Those spans are the address book the model uses later to drop the real image features into place.

The placeholder spans become image features

The actual splice happens in model.py, inside forward. The input ids go through the text embedding table as usual, then the recorded spans are overwritten with VAE-encoded, patch-embedded image latents:

condition_embeds = self.llm.embed_tokens(input_ids).clone()
input_img_inx = 0
for b_inx in input_image_sizes.keys():
    for start_inx, end_inx in input_image_sizes[b_inx]:
        condition_embeds[b_inx, start_inx:end_inx] = input_latents[input_img_inx]
        input_img_inx += 1

This is the whole trick. The [0] placeholders were embedded to some throwaway vectors; this loop replaces those vectors, span by span, with the features of the actual reference photos. After it runs, condition_embeds is one sequence where some positions are text and some positions are "here is exactly what this person's face looks like, at this patch." Then time embedding and the noisy latent are concatenated on, and the whole thing goes through the Phi-3 transformer backbone (Abdin et al., 2024) that OmniGen builds on. Text and image are processed by the same attention layers, in the same sequence. That shared attention is how a token of the prompt can look directly at a patch of the reference face.

One more thing: image tokens see each other fully

There is a detail in the collator that matters for identity. The base attention mask is causal (each token sees only earlier tokens), but for the image spans that is wrong; a patch in the top-left of a face should be able to see a patch in the bottom-right. So the collator opens up full attention inside each image block:

def adjust_attention_for_input_images(self, attention_mask, image_sizes):
    for b_inx in image_sizes.keys():
        for start_inx, end_inx in image_sizes[b_inx]:
            attention_mask[b_inx][start_inx:end_inx, start_inx:end_inx] = 1
    return attention_mask

Inside a reference image, every patch attends to every other patch. Between images and text, the causal order still holds. So each photo is read as a coherent whole, not as a left-to-right scan, which is what you want when the thing you are trying to preserve is a face.

Gotchas you will actually hit

A few things that bite in practice, all visible in the code above:

Tag your images, do not just pass them. If your instruction does not contain <|image_1|>, the assertion path is skipped and the photos are never spliced in. Passing images to the pipeline is not enough; the tag is the anchor.
Number from 1, no gaps. [1, 2, 3], never [0, 1] or [1, 3]. The assertion is strict on purpose.
Reference resolution is sequence length. Each 1024 image is 4096 tokens before your prompt and the output canvas are even counted. If you are running out of memory with three references, that is why. Smaller reference crops cost fewer tokens.
Order is semantic. Because images are reordered to tag order, "<|image_1|> wearing the hat from <|image_2|>" is not the same prompt as swapping the two tags. Be deliberate about which photo is which number.

Takeaway

Identity preservation across generations is not magic in the sampler. It is bookkeeping in the processor: split the prompt at image tags, tokenize the text between, reserve H*W//16//16 placeholder slots per image, record the span of each, then overwrite those spans with real image features so text and face share one attention stream. The model can only keep the same person if the same person's pixels are sitting in the sequence, in known positions, with full attention inside each photo.

The code is in visual_generation/generation/omnigen/processor.py and model.py in the Wunjo Make repo. If your reference-based results drift, start by printing image_sizes and confirming the spans line up with the images you think you passed.

References

Xiao, Wu, Yang, et al. "OmniGen: Unified Image Generation." 2024. arXiv:2409.11340
Abdin et al. "Phi-3 Technical Report." 2024. arXiv:2404.14219

Picking video keyframes with scene cuts plus a fixed stride

Wlad Radchenko — Wed, 12 Aug 2026 12:48:00 +0000

You have a 90-second clip and you need to pick a handful of frames out of it. Not random frames. Frames that, between them, describe the whole clip well enough that a downstream model can fill in the rest. This shows up everywhere in video tooling: you stylize a few frames and warp the look across the gaps, you run an expensive diffusion pass on keyframes only, you build a thumbnail strip, you sample frames for a vision model. The whole job hangs on which frames you keep.

The lazy answer is "every tenth frame." It is one line and it is wrong in a specific, annoying way. The fancier answer is "frames where the shot cuts." That is better and still wrong, in the opposite direction. The function I want to walk through picks both, and the reason it picks both is the interesting part.

The code is SceneDetect.detecting in portable/src/visual_generation/inference.py in Wunjo Make. It is short. Around 30 lines do the whole thing.

Why "every Nth frame" is dumb

Say you take every tenth frame. The video has a hard cut at frame 47, the camera jumps from a face to a wide landscape. Your stride lands you on frames 40 and 50. Frame 40 is the face, frame 50 is the landscape, and the cut between them is invisible to you. Now whatever you do downstream has to bridge frame 40 to frame 50 as if they were the same continuous shot. They are not. A motion-based fill will try to warp a face into a mountain and produce mush. A thumbnail strip will skip the cut entirely and make two scenes look like one.

A fixed stride has no idea where the content actually changes. It samples the timeline, not the footage. On a slow, single-shot clip it oversamples, ten near-identical frames where two would do. On a fast-cut montage it straddles every cut and never lands cleanly on the start of a new shot. The stride is blind.

Why "scene cuts only" is too sparse

So detect the cuts. PySceneDetect does this well, and the function uses it:

from scenedetect import detect, ContentDetector

scene_list = detect(source_file, ContentDetector(threshold=threshold))

ContentDetector works on content difference between consecutive frames. It converts each frame to HSV, measures how much hue, saturation and value changed frame to frame, and when that change spikes past a threshold it calls a cut. There is no neural network here and no training. It is a hand-tuned heuristic, and that is fine, because shot boundaries are mostly a sudden jump in pixels and HSV captures that jump cheaply. The default threshold here is 50.0. Lower it and the detector gets twitchy, calling cuts on fast pans and lighting changes. Raise it and it misses soft cuts. Fifty is a sane middle for general footage.

Now you have a list of scenes. The problem: a scene can be ten seconds long. If you keep only the first frame of each scene, a ten-second continuous shot of someone walking across a room gives you one keyframe. Everything that happens inside that shot, the person crossing from left to right, is gone. For a thumbnail that might be fine. For anything that has to reconstruct the in-between frames, one keyframe per shot is far too little. The motion inside the shot has no anchors.

Scene cuts tell you where the content changes hard. They say nothing about the slower drift inside a shot.

Combining both

Here is the whole selection block:

if len(scene_list) > 0:
    for i, scene in enumerate(scene_list):
        start_frame = scene[0].get_frames()
        end_frame = scene[1].get_frames()
        if interval:
            scene_frames.extend(range(start_frame, end_frame, interval))
            scene_frames.append(end_frame - 1)
        else:
            scene_frames.extend([start_frame, end_frame - 1])
    else:
        scene_frames.append(total_frames - 1)
else:
    if interval:
        scene_frames = list(range(0, total_frames - 1, interval))
        scene_frames.append(total_frames - 1)
    else:
        scene_frames = [0, total_frames - 1]

Read it from the top. For each detected scene you have a start_frame and an end_frame. The key two lines are these:

scene_frames.extend(range(start_frame, end_frame, interval))
scene_frames.append(end_frame - 1)

The first line walks the scene from its start to its end in steps of interval (default 10), and adds every one of those frames. So inside the shot you get the start, then start+10, then start+20, and so on. That is the fixed stride, but now it is reset at the start of every scene. It cannot straddle a cut, because each scene gets its own fresh stride that begins exactly on the cut. The stride does the sampling inside the shot; scene detection makes sure the stride never crosses a boundary.

The second line appends end_frame - 1, the last frame of the scene. This matters because range(start, end, interval) rarely lands on the final frame. If a scene runs from frame 100 to frame 137 with a stride of 10, range gives you 100, 110, 120, 130, and stops. Frame 136, the last real frame before the cut, never gets picked. But that last frame is the most important one to keep, because it is the frame right before the content changes. It is what a downstream fill needs to hand off cleanly to the next shot. So the code explicitly adds it. end_frame - 1, not end_frame, because the scene's end_frame is exclusive: it is the first frame of the next scene, not the last frame of this one.

Put the two ideas together and you get the property you actually want. Keyframes are dense enough inside each shot to capture motion, and they always sit cleanly on shot boundaries, never across them. Scene detection gives the structure, the stride gives the density, and the explicit last-frame append seals each scene so nothing falls through the gap at the cut.

A keyframe per cut is too coarse; every Nth frame straddles the cuts. Striding inside each detected scene, plus the scene's last frame, gives both. Photo: Unsplash

The fallback when nothing is detected

There is a second branch, and skipping it is how you ship a crash. Some inputs produce no scenes at all. A single static shot, a very short clip, a screen recording with no real cuts. PySceneDetect returns an empty list, and if your code only handles the "scenes found" path, you get zero keyframes and a confusing downstream failure.

The else covers it:

else:
    if interval:
        scene_frames = list(range(0, total_frames - 1, interval))
        scene_frames.append(total_frames - 1)
    else:
        scene_frames = [0, total_frames - 1]

No scenes means the whole video is treated as one big scene. Stride across the entire thing, then append the true last frame. It degrades to plain "every Nth frame" exactly when there are no cuts to respect, which is the one case where plain stride is the right call. The combined approach and the dumb approach converge when the footage has no structure to detect.

Dedup before you read the disk

The stride and the last-frame append can collide. A short scene where end_frame - 1 happens to be a multiple of the stride from the start will have that frame in the list twice. Reading and writing the same frame twice is wasted I/O. One line handles it:

for i in sorted(set(scene_frames)):
    if i <= total_frames - 1:
        cap.set(cv2.CAP_PROP_POS_FRAMES, i)
        ret, frame = cap.read()
        if ret:
            file_path = os.path.join(local_save_dir, f"{i}.jpg")
            cv2.imwrite(file_path, frame)

sorted(set(scene_frames)) does two jobs in one expression. set removes the duplicates so each frame is read once. sorted puts them back in ascending order so the seek moves forward through the file instead of jumping around. Seeking forward is much cheaper than seeking backward with cv2.VideoCapture, so reading frames in order is not just tidy, it is faster.

The i <= total_frames - 1 guard is a seatbelt. total_frames from cv2.CAP_PROP_FRAME_COUNT is not always exact, some containers report a frame or two more than you can actually decode, so the guard keeps a bad count from asking for a frame that does not exist.

Gotchas you will actually hit

The frame numbers in the output are one-based. Look at how each kept frame is recorded: the saved entry uses "frame": i + 1. The file on disk is named by the zero-based index i (f"{i}.jpg"), but the metadata reports i + 1. If you consume this output and assume the frame field matches the .jpg filename, you are off by one. The filename is the seek index, the frame field is the human-facing count.

end_frame is exclusive. The single most common bug when you reimplement this is appending end_frame instead of end_frame - 1 and writing a frame that belongs to the next scene, or running off the end of the file. PySceneDetect's scene end is the boundary, not the last frame.

The threshold is footage-dependent. threshold=50.0 is a default, not a law. Animation, screen recordings, and high-contrast edited content cut differently from handheld video. If you get too many keyframes, the detector is over-firing; raise the threshold. Too few, lower it. The stride is the other dial: a wide interval saves frames but misses fast motion inside a shot, a narrow one captures motion but costs you frames downstream.

interval=0 is a real mode. Passing a falsy interval switches every branch to "endpoints only," start and last frame of each scene, or [0, total_frames - 1] for the no-scene case. That is the right setting when you only want shot boundaries and will handle the in-between frames yourself.

Wrap-up

Keyframe selection looks trivial until you try to use the keyframes for something. A fixed stride samples the clock and ignores the footage, so it straddles every cut. Scene detection finds the cuts but leaves long shots almost unsampled. The fix is to use scene detection for structure and a per-scene stride for density, then explicitly keep the last frame of each scene so the handoff at every cut is clean. Add a fallback that treats a cutless video as one scene, dedup with sorted(set(...)) so you read each frame once and in order, and guard against an over-reported frame count.

Thirty lines, no model weights, and the output is a frame list you can actually build on. The code is SceneDetect.detecting in visual_generation/inference.py in the Wunjo Make repo. If your own keyframe picker chokes on cuts, this is the shape to copy.

References

PySceneDetect, ContentDetector: detects shot changes from frame-to-frame HSV content difference. Library, no associated paper. https://www.scenedetect.com

Auto-highlights with a small VLM or how the model votes yes or no on every 10-second cut

Wlad Radchenko — Tue, 11 Aug 2026 11:42:00 +0000

You point a tool at a 40-minute gameplay video and ask for a one-minute highlight reel. What should it actually do?

The naive answer is "find the exciting parts." But a model has no idea what exciting means for your video until you tell it. A goal in football, a punchline in a podcast, and a jump scare in a let's-play are all highlights, and none of them look alike. So the real problem is two problems stacked on top of each other. First, figure out what counts as a highlight for this specific video. Then go through the video and decide, piece by piece, whether each piece is one of those.

That is exactly the shape of the code in portable/src/visual_generation/highlights/processing.py in Wunjo Make. It uses a small vision-language model called SmolVLM2 and, for the audio path, OpenAI's Whisper. What I like about it is that the "scoring" is not a regression head or a learned ranker. It is the model answering the word yes. Let me walk through how that works and where the sharp edges are.

The whole pipeline in one breath

Here is the visual path, top to bottom, from generate_visual_highlight_moments:

duration = self.get_video_duration_seconds(video_path)
description = self.process_video_analyser(video_path)
highlight_response = self.process_highlight(description, highlight_description)
self.cut_scenes(video_path, save_dir, segment_length)   # 10s chunks via ffmpeg
# ...shuffle segment indices, chunk them...
if self.process_video_segment(segment_path, highlight_response):
    kept_segments.append((start_time, end_time))

Four model calls, in order:

Describe the video. Ask the VLM what kind of video this is.
Invent the highlight menu. Feed that description back and ask for a list of dramatic moments that would make good highlights for this type of video.
Cut into segments. Use ffmpeg to split the file into 10-second clips. No model here.
Vote per segment. For each clip, show it to the VLM with the menu and ask: does this clip contain one of these? Keep the ones where the answer is yes.

The model wrapper, and the one line that parses the answer

class SmolVLM2:
    def __init__(self, model_path: str = "HuggingFaceTB/SmolVLM2-2.2B-Instruct", device: str = "cuda", download_root: str = None):
        self.processor = AutoProcessor.from_pretrained(model_path)
        self.model = AutoModelForImageTextToText.from_pretrained(
            model_path,
            torch_dtype=torch.bfloat16,
            device_map=self.device,
            _attn_implementation="flash_attention_2" if FLASH_ATTN_2_AVAILABLE else None
        )

SmolVLM2 is a small multimodal model, the 2.2B-Instruct checkpoint here. Small matters: this runs per segment in a loop, so a 70B model would be unusable on a desktop. It loads in bfloat16 and turns on FlashAttention 2 only if the import succeeded at the top of the file, otherwise it passes None and lets transformers fall back.

The generate method is where the interface to the rest of the pipeline lives:

def generate(self, messages: list, max_new_tokens: int = 64):
    inputs = self.processor.apply_chat_template(
        messages[:8192],
        add_generation_prompt=True,
        tokenize=True,
        return_dict=True,
        return_tensors="pt",
    ).to(self.model.device, dtype=torch.bfloat16)
    generated_ids = self.model.generate(**inputs, max_new_tokens=max_new_tokens, do_sample=True, temperature=0.7)
    generated_output = self.processor.decode(generated_ids[0], skip_special_tokens=True).lower().split("assistant: ")[1]
    return generated_output

Three things to notice here, because they all affect the highlight decision.

do_sample=True, temperature=0.7. The voting is not deterministic. The same clip with the same prompt can come back yes once and no the next run. For a highlight reel that is fine, even pleasant. You get variety. For a unit test it is a headache.

max_new_tokens=64 for the per-segment call. The model is not asked to write an essay, it is asked for a quick verdict. Sixty-four tokens is enough for "yes" plus a one-line justification and nothing more.

.lower().split("assistant: ")[1]. This is the cheap, fragile glue that turns chat output into a usable string. It lowercases everything (so the later "yes" in response check is case-insensitive) and keeps only the text after the literal assistant: marker. If the decoded output ever does not contain that exact marker, this line raises an IndexError. Worth knowing if you swap the checkpoint.

The model writes its own rubric

This is my favorite part. Before scoring anything, the code asks the model to enumerate what a highlight even is for this video. Look at the system messages:

SYSTEM_MESSAGES = {
    "highlight_editor": "You are a highlight editor. List archetypal dramatic moments that would make compelling highlights if they appear in the video. Each moment should be specific enough to be recognizable but generic enough to potentially exist in other videos of this type.",
    "highlight_assistant": "You are a helpful visual-language assistant ... Highlights should be rare and important events in the video in question.",
    "highlight_analyzer": "You are a video highlight analyzer. Your role is to identify moments that have high dramatic value ... Be categorical and choose only the brightest moments. ..."
}

And the prompt that builds the menu:

@staticmethod
def prompt_highlight(description: str, highlight_description: str = None, behaviour: int = None):
    prompts = {
        1: "List potential highlight moments to look for in this video:",
        2: "List dramatic moments that would make compelling highlights if they appear in the video. ...:"
    }
    behaviour = 1 if highlight_description is None and behaviour is None else 2
    highlight_description = prompts[behaviour] if highlight_description is None else highlight_description
    return [
        {"role": "system", "content": [{"type": "text", "text": SYSTEM_MESSAGES["highlight_editor" if behaviour == 1 else "highlight_assistant"]}]},
        {"role": "user", "content": [{"type": "text", "text": f"""Here is a description of a video:\n\n{description}\n\n{highlight_description}"""}]}
    ]

The behaviour switch decides the personality. If the caller passed nothing, behaviour is 1 and the model plays "highlight editor", listing archetypal moments. If the caller passed their own highlight_description (say, "show me every time someone laughs"), behaviour becomes 2 and the system role switches to "highlight assistant". The wording in those two system prompts is doing real work: "specific enough to be recognizable but generic enough to potentially exist in other videos of this type" is what keeps the menu from being too tied to one exact frame.

There is one more wrinkle in process_highlight:

def process_highlight(self, description: str, highlight_description: str = None) -> str:
    if highlight_description is not None:
        highlight_description = f"List potential {highlight_description.lower()} to look for in this video:"
    highlight_response_messages = self.prompt_highlight(description, highlight_description)
    highlight_response = self.generate(highlight_response_messages, max_new_tokens=256)
    return highlight_response

When you give your own description, it is wrapped into "List potential {your text} to look for in this video:". So a user request like "funny reactions" becomes "List potential funny reactions to look for in this video:". The output of this call, up to 256 tokens, is the rubric. It is plain text. It gets pasted verbatim into every per-segment prompt later.

The yes/no vote, and why scoring is just `"yes" in response`

Here is the prompt that scores one clip:

@staticmethod
def prompt_video_segment(video_segment_path: str, highlight_types: str):
    return [
        {"role": "system", "content": [{"type": "text", "text": SYSTEM_MESSAGES["highlight_analyzer"]}]},
        {"role": "user", "content": [
            {"type": "video", "path": video_segment_path},
            {"type": "text", "text": f"""Given these highlight examples:\n{highlight_types}\n\nDoes this video contain a moment that matches the core action of one of the highlights? Answer with:\n'yes' or 'no'\nIf yes, justify it"""}]}
    ]

And the scoring:

def process_video_segment(self, video_segment_path: str, highlight_types: str) -> bool:
    messages = self.prompt_video_segment(video_segment_path, highlight_types)
    response = self.generate(messages, max_new_tokens=64)
    print(f"Segment response {response}")
    return "yes" in response

That is the entire scorer. No score between 0 and 1, no ranking, no threshold you tune. The clip is shown to the model alongside the rubric, the model is told to answer yes or no and justify a yes, and the code does a substring check for yes on the lowercased text. It is a vote, not a rank.

This is a real design choice with real consequences. The upside: it is dead simple and it leans on the model's own judgment, which is the whole point of using a VLM. The downside is the substring check. The word "yes" appearing anywhere in 64 tokens flips the clip to "keep". A justification like "there is no clear yes-or-no moment here" contains yes and would be kept. In practice the instruction to lead with yes or no makes this rare, but it is the failure mode to watch if you fork this.

Stopping early: shuffle, chunk, budget

You do not want to score all 240 clips of a 40-minute video. You want roughly one minute of highlights and then stop. Here is how the loop manages that:

duration_limit_sec = duration_limit_min * 60
duration_highlight = 0

indices = list(range(len(segments_path)))
random.shuffle(indices)

chunk_size = 10
chunks = [indices[i:i + chunk_size] for i in range(0, len(indices), chunk_size)]

for num, chunk in enumerate(chunks):
    for i in chunk:
        segment_path = segments_path[i]
        start_time = float(i * segment_length)
        end_time = min(float(i * segment_length + segment_length), duration)
        if duration_highlight + segment_length > duration_limit_sec:
            break
        if self.process_video_segment(segment_path, highlight_response):
            kept_segments.append((start_time, end_time))
            duration_highlight += segment_length
    else:
        continue   # limit not reached, process next chunk
    break          # limit reached, stop

Three deliberate moves here.

The indices are shuffled before scoring. If you scanned in order and your budget filled up after the first six clips, your highlight reel would be the first sixty seconds of the video, every time. Shuffling spreads the sampled clips across the whole timeline, so a one-minute reel can pull a moment from the start, the middle, and the end.

The duration_highlight counter is the budget. Every kept clip adds segment_length (ten seconds). The check if duration_highlight + segment_length > duration_limit_sec: break stops adding once the next clip would overflow the requested length. So duration_limit_min=1 gives you up to six ten-second clips.

The for/else/break is the early exit. This is one of Python's least-used features. The else on a for loop runs only if the loop finished without hitting break. So: if a chunk completes without filling the budget, else: continue moves to the next chunk. If the budget filled mid-chunk and we breaked out of the inner loop, the else is skipped, and the outer break ends everything. Chunking by ten just bounds how often the budget is checked.

One last line that matters:

kept_segments.sort(key=lambda x: x[0])

We scored in shuffled order, so the kept clips are out of order. This sorts them back by start time, so the reel plays chronologically. Without it your highlights would jump around in time.

The audio twin: Whisper instead of frames

generate_text_highlight_moments is the same machine with a different sensor. Instead of looking at frames, it reads what was said. The transcription comes from the Whisper class:

class Whisper:
    def __init__(self, download_root: str, model_id: str = "large-v3-turbo", device: str = "cuda"):
        self.model = whisper.load_model(model_id, device=device, download_root=download_root)

    def __call__(self, audio_path, language=None, task="transcribe"):
        audio = self.load_audio(audio_path)
        output = whisper.transcribe(self.model, audio, detect_disfluencies=True, vad=True, language=language, task=task)
        return output

It uses large-v3-turbo with vad=True (voice activity detection, so silence is dropped) and detect_disfluencies=True (it marks ums and hesitations). The transcript gets grouped into roughly 30-second blocks by get_timestamp, each block carrying start, end, and text. Those blocks are the segments now. The scorer is process_text_segment, which asks the same yes/no question about a chunk of transcript text, and the same "yes" in response vote decides it. Same shuffle, same budget, same early exit, same final sort.

There is also a small cleanup helper worth a mention, because small VLMs love to repeat themselves:

@staticmethod
def strip_repeats(text, min_words=2, max_words=6):
    pattern = re.compile(r'\b((?:\w+\s*){' + str(min_words) + ',' + str(max_words) + r'})\1+', flags=re.IGNORECASE)
    return pattern.sub(r'\1', text)

It collapses a phrase of 2 to 6 words that repeats back-to-back into a single copy. The text path runs the description through this before re-summarizing. It is a band-aid for a known habit of small models, and it is honest about being one.

A highlight reel is just the clips that survived the vote, stitched back in time order. Photo: Unsplash

Stitching the survivors

Once you have kept_segments, the cut is plain ffmpeg. concatenate_scenes builds a filter_complex that trims and concatenates each kept range:

for i, (start_sec, end_sec) in enumerate(scene_times):
    filter_complex_parts.append(f"[0:v]trim=start={start_sec}:end={end_sec},setpts=PTS-STARTPTS[v{i}];")
    filter_complex_parts.append(f"[0:a]atrim=start={start_sec}:end={end_sec},asetpts=PTS-STARTPTS[a{i}];")
    concat_inputs.append(f"[v{i}][a{i}]")

concat_filter = f"{''.join(concat_inputs)}concat=n={len(scene_times)}:v=1:a=1[outv][outa]"

setpts=PTS-STARTPTS resets the timestamps on each trimmed piece so they line up at zero before concatenation. Audio gets the same treatment with asetpts. The output is re-encoded to H.264 with AAC audio. There is a separate concatenate_audio for audio-only output. Both honor a DEBUG env var: when DEBUG=True the command runs through os.system so you see ffmpeg's output, otherwise it runs silently through subprocess.run.

Gotchas if you run this yourself

A few things that will bite you, all visible in the code:

Sampling makes it non-reproducible. do_sample=True, temperature=0.7 means two runs on the same video give different reels. Set do_sample=False in generate if you need repeatability while debugging.
The "yes" in response substring is greedy. Any clip whose 64-token answer contains the letters yes is kept. If you change the checkpoint or the prompt, check that the model still leads with a clean yes/no.
The split("assistant: ")[1] will throw on the wrong format. A different chat template that does not emit that exact lowercase marker breaks generate with an IndexError. This is the first thing to check when swapping models.
cut_scenes re-reads the directory with os.listdir. Segments are matched by index against segments_path, and start_time is computed as i * segment_length. That assumes ffmpeg produced evenly sized chunks in sorted order. The last chunk is shorter, which is why end_time is clamped with min(..., duration).
The budget is coarse. It counts in whole segment_length units (10s), not in actual kept seconds, so a one-minute request gives "up to six clips", not "exactly sixty seconds".

That is the whole selection engine. No trained ranker, no score threshold, just a small model that writes its own rubric, then votes yes or no on each ten-second slice while a budget and a shuffle keep the reel short and spread out. The simplicity is the feature. You can read every decision the system makes, because every decision is a word the model said.

References

Marafioti et al. "SmolVLM: Redefining Small and Efficient Multimodal Models." 2025. arXiv:2504.05299. https://arxiv.org/abs/2504.05299
Radford, Kim, Xu, Brockman, McLeavey, Sutskever. "Robust Speech Recognition via Large-Scale Weak Supervision." 2022. arXiv:2212.04356 (Whisper). https://arxiv.org/abs/2212.04356
Source file: portable/src/visual_generation/highlights/processing.py in Wunjo Make.

Growing an image past its border without a visible seam

Wlad Radchenko — Wed, 05 Aug 2026 13:38:00 +0000

You have a photo and you want more of it. More sky above the head, more room to the left of the subject, a wider frame than the camera gave you. Outpainting is the name for inventing those new pixels. The model part is well covered everywhere. The part nobody talks about is the few dozen lines that run before the model, and that part is where the seam is won or lost.

If you get the prep wrong, the new region meets the old photo at a hard line. The colors step. The texture changes mid-wall. You can point at exactly where the real photo ended. This post is a walkthrough of the prep code in Wunjo Make that stops that from happening. It lives in one file, visual_generation/generation/outpaint.py, and it is all NumPy and OpenCV. No network in sight.

Outpainting is asking for canvas that was never photographed. The trick is making the new canvas agree with the old one at the join. Photo: Unsplash.

The shape of the problem

Say you want to extend an image to the left. You need a bigger canvas, the old pixels parked on the right side of it, and an empty strip on the left for the new content. Three things have to be true at the end:

The known pixels survive untouched in the middle and far side.
The model is told, precisely, which strip is allowed to change.
The old edge does not become a wall the model paints up against.

The whole function is built around those three. Here are the defaults it runs with:

def process_image(image, fill_color=(0, 0, 0), mask_offset=50, blur_radius=500,
                  expand_pixels=256, direction="left",
                  inpaint_mask_color=50, max_size=1024):

expand_pixels=256 is how much new canvas you grow. mask_offset=50 is the overlap that hides the seam, and it is the most important number in the file. blur_radius=500 feathers the mask edge. max_size=1024 is the hard cap on the output dimension. Keep those four in mind.

Step 1. Size the new canvas, and crop if it would blow the cap

First it computes the new size by adding expand_pixels on the axis you are growing:

new_height = height + (expand_pixels if direction in ["top", "bottom"] else 0)
new_width  = width  + (expand_pixels if direction in ["left", "right"] else 0)

Then the part people forget. If growing the image would push it past max_size, it does not let the canvas grow without limit. It crops the same number of pixels off the opposite side first, so the output stays at max_size:

if new_width > max_size:
    if direction == "left":
        image = image[:, :max_size]            # drop the far-right columns
    elif direction == "right":
        image = image[:, new_width - max_size:] # drop the far-left columns
    new_width = max_size

This is a practical decision, not a cosmetic one. The model downstream has a size it is happy with. Letting the canvas creep past 1024 on every pass would either run out of memory or force a resize that softens the whole image. So the canvas slides: you gain 256 new pixels on one edge and lose the equivalent off the other, holding the frame at a fixed size.

Step 2. Three buffers

After the size is settled, it builds three arrays the same size as the new canvas:

new_image    = np.full((new_height, new_width, 3), fill_color, dtype=np.uint8)  # the canvas
mask         = np.full_like(new_image, 255, dtype=np.uint8)  # soft guidance mask
inpaint_mask = np.full_like(new_image, 0,   dtype=np.uint8)  # hard fill region

mask         = cv2.cvtColor(mask, cv2.COLOR_BGR2GRAY)
inpaint_mask = cv2.cvtColor(inpaint_mask, cv2.COLOR_BGR2GRAY)

Two different masks, and the difference matters.

mask is the soft one that the generation step uses. It starts all white (255), and white here means "keep this." inpaint_mask is the hard one, a quick scratch buffer used a moment later to pre-fill the empty strip. It starts all black (0).

Step 3. Place the pixels and carve the strips

This is the heart of it. For the left direction:

new_image[:, expand_pixels:] = image[:, : max_size - expand_pixels]
mask[:, : expand_pixels + mask_offset] = inpaint_mask_color   # 50
inpaint_mask[:, :expand_pixels] = 255

Line one drops the real pixels into the canvas, shifted right by expand_pixels so the left strip is left empty for the new content.

Line three marks the hard fill region. Exactly the expand_pixels columns on the left, set to 255. That is the genuinely empty part, the 256 columns that were never photographed.

Line two is the trick. It paints the soft mask to a low value (inpaint_mask_color, which is 50) across expand_pixels + mask_offset columns. Not 256. 306. It reaches 50 pixels past the new region, into the real photo.

That 50-pixel overlap is the whole game. If the editable region stopped exactly at the old border, the model would have to match the real photo perfectly at a single column, and it never will, so you would see a line. By letting the model also repaint a thin band of real pixels right at the join, you give it room to ramp the new content into the old. It blends across the edge instead of butting up against it.

Note the mask value is 50, not 0. The region is not marked "fully replace" but "mostly free," a gray instruction rather than a hard switch. Softer guidance near the seam, which is exactly what you want there.

Step 4. Father the mask edge

A sharp transition in the mask still prints a sharp transition in the result. So the mask gets a big Gaussian blur:

if blur_radius % 2 == 0:
    blur_radius += 1                 # OpenCV needs an odd kernel
mask = cv2.GaussianBlur(mask, (blur_radius, blur_radius), 0)

blur_radius=500 becomes a 501x501 kernel. That is enormous, and on purpose. It smears the boundary between the kept region and the editable region across hundreds of pixels, so there is no single row or column where guidance flips. The model gets a gradient of freedom, most free in the new strip, gently constrained as it moves back into real territory. The odd-number guard is there because OpenCV rejects an even kernel size.

Step 5. Pre-fill the empty strip with Telea

Here is the last and most underrated step. The empty strip is still flat fill_color (black, by default). Handing a model a black bar to start from is a bad init. So before returning, it fills that strip with a cheap classical inpaint:

_, mask_np = cv2.threshold(inpaint_mask, 128, 255, cv2.THRESH_BINARY | cv2.THRESH_OTSU)
inpaint = cv2.inpaint(new_image, mask_np, 3, cv2.INPAINT_TELEA)

cv2.inpaint with INPAINT_TELEA (the Fast Marching Method) walks inward from the border of the hole and paints each new pixel from its neighbors. The 3 is the inpaint radius, how far it looks for those neighbors. It is not smart. It will not invent a tree or a window. What it does is bleed the edge colors of the real photo into the empty strip, so the strip starts as a soft smear of the right colors instead of a black void.

Why bother, if a generator is about to repaint it anyway? Because the generator does better work when it starts from a plausible color field than from flat black. The smear carries the lighting and palette of the old photo into the new region before generation even begins. The black-bar init pulls the result toward dark, muddy borders. The Telea smear pulls it toward the photo it is extending.

The function returns two things: inpaint, the canvas with real pixels plus a softly pre-filled strip, and mask, the feathered soft mask. That pair is the init image and the mask you hand to an inpainting or diffusion model. A conditioning setup like this is the same idea behind ControlNet (Zhang, Rao, Agrawala, ICCV 2023): you do not just ask the model to generate, you hand it a structured starting point and a map of where it is allowed to work.

Doing it on both sides: the centering pass

decode_image is the wrapper that turns a single landscape-or-portrait image into a fixed canvas (default 576x1024). It figures out which axis is short, then calls process_image for both sides of that axis and stitches them:

expand_pixels_half_x = int((max_width - width) / 2)
expand_pixels_half_y = int((max_height - height) / 2)

if expand_pixels_half_x > 0:
    expand_pixels_half = expand_pixels_half_x
    directions = ["left", "right"]
else:
    expand_pixels_half = expand_pixels_half_y
    directions = ["top", "bottom"]

Splitting the needed growth in half and growing both edges keeps the original subject centered, instead of shoving it to one side. merge_images then pastes the two halves back together into one canvas.

The wrapper also returns its own combined binary mask, and it pulls in by padding=20 on the long axis:

mask_combined[pad_y_start:pad_y_end, pad_x_start:pad_x_end] = 0

That 20-pixel inset is the same instinct as the 50-pixel offset upstream: do not trust the boundary exactly, let the editable zone eat slightly into the known region so the join has somewhere to blend.

Gotchas you will actually hit

A few things that bit me, in case you build something similar.

The mask polarity is easy to get backwards. In this code white (255) means keep and the low value (50) means free to change. If your downstream model expects the opposite, you will outpaint the wrong region and destroy the photo. Check the convention before anything else.

The 50-pixel overlap is tunable, but do not drop it to zero. Zero overlap brings the hard seam right back. Too large, and the model starts rewriting parts of the photo you wanted kept. Fifty pixels on a 1024 canvas is a reasonable middle.

The Gaussian kernel must be odd. The code guards for this with the % 2 check, and if you change blur_radius, keep that guard or OpenCV will throw.

The opposite-side crop in step 1 is silent. If you feed in an image that is already near max_size and keep extending, you are quietly losing content off the far edge each pass. That is correct for a fixed-canvas product, but surprising if you expected pure growth.

Takeaway

The model invents the new pixels. Everything that makes them join cleanly happens in classical code before the model runs: pad the canvas, place the known pixels, mark the empty strip, overlap the editable mask 50 pixels into the real photo so it can blend across the old edge, feather that mask with a 501-pixel blur, and pre-fill the empty strip with a Telea smear so the model starts from the right colors instead of black. No generation in any of it.

The full file is visual_generation/generation/outpaint.py in the Wunjo Make repo. If you are fighting a visible seam in your own outpainting, the mask_offset overlap and the Telea pre-fill are the two places I would look first.

References

Zhang, Rao, Agrawala. "Adding Conditional Control to Text-to-Image Diffusion Models." ICCV 2023. arXiv:2302.05543
Telea. "An Image Inpainting Technique Based on the Fast Marching Method." Journal of Graphics Tools, 2004. (The INPAINT_TELEA method in OpenCV.)

The file is called encryption.py. It is actually an invisible watermark!

Wlad Radchenko — Tue, 04 Aug 2026 11:37:00 +0000

There is a file in my app called encryption.py. The class is EncryptionEncoder. The methods on the video manager are encrypted() and decrypted(). Every label says crypto.

It is not crypto. There is no cipher in that file, no key, no AES, no Fernet. What it actually does is hide a short text string inside the pixels of a frame so you cannot see it but a decoder can read it back. That is a watermark. The naming is just unfortunate, and I am going to use the honest word for the rest of this post.

I want to walk through the code, because the mechanism is small and clever, and then be straight about what a thing like this can and cannot do when it ships inside a fully local app that the user owns.

The name on the file says encryption. The code says watermark. Photo: Unsplash.

What problem this solves

The app generates synthetic media. Face swaps, lip-sync, retargeted portraits. At some point you want to answer one question about a file: did this come out of my tool, or is it an untouched original?

The honest way to answer that without a visible logo is to embed a marker the eye does not notice and a program can find. So when the app finishes a video, it writes a hidden string into a corner of each frame. Later, an analysis step reads the corners and checks for that string.

Here is the read side, from inference.py:

is_original = VideoManipulation.decrypted(source_file)
if is_original:
    lprint(msg="Content is original.", ...)
else:
    lprint(msg="Content is Wunjo fake.", ...)

So the whole point is a yes/no flag: is this our synthetic output. The marker string in the code is literally name="fake". That is the payload it stamps and the payload it looks for.

How the marker gets into a frame

The payload is just bytes. EncryptionEncoder turns a UTF-8 string into a flat list of bits:

seq = np.array([n for n in content], dtype=np.uint8)
self._encryptions = list(np.unpackbits(seq))   # e.g. b"fake" -> 32 bits
self._wmLen = len(self._encryptions)

Four ASCII characters give 32 bits. That 32 matters later, the decoder is hard-coded to expect 32-bit payloads.

The default embedding method in the app is dwtDctSvd. Three transforms stacked, and each one earns its place. Walk it from the outside in.

Step one, color space. Convert the patch from BGR to YUV and only touch the Y (luma) and U channels:

yuv = cv2.cvtColor(bgr, cv2.COLOR_BGR2YUV)
for channel in range(2):   # only Y and U
    ...

Luma and chroma are where you can nudge values without the change jumping out at a viewer. The V channel is left alone.

Step two, wavelet. Run a single-level Haar dwt2 on the channel. This splits the image into a coarse approximation ca1 and three detail bands:

ca1, (h1, v1, d1) = pywt.dwt2(yuv[:row//4*4, :col//4*4, channel], 'haar')

Everything gets embedded into ca1, the coarse band. Coarse coefficients survive compression and resizing far better than fine detail, which is the first reason this watermark is hard to wipe by accident.

Step three, the actual bit. Chop ca1 into 4x4 blocks. For each block, take a DCT, then an SVD, and quantize the top singular value:

def diffuse_dct_svd(self, block, wmBit, scale):
    u, s, v = np.linalg.svd(cv2.dct(block))
    s[0] = (s[0] // scale + 0.25 + 0.5 * wmBit) * scale
    return cv2.idct(np.dot(u, np.dot(np.diag(s), v)))

That one line is the trick. Read it slowly.

scale is 36. s[0] is the largest singular value of the block, the bulk of its energy. s[0] // scale snaps it down to a multiple of 36. Then you add back a fixed offset that depends only on the bit you want to store: 0.25 * scale for a 0, 0.75 * scale for a 1. So after this, every block sits at either "a quarter of the way into its 36-wide bin" or "three quarters of the way in," and which one it is encodes the bit.

Decoding just asks where in the bin the value landed:

score = int((s[0] % scale) > scale * 0.5)

Below the halfway mark reads 0, above reads 1. The 0.25 and 0.75 placement is deliberate, it parks each bit as far from the decision boundary as possible so JPEG noise has to be large to flip it.

The part that makes it survive: repeat and average

A 32-bit payload does not need many blocks. A 256x256 patch gives you hundreds of 4x4 blocks. So the encoder writes the same 32 bits over and over:

wmBit = self._encryptions[(num % self._wmLen)]   # cycle through the 32 bits

Block 0 holds bit 0, block 32 holds bit 0 again, and so on. Every bit is stamped dozens of times across the patch.

The decoder collects all the readings for each bit position and takes the mean before thresholding:

scores = [[] for i in range(self._wmLen)]
# ... append a score per block to scores[num % wmLen] ...
avgScores = list(map(lambda l: np.array(l).mean(), scores))
bits = (np.array(avgScores) * 255 > 127)

This is the cheap, effective version of error correction. Compression might flip a handful of blocks, but the average over a hundred copies of the same bit barely moves. That is why the marker survives a save-and-reopen.

Where it gets written, and the skip check

The app does not watermark the whole frame. It picks a 256x256 box, usually a corner or the center, and stamps only that:

positions = [
    (0, box_size, 0, box_size),                       # top-left
    (0, box_size, int(width - box_size), width),      # top-right
    (int(height - box_size), height, 0, box_size),    # bottom-left
    (int(height - box_size), height, int(width - box_size), width),
    # ...center, plus some (None, None, None, None) no-op slots
]
y1, y2, x1, x2 = random.sample(positions, 1)[0]

Note the None slots. On video, some frames get no watermark at all, picked at random. That spreads the marker across frames instead of branding every single one, which keeps any per-frame artifact from accumulating into something visible.

Before stamping, the writer decodes first and skips if the marker is already there:

decoded_text = decoder.decode(bgr, 'dwtDctSvd').decode('utf-8')
if decoded_text != name and width >= box_size and height >= box_size:
    # encode

So re-running the tool on its own output does not double-stamp. And the 256x256 minimum is enforced in the encoder too, it raises if r*c < 256*256. Below that you do not have enough blocks for the repeat-and-average to be reliable.

On the read side, decrypted() checks the four corners of each frame and returns False (not original) the moment any corner decodes to the marker. One hit is enough.

Gotchas you will actually hit

A few things bit me, and will bit you if you reuse this code:

The payload length is wired to 32 bits. EncryptionDecoder('bytes', 32) everywhere. Your marker string has to be exactly four bytes, or you change the length in both encoder and decoder. set_by_ipv4 even asserts self._wmLen == 32.
Lossless transcode matters. The video is reassembled with ffmpeg ... -c:v copy, a stream copy, no re-encode. If you instead re-encode at a low bitrate, you are testing how much the watermark survives, and the answer depends on scale. A bigger scale survives harder compression but distorts the patch more.
The rivaGan method needs ONNX models. The third method loads encoder.onnx and decoder.onnx from the module directory and even runs chmod/icacls on them to fix file permissions. The app does not use this path by default; dwtDctSvd is what videoio.py calls.
There is dead code. infer_dct_svd has a return score followed by an unreachable if score >= 0.5. Do not be confused by it, the function returns the int and exits.

The honest part: this is obfuscation, not protection

Now the uncomfortable bit. This ships inside a fully local desktop app. Everything is on the user's machine: the encoder, the decoder, and the marker string fake, all in plain Python in an open-source repo.

So be clear about what that means:

There is no secret. Anyone who reads the repo knows the algorithm, the scale, and the payload. They can read the marker, strip it, or forge it onto an untouched file.
It is not DRM. It does not stop anyone from using, copying, or editing the output. It does not lock anything.
It is not a cipher, despite the filename. A cipher needs a key the attacker does not have. A local app has no place to hide a key from its own user, so this scheme does not try.

What it actually buys you is provenance under cooperative conditions. If a file passes through normal handling, a re-save, a crop that leaves a corner intact, a platform transcode, the marker is likely still readable, and the app can honestly say "this came out of me." That is genuinely useful for a tool that generates synthetic media and wants to flag its own output. It is a fragile-but-invisible label, not a lock.

The threat it addresses is accidental loss of provenance, not a motivated adversary. Knowing which threat you are defending against is the whole game. Calling it encryption oversells it. Calling it a watermark, and being clear that the watermark is removable by anyone who cares, is the truth.

Wrap-up

The mechanism is one quantized singular value per 4x4 block, the same 32 bits repeated across a 256x256 patch, decoded by averaging. Small, hard to wipe by ordinary handling, and completely open. If you want to read the whole thing, it is portable/src/visual_processing/utils/encryption.py in the repo, with the call site in videoio.py.

If you take one thing away: name your modules for what they do. A watermark called encryption.py will make every future reader, including you, reason about a threat model that the code never had.

LivePortrait without the puppet look: stitching and retargeting in a 126-number MLP

Wlad Radchenko — Wed, 29 Jul 2026 12:36:00 +0000

You animate a still portrait with LivePortrait. The face moves, the mouth talks, the eyes blink. And then you notice it: the head floats. It drifts a little off the neck, the jaw slides past the shoulders, and the whole thing reads like a paper mask on a stick. The puppet look.

I hit this in Wunjo Make while wiring up the portrait animation path. The fix is not a bigger generator or a fancier warp. It is three small multi-layer perceptrons that predict tiny corrections to the face keypoints. One keeps the head attached to the body. One controls the eyes. One controls the lips. They are so small you could miss them in the repo, and they are the reason the output stops looking like a puppet.

This post walks through what those networks take in, what they output, and how a handful of numbers gets added back to the keypoints to fix the artifact. All of it from the actual code in the repo.

The input is one still photo. Everything after is keypoints and deltas. Photo: Unsplash

The one network class, used three ways

Here is the whole network definition. It lives in portable/src/visual_processing/portrait_animation/modules/stitching_retargeting_network.py:

class StitchingRetargetingNetwork(nn.Module):
    def __init__(self, input_size, hidden_sizes, output_size):
        super(StitchingRetargetingNetwork, self).__init__()
        layers = []
        for i in range(len(hidden_sizes)):
            if i == 0:
                layers.append(nn.Linear(input_size, hidden_sizes[i]))
            else:
                layers.append(nn.Linear(hidden_sizes[i - 1], hidden_sizes[i]))
            layers.append(nn.ReLU(inplace=True))
        layers.append(nn.Linear(hidden_sizes[-1], output_size))
        self.mlp = nn.Sequential(*layers)

    def forward(self, x):
        return self.mlp(x)

That is it. Linear, ReLU, Linear, ReLU, Linear. No convolutions, no attention, no recurrence. A plain stack of fully connected layers. The interesting part is that this same class is instantiated three times with different sizes, and each instance solves a different problem.

The sizes come straight from the model config in live_portrait_wrapper.py:

'stitching_retargeting_module_params': {
    'stitching': {
        'input_size': 126, 'hidden_sizes': [128, 128, 64], 'output_size': 65
    },
    'lip': {
        'input_size': 65, 'hidden_sizes': [128, 128, 64], 'output_size': 63
    },
    'eye': {
        'input_size': 66, 'hidden_sizes': [256, 256, 128, 128, 64], 'output_size': 63
    }
}

Those numbers are the whole story. To read them you need one fact: LivePortrait represents a face as 21 implicit 3D keypoints. The config sets num_kp: 21. Twenty one points, each with an x, y, z. So one full set of keypoints is 21 * 3 = 63 numbers. Hold onto 63 and 21, because every size below is built from them.

Stitching: keep the head on the shoulders

Think about what causes the floating head. The animation pipeline takes the source keypoints (the still photo) and the driving keypoints (the motion you want), warps the source feature volume from one to the other, and decodes an image. The warp is good at the face. It does not know or care that the face has to connect to a neck and shoulders that are not moving. So the face can shift as a block and detach from the body.

Stitching is a learned correction that nudges the driving keypoints so the face lines up with where the body already is. Plain version: before we render, we ask a small network "given where the face started and where you want it to go, what small shift puts it back on the shoulders?" Then we add that shift.

The input is 126 numbers. That is 63 for the source keypoints plus 63 for the driving keypoints, flattened and concatenated. Here is the concat, from utils/helper.py:

def concat_feat(kp_source: torch.Tensor, kp_driving: torch.Tensor) -> torch.Tensor:
    # kp_source: (bs, k, 3), kp_driving: (bs, k, 3) -> Return: (bs, 2k*3)
    feat = torch.cat([kp_source.view(bs_src, -1), kp_driving.view(bs_dri, -1)], dim=1)
    return feat

So the stitching MLP sees both poses at once: where the head is anchored, and where the animation wants it. Now the output, 65 numbers. Not 63. The extra two matter. Here is how the wrapper unpacks them in stitching():

def stitching(self, kp_source, kp_driving):
    bs, num_kp = kp_source.shape[:2]
    kp_driving_new = kp_driving.clone()
    delta = self.stitch(kp_source, kp_driving_new)

    delta_exp = delta[..., :3*num_kp].reshape(bs, num_kp, 3)     # 63 -> (21, 3)
    delta_tx_ty = delta[..., 3*num_kp:3*num_kp+2].reshape(bs, 1, 2)  # the last 2

    kp_driving_new += delta_exp
    kp_driving_new[..., :2] += delta_tx_ty
    return kp_driving_new

Read it slowly. The first 63 outputs are a per-keypoint 3D delta. Each of the 21 points gets its own small x, y, z correction. Those get added to every keypoint. The last two outputs are a single global translation in x and y, added to all keypoints at once. So the network can both reshape the face locally and slide the whole face as one piece to meet the shoulders. That global tx, ty is the literal "stop floating" knob.

The correction is small by design. Note that the network has an initialize_weights_to_zero method that zeros every weight and bias. Trained from a zero start, the natural resting state of the delta is "change nothing," and the network only learns to add a nudge where a nudge is needed. A correction network, not a generator.

Eye and lip retargeting: control the blink and the mouth

The other two instances solve a different problem. When you drive one person's face with another's, the eyes and lips do not always transfer cleanly. A person with small eyes driving a person with large eyes leaves the large eyes half open when they should be shut. Lips that should close stay slightly parted. The driving motion is in the right direction but the wrong amount.

Retargeting fixes the amount. The trick: instead of feeding the network a pose, you feed it a target. You tell it "I want the eyes this open" or "I want the lips this closed," and it returns the keypoint delta that gets you there.

That target is a ratio computed straight from facial landmarks, in utils/retargeting_utils.py:

def calculate_distance_ratio(lmk, idx1, idx2, idx3, idx4, eps=1e-6):
    return (np.linalg.norm(lmk[:, idx1] - lmk[:, idx2], axis=1, keepdims=True) /
            (np.linalg.norm(lmk[:, idx3] - lmk[:, idx4], axis=1, keepdims=True) + eps))

def calc_eye_close_ratio(lmk, target_eye_ratio=None):
    lefteye_close_ratio  = calculate_distance_ratio(lmk, 6, 18, 0, 12)
    righteye_close_ratio = calculate_distance_ratio(lmk, 30, 42, 24, 36)
    ...

def calc_lip_close_ratio(lmk):
    return calculate_distance_ratio(lmk, 90, 102, 48, 66)

A close ratio is vertical opening over horizontal width. For an eye: distance between the top and bottom lids divided by the eye's corner-to-corner width. A wide open eye has a high ratio, a shut eye a low one. Dividing by width makes it scale-free, so it works whether the face is big or small in the frame. The eps = 1e-6 is there so a degenerate landmark set never divides by zero.

Now the sizes click into place. The lip network takes input_size 65: that is 63 for the source keypoints plus a lip ratio of 2 numbers. From the wrapper:

def retarget_lip(self, kp_source, lip_close_ratio):
    # kp_source: BxNx3, lip_close_ratio: Bx2
    feat_lip = concat_feat(kp_source, lip_close_ratio)
    delta = self.stitching_retargeting_module['lip'](feat_lip)
    return delta.reshape(-1, kp_source.shape[1], 3)   # -> (B, 21, 3)

The eye network takes input_size 66: 63 keypoints plus an eye ratio of 3 numbers (left eye, right eye, and a target). Both output 63, a per-keypoint 3D delta that you add to the source keypoints. The eye network has the deeper stack, [256, 256, 128, 128, 64], because opening and closing eyes the right amount across identities is the harder regression. Same class, more layers.

The pattern is identical to stitching: feed in the current keypoints plus a small control signal, get back a keypoint delta. The only difference is what the control signal is. For stitching it is the driving pose. For retargeting it is a ratio you can dial yourself.

How it actually runs

These three calls compose. In the wrapper you can see the full chain when you control everything by hand:

eyes_delta = self.retarget_eye(x_s_user, combined_eye_ratio_tensor)
lip_delta  = self.retarget_lip(x_s_user, combined_lip_ratio_tensor)
x_d_new    = x_s_user + eyes_delta + lip_delta
x_d_new    = self.stitching(x_s_user, x_d_new)

Deltas just add. You take the source keypoints, add the eye correction, add the lip correction, then run stitching last so the final pose is glued back to the body after you have changed the eyes and lips. Order matters: stitch after retargeting, not before, so the global tx, ty accounts for everything you moved.

Every call runs under torch.no_grad(). These are inference-only correction heads. They are also cheap. A 126-input MLP with three hidden layers is a rounding error next to the appearance extractor and the dense motion network, which is why LivePortrait can offer this control without a real frame-rate cost.

Gotchas worth knowing

A few things I learned the slow way.

Stitching is on by default, retargeting is not. In the config, flag_stitching defaults to True and the eye and lip retargeting flags default to False. The comments in argument_config.py flag eye and lip retargeting as "not recommend to be True, WIP." Stitching is the safe, always-want-it correction. Eye and lip retargeting are an expressive extra that can introduce jitter, which is why in the Wunjo path the per-frame deltas get run through a Kalman filter before they are applied. If your blink flickers frame to frame, that smoothing is the lever.

Stitching does not load by default for animals. The same comment notes you want stitching off for large head movement or for animal sources. The networks were trained on the human face keypoint layout, so the corrections only make sense there.

The whole module is optional. Look at the wrapper init: if checkpoint_S is missing, self.stitching_retargeting_module = None, and stitching() returns the driving keypoints untouched. The pipeline still runs. It just runs with the floating head. That is the cleanest way to see what these tiny networks buy you: delete them and watch the head detach.

Wrap-up

The lesson I keep relearning: the artifact you cannot stand is often fixed by the smallest part of the system. Here the floating head, the puppet look, the half-open eyes, all come down to three ReLU MLPs that predict a few keypoint deltas and add them back. 126 numbers in, 65 out for stitching. 65 in, 63 out for lips. 66 in, 63 out for eyes. Everything else in the pipeline is huge by comparison, and none of it would look right without these.

If you want to read the code yourself, start at stitching_retargeting_network.py for the class, then live_portrait_wrapper.py for stitching, retarget_eye, and retarget_lip. The full repo is open source on GitHub.

References

Guo, Zhang, Liu, et al. "LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control." 2024. arXiv:2407.03168. https://arxiv.org/abs/2407.03168

A safety gate inside a face-swap tool or how the NSFW check is wired into the pipeline

Wlad Radchenko — Tue, 28 Jul 2026 13:33:00 +0000

A face-swap tool can be pointed at the wrong content. That is not a hypothetical, it is the first thing some users try. So before the swap runs, there is a check: a small classifier looks at the target frame, and if it sees explicit content, the swap is skipped. The original frame goes through untouched.

This is a walkthrough of that check in an open-source repo, down to the labels and the number that decides. It is short code. The interesting part is where it sits in the pipeline and what it does when it fires.

The check sits in front of the swap, not after it. Nothing explicit ever gets a new face. Photo: Unsplash.

What the check actually is

The classifier is NudeNet, run as an ONNX model called filter.onnx. It is a single-stage object detector, the same family as YOLO: you give it an image, it gives you back boxes, each box with a class label and a confidence score. It has no academic paper, so I will describe it only from what the code does with it.

The model knows 18 labels. Here they are, straight from the file:

__labels = [
    "FEMALE_GENITALIA_COVERED", "FACE_FEMALE", "BUTTOCKS_EXPOSED",
    "FEMALE_BREAST_EXPOSED", "FEMALE_GENITALIA_EXPOSED", "MALE_BREAST_EXPOSED",
    "ANUS_EXPOSED", "FEET_EXPOSED", "BELLY_COVERED", "FEET_COVERED",
    "ARMPITS_COVERED", "ARMPITS_EXPOSED", "FACE_MALE", "BELLY_EXPOSED",
    "MALE_GENITALIA_EXPOSED", "ANUS_COVERED", "FEMALE_BREAST_COVERED",
    "BUTTOCKS_COVERED",
]

Notice the split: most labels come in COVERED and EXPOSED pairs. The model is not a single yes/no nudity switch. It localizes specific body parts and tells you whether each one is covered or exposed. That distinction is the whole basis of the gate, because the gate only cares about the exposed ones.

Preprocessing: letterbox to 320x320

The input side is plain object-detector plumbing, but it matters because the boxes have to map back to the original image. The model wants a square 320x320 input. An arbitrary photo is not square, so the image is resized to fit one dimension, then padded with black bars to fill the rest:

def _read_image(img, target_size=320):
    img_height, img_width = img.shape[:2]
    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
    aspect = img_width / img_height

    if img_height > img_width:
        new_height = target_size
        new_width = int(round(target_size * aspect))
    else:
        new_width = target_size
        new_height = int(round(target_size / aspect))

    resize_factor = math.sqrt(
        (img_width**2 + img_height**2) / (new_width**2 + new_height**2)
    )
    img = cv2.resize(img, (new_width, new_height))
    # ... pad with cv2.copyMakeBorder to reach target_size ...
    image_data = img.astype("float32") / 255.0
    image_data = np.transpose(image_data, (2, 0, 1))  # HWC -> CHW
    image_data = np.expand_dims(image_data, axis=0)   # add batch dim
    return image_data, resize_factor, pad_left, pad_top

The three values returned alongside the tensor, resize_factor, pad_left, and pad_top, are the undo instructions. The detector finds boxes in the padded 320x320 space, and these let postprocessing put them back where they belong in the real image. Pixels are scaled to 0..1 and the layout is flipped from height-width-channels to channels-first, which is what the ONNX graph expects.

Postprocessing: two thresholds, then NMS

The model outputs one row per candidate box. Each row is four box coordinates followed by the per-class scores. Postprocessing keeps a candidate only if its best class scores at least 0.2, then maps the box back to image space:

for i in range(rows):
    classes_scores = outputs[i][4:]
    max_score = np.amax(classes_scores)
    if max_score >= 0.2:
        class_id = np.argmax(classes_scores)
        x, y, w, h = outputs[i][0], outputs[i][1], outputs[i][2], outputs[i][3]
        left = int(round((x - w * 0.5 - pad_left) * resize_factor))
        top  = int(round((y - h * 0.5 - pad_top)  * resize_factor))
        # ... collect boxes, scores, class_ids ...

indices = cv2.dnn.NMSBoxes(boxes, scores, 0.25, 0.45)

That 0.2 is a loose first pass: keep almost anything the model is not sure is nothing. The pad_left and pad_top subtractions strip off the black bars, and multiplying by resize_factor scales the box back up to the original resolution. Then non-maximum suppression (NMSBoxes with a 0.25 score cutoff and 0.45 IoU) throws away overlapping duplicate boxes so each region is reported once.

The output of detect() is a list of dicts, each {"class", "score", "box"}. That is the raw material. The gate is built on top of it.

The gate itself: status()

This is the method the pipeline calls. It runs detection, keeps only the labels it cares about above a strict score, and returns a single boolean:

def status(self, img, classes=None):
    detections = self.detect(img)
    if classes is None:
        classes = [
            "BUTTOCKS_EXPOSED",
            "FEMALE_BREAST_EXPOSED",
            "FEMALE_GENITALIA_EXPOSED",
            #"MALE_BREAST_EXPOSED",
            "ANUS_EXPOSED",
            "MALE_GENITALIA_EXPOSED",
        ]
    detections = [d for d in detections
                  if d["class"] in classes and d["score"] > 0.7]
    if len(detections) > 0:
        return False
    return True

Read the return values carefully, because the polarity is easy to get backwards. True means clean, go ahead. False means a blocking detection was found, stop.

Three design choices live in those lines:

Only EXPOSED classes block. The default list is five exposed body parts. The COVERED variants are not in it. A swimsuit or underwear photo, which the model would tag as COVERED, does not trip the gate. The tool is filtering explicit nudity, not skin.
The threshold jumps from 0.2 to 0.7. Detection keeps candidates at 0.2 so nothing is missed early. The gate then demands 0.7 confidence before it will block. That gap is a deliberate bias toward not falsely rejecting clean content. A low-confidence guess does not stop a user's legitimate edit.
MALE_BREAST_EXPOSED is commented out. A bare male chest is an everyday photo, so it was pulled from the block list. That one commented line is a small, honest record of a real product call: where to draw the line between explicit and ordinary.

There is also a sibling method, censor(), that uses the same class list and threshold but instead of returning a boolean it paints black rectangles over the detected boxes. The pipeline does not use censor() for the swap gate; it uses status(). Same detector, two policies, and the swap path chose the hard stop.

Where it is wired in

A detector that nobody calls is decoration. The point of this article is the wiring, so here is every place the swap pipeline consults the gate. It is constructed once, in the processor's __init__:

self.filter_model = NudeDetector(
    providers=self.access_providers if self.device != "cpu" else None
)

Then it guards every swap path. For a single image:

def swap_image(self, target_frame, source_face, face_fields, save_dir, multiface=False):
    save_file = os.path.join(save_dir, "swapped_image.png")
    if self.filter_model.status(target_frame):
        # ... detect faces, run face_swap_model.get(...) ...
    else:
        print("Face swap cannot be applied to nude or explicit content. "
              "Please upload images with appropriate content.")
        cv2.imwrite(save_file, target_frame)   # save the ORIGINAL, unswapped
        return target_frame

For video on the threaded CPU path, the same guard sits inside the per-frame worker:

def process_frame(self, args):
    frame, face_det_result, source_face, ... = args
    tmp_frame = frame.copy()
    if self.filter_model.status(tmp_frame):
        for face in face_det_result:
            if face is None:
                break
            tmp_frame = self.face_swap_model.get(tmp_frame, face, source_face, paste_back=True)
    else:
        print("Face swap cannot be applied to nude or explicit content. ...")
    cv2.imwrite(filename, tmp_frame)   # explicit frame is written unmodified
    return filename

And the CUDA video path has the identical check before its swap call. Three entry points, one rule: status() runs on the target frame first, and face_swap_model.get(...) only runs if it returns True.

Two details worth pulling out:

The check runs on the target, per frame. The target is the content you are swapping a face into. For a video, that means the classifier runs on every frame independently. A clip that is clean for ten seconds and explicit for two will get faces swapped only on the clean frames. The block is granular, not all-or-nothing for the file.
A blocked frame is not dropped, it is passed through. Look at the else branches: the original frame still gets written to disk and into the output video. The gate removes the swap, not the content. The user gets their video back, just without a new face on the frames that tripped the filter.

The face side, for context

The swap that the gate protects is built on InsightFace. Recognition uses the buffalo_l pack, which is an ArcFace embedding model (Deng et al., CVPR 2019). ArcFace turns a face into a vector where same-person faces sit close together, which is how the pipeline finds and aligns the right face to replace. The gate is upstream of all of it. The recognition and the swap model never even see a frame that status() rejected.

Gotchas if you reuse this

A few things that will bite you if you lift this pattern into your own pipeline.

Get the boolean polarity right. status() returns True for clean. I have seen people wire if status(): block() and silently invert the whole safety behavior. Test both cases with a known-clean and a known-explicit image before you trust it.

It runs on CPU by default, and per frame. In the constructor, the detector is given the CUDA provider only when a GPU device is set; otherwise it is CPU. On a long video the gate is real per-frame work. It is cheap next to the swap itself, but it is not free, and it is in the hot loop.

A detector is not a guarantee. This is a 320x320 model with a 0.7 threshold tuned to avoid false blocks. It will miss things, and it can be fooled. It is a reasonable first line, not a proof. If you ship a tool like this, the filter is one layer; clear terms of use and the open-source license are others.

The threshold and class list are policy, not magic numbers. The 0.7 cutoff and the five-item list are where "what counts as explicit" is actually decided in code. If you change them, you are changing the product's stance, not tuning performance. The commented-out male chest line is the example to study.

Takeaway

The safety gate is about thirty lines of real logic. A YOLO-style ONNX model detects body parts, status() keeps only the five exposed classes above 0.7 confidence and returns a boolean, and three swap entry points refuse to run the swap when that boolean is False, passing the original frame through instead. The check runs on the target content, per frame, before recognition or swapping touches it.

The numbers are the policy. The 0.7 threshold biases toward not blocking clean work, the exposed-only class list filters nudity rather than skin, and the commented MALE_BREAST_EXPOSED line marks one place the line was drawn by hand. The code is in visual_processing/face_swap/nudenet.py and swap.py in the Wunjo Make repo.

References

Deng, Guo, Xue, Zafeiriou. "ArcFace: Additive Angular Margin Loss for Deep Face Recognition." CVPR 2019. arXiv:1801.07698 (InsightFace / buffalo_l recognition.)

Filling a hole in video with the next frame, not a guess

Wlad Radchenko — Wed, 22 Jul 2026 12:32:00 +0000

You paint a mask over a person walking across a shot and ask the app to remove them. Behind that person, in this exact frame, there is nothing. The pixels do not exist. But two frames later the person has moved, and the patch of wall they were standing in front of is right there, fully visible.

So why would you ask a network to imagine that wall when the camera already filmed it?

That is the whole idea behind the object remover in Wunjo Make. Most of the hole does not get hallucinated. It gets carried in from frames where the same spot was not covered. The code does this with optical flow, and the interesting part is the ordering: flow first, pixels second, and a transformer only for the leftovers. This is a walkthrough of that pipeline as it runs in portable/src/visual_processing/remover/.

Across a strip of frames, a covered spot in one frame is usually open in another. Flow is the map that connects them. Photo: Unsplash.

The plan in plain words

Think of flow as a set of tiny arrows, one per pixel, that say "this pixel moved here in the next frame." If you have those arrows, you can follow a pixel backward and forward through time. A spot hidden by the object in frame 10 might be visible in frame 7. Follow the arrows from 7 to 10 and you can drop frame 7's real pixel into frame 10's hole.

There is a catch. The arrows themselves are missing inside the mask. Optical flow is computed from image content, and inside the hole the content is the object you want gone. So the motion of the background under the object is unknown. The pipeline solves this in a specific order:

Estimate flow between every pair of adjacent frames (RAFT).
Fill in the flow inside the masked region, so the arrows exist even where the background was hidden (recurrent flow completion).
Use that completed flow to warp real pixels from neighbor frames into the hole, and shrink the mask wherever this works.
Whatever is still masked after all that, because it was never visible in any frame, gets painted by a transformer.

Four stages, and only the last one invents anything.

Stage 1. The arrows (RAFT)

The flow estimator is RAFT (Teed and Deng, ECCV 2020). In the pipeline it runs on short clips and produces both directions at once:

flows_f, flows_b = self.fix_raft(frames[:, f:end_f], iters=raft_iter)

flows_f is forward flow (frame t to t+1), flows_b is backward (t to t-1). raft_iter defaults to 20. RAFT is iterative: it starts from a zero flow field and refines it a fixed number of times, each step looking up a correlation volume and adding a small delta. You can see that loop in raft.py:

for itr in range(iters):
    coords1 = coords1.detach()
    corr = corr_fn(coords1)              # index correlation volume
    flow = coords1 - coords0
    net, up_mask, delta_flow = self.update_block(net, inp, corr, flow)
    coords1 = coords1 + delta_flow       # F(t+1) = F(t) + delta

The clip length is tied to resolution. The code drops it as frames get bigger:

if frames.size(-1) <= 640:
    short_clip_len = 12
elif frames.size(-1) <= 720:
    short_clip_len = 8
elif frames.size(-1) <= 1280:
    short_clip_len = 4
else:
    short_clip_len = 2

This is a memory budget, not an accuracy choice. RAFT's correlation volume is large, so wider frames mean fewer of them in VRAM at once. The result of this stage is gt_flows_bi, a clean flow field everywhere except inside the mask, where the arrows are garbage because they were computed over the object.

Stage 2. Completing the flow inside the hole

This is the part people skip past, and it is the one that makes the whole thing work. Before you can move pixels along the arrows, you need arrows inside the hole. The background moved under that object even though you could not see it, and the model has to guess that motion.

Here is the key intuition: guessing motion is far easier than guessing pixels. A wall's appearance can be anything. A wall's motion is almost always smooth and continuous with the motion just outside the mask. If the visible background is panning left, the hidden background behind it is panning left too. Flow is a slowly-varying field, so you can extend it inward from the edges of the hole with real confidence.

The completion network starts by deleting the unreliable flow inside the mask, on purpose:

masked_flows_forward  = masked_flows_bi[0] * (1 - masks_forward)
masked_flows_backward = masked_flows_bi[1] * (1 - masks_backward)

Multiplying by (1 - mask) zeroes the flow wherever the mask is 1. It does not try to salvage the bad arrows. It throws them out and rebuilds them. The masked flow plus the mask itself go into a 3D convolutional encoder, a dilated middle block, a bidirectional propagation module, and a decoder, all in recurrent_flow_completion.py:

inputs = torch.cat((masked_flows, masks), dim=1)
x = self.downsample(inputs)
feat_e1 = self.encoder1(x)
feat_e2 = self.encoder2(feat_e1)
feat_mid = self.mid_dilation(feat_e2)
...
feat_prop = self.feat_prop_module(feat_mid)   # bidirectional, second-order
...
flow = self.upsample(feat_d1)

The propagation runs both directions, forward and backward, so a frame's flow is informed by its neighbors on both sides. The backward pass literally flips the clip in time, completes it, and flips it back:

masked_flows_backward = torch.flip(masked_flows_backward, dims=[1])
masks_backward        = torch.flip(masks_backward, dims=[1])
pred_flows_backward, _ = self.forward(masked_flows_backward, masks_backward)
pred_flows_backward    = torch.flip(pred_flows_backward, dims=[1])

Then the last step, and it is a small line that matters a lot. combine_flow keeps the original measured flow everywhere outside the mask and only uses the predicted flow inside it:

pred_flows_forward  = pred_flows_bi[0] * masks_forward  + masked_flows_bi[0] * (1 - masks_forward)
pred_flows_backward = pred_flows_bi[1] * masks_backward + masked_flows_bi[1] * (1 - masks_backward)

The network never gets to "improve" the flow it did not need to touch. Outside the hole, RAFT's measurement wins. Inside the hole, the prediction fills in. You end up with one continuous flow field with no seam at the mask boundary.

There is an edgeDetector in this network too, but check the forward pass: it only runs under self.training. At inference it is skipped. So in the shipped app it contributes nothing, it was a training-time loss helper.

Stage 3. Moving real pixels along the completed flow

Now the arrows exist everywhere. Time to follow them and carry pixels in. First the object is blanked out of the frames:

masked_frames = frames * (1 - masks_dilated)

Then image propagation runs with the completed flow:

prop_imgs, updated_local_masks = model.img_propagation(
    masked_frames, pred_flows_bi, masks_dilated, 'nearest'
)

Inside, this is a BidirectionalPropagation module built with learnable=False. That flag matters. The feature-level propagation later in the pipeline is a trained network. This one is not. It is plain warping and bookkeeping, no weights, because here we are moving actual RGB pixels and we do not want a network reinterpreting them. The non-learnable branch in propainter.py is where the real work happens:

mask_prop_valid = flow_warp(mask_prop, flow_prop.permute(0, 2, 3, 1))
mask_prop_valid = self.binary_mask(mask_prop_valid)

union_vaild_mask = self.binary_mask(mask_current * flow_vaild_mask * (1 - mask_prop_valid))
feat_prop = union_vaild_mask * feat_warped + (1 - union_vaild_mask) * feat_current
mask_prop = self.binary_mask(mask_current * (1 - (flow_vaild_mask * (1 - mask_prop_valid))))

Read union_vaild_mask slowly, because it is the gate that decides "can I trust the pixel I just dragged in here." Three conditions, all multiplied:

mask_current is 1 where this frame still has a hole to fill.
flow_vaild_mask is 1 where the flow is trustworthy.
(1 - mask_prop_valid) is 1 where the source pixel I am pulling from was itself not a hole.

Only where all three hold does the warped pixel get written in. Everywhere else the frame keeps what it had. Then mask_prop updates: the hole shrinks wherever the fill succeeded.

That flow trust check is its own neat trick, fbConsistencyCheck. Follow a pixel forward with the forward flow, then backward with the backward flow. If you land where you started, the flow is consistent and probably correct. If you land somewhere else, something is occluded or wrong, so do not trust it:

flow_bw_warped = flow_warp(flow_bw, flow_fw.permute(0, 2, 3, 1))
flow_diff_fw   = flow_fw + flow_bw_warped
mag_sq_fw      = length_sq(flow_fw) + length_sq(flow_bw_warped)
occ_thresh_fw  = alpha1 * mag_sq_fw + alpha2
fb_valid_fw    = (length_sq(flow_diff_fw) < occ_thresh_fw).to(flow_fw)

flow_fw + flow_bw_warped should cancel to near zero for a good pixel. The threshold scales with the flow magnitude (alpha1 = 0.01) plus a constant floor (alpha2 = 0.5), so fast motion gets more slack than slow motion. This is what stops the propagation from smearing a pixel across an occlusion boundary.

After this stage the frames are updated and the mask is smaller. The bookkeeping line that writes it back:

updated_frames = frames * (1 - masks_dilated) + prop_imgs.view(b, t, 3, height, width) * masks_dilated
updated_masks  = updated_local_masks.view(b, t, 1, height, width)

For a lot of videos, a clean pan across a static background, this stage alone fills almost the entire hole with genuine filmed pixels. The output is real footage rearranged in time, not a generated guess.

Stage 4. The transformer fills only what was never seen

Some pixels were covered in every single frame. The object stood still over a spot the camera never caught. No amount of flow can carry in a pixel that does not exist anywhere. That is the only thing left for a generative model.

The inpainting generator (ProPainter, Zhou et al., ICCV 2023) runs on a sliding window of neighbor frames plus a few reference frames:

pred_img = model(selected_imgs.half(), selected_pred_flows_bi,
                 selected_masks, selected_update_masks, l_t)

Notice it takes two masks. selected_masks is the original hole, selected_update_masks is the smaller hole left after stage 3. The encoder gets the masked frame plus both masks stacked as channels, so it knows what was filled by propagation and what still needs inventing:

enc_feat = self.encoder(torch.cat([masked_frames.view(b * t, 3, ori_h, ori_w),
                                   masks_in.view(b * t, 1, ori_h, ori_w),
                                   masks_updated.view(b * t, 1, ori_h, ori_w)], dim=1))

Inside InpaintGenerator.forward, the flow is reused once more, this time to propagate features (the learnable feat_prop_module, downsampled flow at quarter resolution), and only then does the transformer attend across time:

_, _, local_feat, _ = self.feat_prop_module(local_feat, ds_flows_f, ds_flows_b, prop_mask_in, interpolation)
enc_feat = torch.cat((local_feat, ref_feat), dim=1)
trans_feat = self.ss(enc_feat.view(-1, c, h, w), b, fold_feat_size)
trans_feat = self.transformers(trans_feat, fold_feat_size, mask_pool_l, t_dilation=t_dilation)

The transformer is the expensive, creative part, and by the time it runs, most of the hole is already real footage. It only has to dream up the slivers that were genuinely never filmed. That is why the output rarely looks hallucinated: there is very little hallucination in it.

Gotchas you will actually hit

A few things bite when you run this:

The flow array is one shorter than the frames. Flow is between pairs, so N frames give N-1 flows. The code slices accordingly, neighbor_ids[:-1] when selecting flows. Off-by-one here corrupts the warps silently, the video does not crash, it just smears.
Dilate the mask. The pipeline uses masks_dilated, not the raw mask. A tight mask leaves a halo of the object's edge pixels just outside the hole, and propagation happily carries that halo into other frames. Grow the mask a few pixels so the object's fringe is inside it.
Flow completion before propagation, always. If you propagate with raw RAFT flow, the arrows inside the hole are nonsense and you drag object pixels around instead of background. The completion step is not optional polish, it is what makes the warp point at the right source.
The edge detector is dead weight at inference. Do not wire it into your inference path expecting an edge map. It returns None outside training.
fp16 is fine downstream, not for RAFT. The pipeline runs RAFT in higher precision and casts to half later (selected_imgs.half()), because the correlation lookup is sensitive to precision in a way the conv layers are not.

Why this ordering is the point

You could build a remover that hands the whole masked video to a transformer and says "figure it out." It would invent the background frame by frame, and it would flicker, because an inventor has no reason to invent the same wall twice. The flow-guided approach inverts that. It treats inpainting as a search problem first ("where else in this video is this pixel visible?") and a generation problem only for what the search cannot find. Real pixels are temporally consistent for free, because they are the same pixels.

The code is in portable/src/visual_processing/remover/. The three model files are raft.py (the arrows), recurrent_flow_completion.py (completing the arrows inside the hole), and propainter.py (warping pixels, then the transformer). The orchestration that runs them in order is in __init__.py, in process_video_with_mask.

References

Zhou, Li, Loy, et al. "ProPainter: Improving Propagation and Transformer for Video Inpainting." ICCV 2023. arXiv:2309.03897. https://arxiv.org/abs/2309.03897
Teed, Deng. "RAFT: Recurrent All-Pairs Field Transforms for Optical Flow." ECCV 2020. arXiv:2003.12039. https://arxiv.org/abs/2003.12039

Removing text without removing the wall behind it: EAST quads to an inpaint mask

Wlad Radchenko — Tue, 21 Jul 2026 11:29:00 +0000

You want to wipe a sign off a wall in a video. A subtitle, a watermark, a logo painted on brick. The hard part is not finding the text. The hard part is removing only the text and leaving the brick alone. Mask one pixel too wide and the inpainter starts inventing wall where there was wall already, and now the patch looks worse than the text did.

So the whole job comes down to one question: how tight is the mask. This is a walkthrough of how Wunjo Make builds that mask, straight from the code in visual_processing/segmentation/east/. I went in expecting one architecture and found a simpler one, so I will also tell you where the real code path differs from the version you might have read about.

The interesting part of text removal is not the detector. It is the few lines that turn boxes into a mask the inpainter can trust. Photo: Unsplash.

The shape of the problem

Think of the detector as a person with a highlighter. A bounding box is the highlighter held straight, so a slanted street sign gets a fat rectangle with a lot of wall caught in the corners. A rotated quadrilateral is the highlighter tilted to follow the text, so the marked area hugs the letters. EAST gives you the tilted highlighter. That tilt is the whole reason it is worth using here.

EAST stands for Efficient and Accurate Scene Text detector (Zhou et al., CVPR 2017). It is a single fully-convolutional network. No region proposals, no per-character steps. Feed it an image, get back two maps the same shape as a downscaled version of the image: a score map that says "text is here" and a geometry map that says "and the box around it looks like this."

Step 1. The network outputs a score and a geometry, not boxes

The model itself is small. A VGG16 backbone pulls features at several scales, a merge branch fuses them back up to a quarter of the input resolution, and an output head produces three things. Here is the head, from east/model.py:

def forward(self, x):
    score = self.sigmoid1(self.conv1(x))
    loc   = self.sigmoid2(self.conv2(x)) * self.scope          # 4 channels
    angle = (self.sigmoid3(self.conv3(x)) - 0.5) * math.pi     # 1 channel
    geo = torch.cat((loc, angle), 1)
    return score, geo

Read those three lines slowly, because they are the design.

score is one channel through a sigmoid, so every pixel gets a 0-to-1 number for "is this text." loc is four channels, each one a distance, scaled by self.scope which is 512. For a text pixel, those four numbers are the distance up, down, left, and right to the edges of its text box. angle is one channel mapped into the range minus pi-over-two to pi-over-two, the rotation of that box.

So the network never draws a box. It labels each text pixel with four distances and an angle. The box is implied. That is the trick that lets one convolution describe a rotated rectangle: a center pixel plus four reach values plus a twist.

Step 2. Rebuild rotated quads from the geometry

Now you have to turn "distance up, down, left, right, and an angle" back into four corner points. That is restore_polys in east/detect.py. For each pixel that scored high enough:

y_min = y - d[0, i]
y_max = y + d[1, i]
x_min = x - d[2, i]
x_max = x + d[3, i]
rotate_mat = get_rotate_mat(-angle[i])

temp_x = np.array([[x_min, x_max, x_max, x_min]]) - x
temp_y = np.array([[y_min, y_min, y_max, y_max]]) - y
coordidates = np.concatenate((temp_x, temp_y), axis=0)
res = np.dot(rotate_mat, coordidates)
res[0, :] += x
res[1, :] += y

In plain terms: build an axis-aligned rectangle from the four distances, move it so the text pixel is the origin, rotate it by the predicted angle, then move it back. Out come four corner points. The result row is the eight numbers of a quadrilateral, [x0,y0, x1,y1, x2,y2, x3,y3], the corners in order.

Two details that matter in practice. The score threshold and the candidate filter both live here:

xy_text = np.argwhere(score > score_thresh)   # score_thresh defaults to 0.9

A 0.9 threshold is high. It throws away anything the network is not very sure about. For text removal that is the right bias. A false positive means you erase a chunk of real image, which is much worse than missing a faint logo. And is_valid_poly quietly drops any quad that lands mostly outside the frame, so edge garbage does not become a mask.

Every high-scoring pixel proposes its own quad, so a single word produces hundreds of nearly identical quads. The code carries a confidence score in the ninth column of each box for exactly this reason. Worth knowing: in this code path the boxes go to the mask without a separate non-max-suppression merge, so you are relying on the high score threshold and the fact that overlapping filled polygons just paint the same white pixels twice. Painting white twice costs nothing.

Step 3. Rasterize the quads into a mask

Here is where the boxes stop being numbers and become a mask. plot_filled_mask:

def plot_filled_mask(img, boxes):
    mask = Image.new('RGB', img.size, (0, 0, 0))   # black canvas
    draw = ImageDraw.Draw(mask)
    if boxes is not None:
        for box in boxes:
            draw.polygon(
                [box[0], box[1], box[2], box[3],
                 box[4], box[5], box[6], box[7]],
                fill=(255, 255, 255))
    return mask

Black image the size of the frame, then every detected quad gets painted solid white. White means "inpaint this," black means "leave it." Because the quads are tilted, the white regions follow slanted and rotated text instead of boxing it in a wide rectangle. That is the wall-saving step. A straight bounding box around a diagonal sign would whitewash the triangles of brick in the corners, and the inpainter would happily repaint brick it did not need to touch.

The full detect call wraps the network and this rasterizer together, in SegmentText.detect_text:

def detect_text(self, img_path, max_resolution=1280):
    img = Image.open(img_path)
    w, h = img.size
    if max(w, h) > max_resolution:                # cap the long side
        scale = max_resolution / max(w, h)
        img = img.resize((int(w*scale), int(h*scale)))
    img, _, _ = resize_img(img)                   # make dims divisible by 32
    boxes = detect(img, self.model, self.device)
    filled_mask = plot_filled_mask(img, boxes)
    return filled_mask.resize((w, h))             # back to original size

The resize_img step is not cosmetic. The VGG backbone halves the resolution five times, so the input has to be divisible by 32 or the feature maps will not line up. The 1280 cap is there so a 4K frame does not blow up memory in the detector for no gain.

Step 4. The optional crop, for "only this corner"

If the user drew a box around just the subtitle area, you do not want to erase a logo elsewhere in the frame. SegmentText.crop handles that by keeping only the part of the mask inside the chosen rectangle:

@staticmethod
def crop(filled_mask, text_area: list = None):
    if text_area is not None:
        x1, y1, x2, y2 = text_area
        img = Image.new('L', filled_mask.size, color=0)    # all black
        region = filled_mask.crop((x1, y1, x2, y2))        # cut the user's box
        img.paste(region, (x1, y1))                        # paste it back, rest stays black
        return img
    return filled_mask

It does not crop the image down. It blacks out everything outside the user's rectangle and keeps the detected text inside it. So you still get tight quads, just restricted to the region you care about.

Step 5. Hand the mask to inpainting (and where SAM actually sits)

Now the part where I corrected my own mental model. I expected the EAST quads to become point or box prompts for SAM (Segment Anything, Kirillov et al., ICCV 2023), letting SAM tighten the mask further. That is a reasonable design. It is not this code.

In visual_processing/inference.py, the remove path is one function that handles two jobs. When the user clicks an object, it runs SAM. When the user asks to remove text, it runs EAST. They are siblings, not a chain. The text branch is gated behind use_remove_text, and SAM is explicitly skipped there:

if predictor is None and not use_remove_text:
    predictor = segmentation.init_vit(sam_vit_checkpoint, vit_model_type, device)
else:
    lprint(msg=f"[{method}] Model predictor will not be loaded "
               f"because of not need to use for text.", ...)

The text branch builds its own detector and writes the EAST mask straight to disk per frame:

segment_text = SegmentText(device=device, vgg16_path=vgg16_baseline_path, east_path=vgg16_east_path)
...
mask_text_frame = segment_text.detect_text(img_path=frame_file_path)
mask_textarea_frame = segment_text.crop(mask_text_frame, coord)
mask_textarea_frame.save(mask_text_frame_path)

So the real flow for text is EAST to mask to inpaint, no SAM in between. SAM is the tool for "remove this person," EAST is the tool for "remove this text," and they share plumbing because both of them produce a white-on-black mask that the same inpainter consumes.

The inpainter is chosen by source. Video on a capable GPU goes to ProPainter (Zhou et al., ICCV 2023), which propagates pixels across frames so the filled region stays consistent over time. Images, or anything on CPU, go to a single-frame inpainting model. Either way, the mask reaches it after one more cleanup. The white mask is dilated and binarized:

binary_mask = cv2.threshold(segment_mask, 128, 1, cv2.THRESH_BINARY)[1]
...
kernel = np.ones((kernel_size, kernel_size), np.uint8)   # kernel_size = 10
mask_to_save = cv2.dilate(mask_to_save, kernel, iterations=1)

The dilation by a 10-pixel kernel is deliberate. EAST hugs the glyphs tightly, but ink has a faint halo, anti-aliasing and shadow, just outside the letter shape. If the mask is exactly the glyph, the inpainter leaves a ghost outline. Growing the mask by a few pixels eats that halo. So the pipeline goes tight, then loosens by a known amount, instead of starting loose and hoping.

Finally the inpainter only writes inside the mask. From process_remover:

merge = generated * mask_raw_3ch + img_raw * (1 - mask_raw_3ch)

Generated pixels where the mask is white, original pixels everywhere else. That single line is the promise that the wall behind the text is the exact wall that was there. The model can hallucinate freely inside the white region and it will never touch a pixel outside it.

Gotchas you will actually hit

A few things I would warn a teammate about before they touch this path.

The 0.9 score threshold is conservative on purpose, and it will miss low-contrast or stylized text. If you lower it, you trade missed text for erased background. For removal, keep it high and let the user draw a text_area box for the stubborn cases instead of dropping the threshold globally.

Divisibility by 32 is silent. If you bypass resize_img and feed an arbitrary size, the merge branch upsamples will not match the skip connections and you get a shape error deep in the model. Always route through resize_img.

The mask is per frame, recomputed independently for each frame. There is no temporal smoothing on the text mask itself. For video, the consistency comes later, from ProPainter propagating the fill, not from the detector. If your text flickers in and out of detection, that flicker can show up before propagation evens it out.

Dilation cuts both ways. The 10-pixel grow is great for eating halos around small subtitles. On very large text it can swallow a sliver of background you wanted to keep. The kernel size is a constant in the code worth tuning for your content.

Takeaway

Removing text without removing the wall is a masking problem, and the mask quality is set in three short functions. The network emits a score map plus four distances and an angle per pixel. restore_polys rebuilds rotated quads from that geometry so the marked area follows slanted text. plot_filled_mask paints those quads white on black. Then a 10-pixel dilation eats the ink halo, and the inpainter writes only inside the white. SAM is in the same file but it is the object-removal sibling, not a stage in the text path.

The code is in visual_processing/segmentation/east/ in the Wunjo Make repo. If you are fighting ghost outlines after text removal, start at the dilation kernel and the score threshold. Those two constants decide whether the wall survives.

References

Zhou, Yao, Wen, et al. "EAST: An Efficient and Accurate Scene Text Detector." CVPR 2017. arXiv:1704.03155
Kirillov et al. "Segment Anything." ICCV 2023. arXiv:2304.02643
Zhou, Li, Loy, et al. "ProPainter: Improving Propagation and Transformer for Video Inpainting." ICCV 2023. arXiv:2309.03897

DEV Community: Wlad Radchenko

Keep the same face across frames with a tiny embedding memory

The everyday version first

The memory

Distance is just L2

The sort that ignores "biggest"

The threshold is not a constant

The running average and why it is clamped

Warm-up: position first, identity later

Gotchas you will actually hit

Wrap-up

References

Upscaling a huge image on a small GPU: tiled super-resolution without seams

The two knobs that matter

Step 1. Lay out the grid

Step 2. Cut one tile, with its halo

Step 3. Stitch the core back, throw the halo away

A note on what "overlap" means here

Step 4. The whole-image padding around the edges

Gotchas I would warn you about

Takeaway

References

A referee for lip-sync: scoring whether audio and video match with two embedding towers

Two towers, never mixed

The trick at the bottom: normalize, then compare

Feeding the audio tower: wav to mel

Where the scoring actually happens

Picking the right face from a crowd

Things that will trip you up

References

Keeping the same face across generations: how reference photos enter an OmniGen prompt

The prompt is a string with image holes in it

Tags carry an order, and the order is enforced

Reserving the right number of slots for each image

The placeholder spans become image features

One more thing: image tokens see each other fully

Gotchas you will actually hit

Takeaway

References

Picking video keyframes with scene cuts plus a fixed stride

Why "every Nth frame" is dumb

Why "scene cuts only" is too sparse

Combining both

The fallback when nothing is detected

Dedup before you read the disk

Gotchas you will actually hit

Wrap-up

References

Auto-highlights with a small VLM or how the model votes yes or no on every 10-second cut

The whole pipeline in one breath

The model wrapper, and the one line that parses the answer

The model writes its own rubric

The yes/no vote, and why scoring is just "yes" in response

Stopping early: shuffle, chunk, budget

The audio twin: Whisper instead of frames

Stitching the survivors

Gotchas if you run this yourself

References

Growing an image past its border without a visible seam

The shape of the problem

Step 1. Size the new canvas, and crop if it would blow the cap

Step 2. Three buffers

Step 3. Place the pixels and carve the strips

Step 4. Father the mask edge

Step 5. Pre-fill the empty strip with Telea

Doing it on both sides: the centering pass

Gotchas you will actually hit

Takeaway

References

The file is called encryption.py. It is actually an invisible watermark!

What problem this solves

How the marker gets into a frame

The part that makes it survive: repeat and average

Where it gets written, and the skip check

Gotchas you will actually hit

The honest part: this is obfuscation, not protection

Wrap-up

LivePortrait without the puppet look: stitching and retargeting in a 126-number MLP

The one network class, used three ways

Stitching: keep the head on the shoulders

The yes/no vote, and why scoring is just `"yes" in response`