Keep the same face across frames with a tiny embedding memory

You click a face in frame 1. Two people are on screen. By frame 40 the tool has quietly swapped to the other person, because for a few frames they leaned closer to the camera and got bigger. Anyone who has built face tracking on real video has hit this. The naive fix everyone tries first is "pick the biggest face" or "pick the most centered face", and it fails the moment a second face exists.

This post walks through the part of Wunjo Make that fixes it: a small per-identity memory of face embeddings that says "this is the person you picked, keep following them" even when someone else is bigger or more centered. The code lives in portable/src/visual_processing/face_detection/recognition.py, in the FaceRecognition class.

The everyday version first

Think about recognizing a friend in a crowd. You do not pick whoever is closest or tallest. You remember their face and scan for a match. If someone walks between you, you do not suddenly decide the stranger is your friend.

That is the whole idea here. We keep a running memory of what the chosen face looks like, as numbers, and each new frame we ask "which of these detections looks most like the one I remember?" before we worry about where it is on screen.

The numbers come from a face recognition network. InsightFace's buffalo_l pack runs an ArcFace model that turns a face crop into a 512-dimension vector, an embedding (Deng et al., CVPR 2019; arXiv:1801.07698). Two crops of the same person land near each other in that 512-D space. Two different people land far apart. The detector that finds the boxes in the first place is RetinaFace (Deng et al., 2019; arXiv:1905.00641). We do not need the math of either to use them. We need one operation: distance between two embeddings.

The memory

Here is the state the class holds:

class FaceRecognition(FaceAnalysis):
    def __init__(self, name='buffalo_l', root='~/.insightface', allowed_modules=None, **kwargs):
        super().__init__(name=name, root=root, allowed_modules=allowed_modules, **kwargs)
        self.id = 0
        self.window = {}           # per-id list of past detections
        self.window_size = 50      # warm-up length / last n distances
        self.max_window_size = 250 # cap on stored embeddings in RAM
        self.running_average = {}
        self.max_rate_of_change = 0.05

window is a dict keyed by a face id, so you can track more than one target at once, each with its own memory. Every entry stored for an id is small:

def update_memory(self, face):
    self.set_face_id()
    center = ((face.bbox[0] + face.bbox[2]) / 2, (face.bbox[1] + face.bbox[3]) / 2)
    self.window[self.id].append({
        'embedding': face.normed_embedding,
        'center': center,
        'gender': face.gender,
    })
    if len(self.window[self.id]) > self.max_window_size:
        self.window[self.id].pop(0)

Note normed_embedding, not the raw embedding. ArcFace embeddings are L2-normalized, so they sit on a unit sphere. That matters for the distance step, coming up. The list is a sliding window: once it passes 250 entries it drops the oldest. So the memory is recent. If a person's lighting drifts over a long shot, the memory drifts with them instead of clinging to frame 1 forever.

Distance is just L2

The "looks like" test is one line:

@staticmethod
def calculate_distance(embedding1, embedding2):
    return np.linalg.norm(embedding1 - embedding2)

Straight L2 (Euclidean) distance. Because both vectors are already normalized, this distance is bounded and behaves consistently across face sizes and lighting. Small distance, same person. Larger distance, probably someone else.

For each detected face in the current frame, we do not compare against one remembered embedding. We compare against every embedding in the window and take the mean:

face.mean_distance = np.mean([
    self.calculate_distance(face.normed_embedding, known_face["embedding"])
    for known_face in self.window[self.id]
]) if len(self.window[self.id]) > 0 else float('inf')

Averaging over the window is the quiet trick. A single bad frame, motion blur, a half-turn of the head, gives a noisy embedding. If you compared against only the last frame, one ugly frame could throw the match off. Averaging across the recent window smooths that out. One bad embedding in 50 barely moves the mean.

The sort that ignores "biggest"

This is the heart of it. Selection happens in sort_by_direction, in the distance-from-embedding branch:

if direction == 'distance-from-embedding':
    return sorted(
        filter(lambda face: face.mean_distance < threshold * face.dynamic_threshold, faces),
        key=lambda face: (((face['bbox'][2]+face['bbox'][0])/2 - face_center[0])**2
                        + ((face['bbox'][3]+face['bbox'][1])/2 - face_center[1])**2)**0.5
    )

Read it in two moves. First the filter: throw away any face whose mean embedding distance is too high. That is the identity gate. A stranger who happens to be huge and dead-center never makes it past this line, because their embedding does not match the memory. Then the sorted: among the faces that did match, pick by plain pixel distance from where the person was last seen. Spatial position is the tie-breaker, not the decision.

That ordering is the fix. "Biggest face" and "most centered face" use geometry as the decision. Here geometry only breaks ties between faces that already passed the identity check.

The threshold is not a constant

threshold * face.dynamic_threshold is the gate. Two pieces. dynamic_threshold depends on how big the face is in the frame:

@staticmethod
def get_dynamic_threshold(face, height):
    face_height = (face.bbox[3] - face.bbox[1]) / height
    if face_height < 0.1:
        return 1.05
    if face_height < 0.2:
        return 1.2
    if face_height < 0.4:
        return 1.3
    elif face_height < 0.8:
        return 1.35
    elif face_height < 0.9:
        return 1.4
    else:
        return 1.45

Why scale by size? A tiny, far-away face gives a noisier, less reliable embedding, so we keep the gate tight (1.05) and demand a close match. A large, clear face is trustworthy, so we loosen the gate (up to 1.45) and forgive more variation, because a big face turning its head can legitimately drift further in embedding space while still being the same person. The constant threshold is the per-id running_average, which I will get to.

About the author. I'm Wlad Radchenko, a software engineer. The code in this article comes from Wunjo Make (open source), local software for video makers, and Wunjo Design, an offline PWA for designers. Get in touch to find more on GitHub and LinkedIn.

Multiple faces in one frame, the exact situation where "pick the biggest" jumps to the wrong person. Photo: Unsplash

The running average and why it is clamped

The memory has no idea what a "matching" distance looks like on your specific video. A studio close-up and a shaky handheld clip have different baseline distances for the same person. So the gate self-calibrates. Once the window has warmed up past window_size - 1 entries, it seeds a per-id running_average:

if len(self.window[self.id]) > self.window_size - 1:
    if self.running_average[self.id] is None:
        self.running_average[self.id] = max([
            self.calculate_distance(ret[0].normed_embedding, known_face["embedding"])
            for known_face in self.window[self.id]
        ])
    else:
        change = max([
            self.calculate_distance(ret[0].normed_embedding, known_face["embedding"])
            for known_face in self.window[self.id]
        ]) - self.running_average[self.id]
        change = np.clip(change, -self.max_rate_of_change, self.max_rate_of_change)
        self.running_average[self.id] += change

The seed is the worst (max) match distance inside the current window, a rough "this is how far the same person can drift." After that, each frame nudges the average toward the new worst-match distance, but the nudge is clamped to ±0.05.

That clamp is the safety rail. Imagine the memory got fooled for one frame and locked onto the wrong face. Without the clamp, that wrong face would yank the running average toward its own distances in a single step, and the gate would re-tune itself around the impostor. With the clamp, the gate can only move 0.05 per frame. One bad frame cannot rewrite the baseline. Recovery is possible because the correct face keeps matching at low distance while the average can only crawl.

Warm-up: position first, identity later

There is a chicken-and-egg problem. On frame 1 the memory is empty, so embedding distance is meaningless. The code handles this by falling back to pure position during warm-up:

ret = sort_by_direction(
    ret,
    'distance-from-retarget-face'
        if direction == 'distance-from-embedding' and len(self.window[self.id]) < self.window_size
        else direction,
    face_center, face_gender,
    self.running_average[self.id] if self.running_average[self.id] else "inf"
)

Until the window holds window_size (50) entries, distance-from-embedding silently degrades to distance-from-retarget-face, which is just Euclidean distance from the click point with no identity gate. And on the very first frame, the seed is the face whose box actually contains the click point:

if len(self.window[self.id]) == 0 and face_center[0] and face_center[1]:
    for face in ret:
        x1, y1, x2, y2 = face.bbox
        if x1 <= face_center[0] <= x2 and y1 <= face_center[1] <= y2:
            ret = [face]
            break
    else:
        ret = []

So the flow is: you click a person, the box under your click seeds the memory, the next ~50 frames are tracked by position while the embedding window fills, and from then on identity drives selection and position is only the tie-breaker.

Gotchas you will actually hit

A few things that cost real debugging time:

• Feed RGB, consistently. The class is built on InsightFace; mixing BGR and RGB between detection and the embedding model gives embeddings that drift for no visible reason. Stay in one color space end to end.
• Call clear() between clips. window and running_average persist on the instance. Reuse the object across two unrelated videos and the second video starts with the first video's memory. clear() resets both.
• Set self.id per target. All the state is keyed by self.id. Track two people by passing id= per call so each gets its own window; forget it and both collapse into id 0.
• Gender filtering is off in this version. There is a commented-out variant of the filter that also matched on face.gender. It is disabled because of a gender-detection bug, so do not count on gender as part of the gate today. The active filter is distance-only.
• The first 50 frames are not identity-aware. During warm-up the tracker can still be lured by a closer face, because it is running on position alone. If your subject is not the nearest face to the click in those early frames, seed more carefully.

Wrap-up

The mechanism is small. A sliding window of normalized ArcFace embeddings per identity, a mean L2 distance as the match score, a size-aware gate that self-calibrates with a clamped running average, and position used only to break ties among faces that already passed the identity check. That combination is why the tracker stays on the person you picked instead of hopping to whoever is biggest.

Read the full file here: recognition.py in Wunjo Make. It is open source, so you can paste the class into a notebook, feed it frames, and watch the window fill.

References

• Deng, Guo, Xue, Zafeiriou. "ArcFace: Additive Angular Margin Loss for Deep Face Recognition." CVPR 2019. arXiv:1801.07698. https://arxiv.org/abs/1801.07698 (InsightFace / buffalo_l recognition.)
• Deng, Guo, Zhou, Yu, Kotsia, Zafeiriou. "RetinaFace: Single-stage Dense Face Localisation in the Wild." 2019. arXiv:1905.00641. https://arxiv.org/abs/1905.00641