Temporal video matting: steadier edges you can't see, and a lost singer you can

The video matting page in a tool I help build runs a temporal model: RVM keeps a recurrent state from frame to frame instead of treating each frame as a fresh image. The matting model isn't my code. My part is everything around it, which means WebCodecs decode and encode, and getting 7.2 MB or 51.3 MB of ONNX weights into the tab and cached. Carrying that state through the pipeline costs real plumbing, so I wanted a number for what it buys over the obvious alternative, which is to decode every frame and hand it to the per-frame image matting we already ship. I got the number. It was smaller than I expected, and while collecting clips for it, the same design showed me a failure I hadn't been looking for.

Measuring flicker without ground truth

The clip is "Tarun speaking 01" from Wikimedia Commons (CC0): 464×832, 59.94 fps, one man talking at an event, a few people moving behind him. I converted it to H.264 so it looks like a phone file, and took frames 120 to 179, one second. Temporal side: the compatibility model (RVM MobileNetV3) and the quality model (RVM ResNet50), each exporting a transparent WebM, alpha decoded back with libvpx. Per-frame side: the same 60 frames pushed one by one through our image matting on its fast setting, ISNet INT8. No two consecutive ISNet mattes were identical, so each frame really was inferred from scratch. There's no true alpha for a real clip, so I only compared how much the edge moves. The band is every pixel that was soft (alpha between 0.1 and 0.9) in any of the three runs, grown by 3 px and limited to a box around the speaker: 16,853 pixels. Background people are outside the box on purpose, because RVM keeps them and ISNet doesn't. Plain frame-to-frame change mixes flicker with motion, so the metric I trust most only counts pixels whose source colour barely changed:

def static_jitter(alpha, src, band, still=3.0):
    vals = []
    for t in range(1, len(alpha)):
        still_px = band & (np.abs(src[t] - src[t-1]).mean(-1) < still)
        vals.append(np.abs(alpha[t] - alpha[t-1])[still_px].mean())
    return float(np.mean(vals))

On those still pixels ISNet moves 0.00483 per frame, MobileNetV3 0.00298 and ResNet50 0.00329, so per-frame is about 1.5× more restless. A second-order measure that cancels steady motion agrees (0.00485 against 0.00360 and 0.00344), and plotted over time the ISNet curve is a sawtooth, one frame up and the next down. Then the absolute scale: alpha moves by less than 1% on average in all three. I put four consecutive frames side by side at 3× and couldn't point at the flicker. What you can see is the edge shape. ISNet gives a soft, wide shoulder and no hair outline, and both RVM models keep strands on top of the head.

Clip: Wikimedia Commons "Tarun speaking 01" (CC0).

One metric went the other way. Counting only transition pixels, MobileNetV3 came out higher than ISNet (0.0363 against 0.0274). My reading is that ISNet's wide soft edge adds a lot of quiet transition pixels and dilutes its own average, but that's a guess. The comparison also leans against the temporal side, since its alpha went through lossy VP9 and ISNet's went straight to PNG. And ISNet's colour pixels came out about 6 levels darker than the source, which has nothing to do with alpha, and I still don't know why.

The encoder side

This is the part I actually own, and it isn't a model choice. In the headless Chromium 149 I test with:

const cfg = { codec: 'vp09.00.10.08', width: 464, height: 832, bitrate: 2_660_000, framerate: 59.94 };
(await VideoEncoder.isConfigSupported({ ...cfg, alpha: 'keep' })).supported     // false
(await VideoEncoder.isConfigSupported({ ...cfg, alpha: 'discard' })).supported  // true

So the encoder config won't give you alpha for free. WebM stores VP9 alpha as a second coded stream in BlockAdditional, and a plain ffmpeg packet count doesn't include it: the transparent export is 5.47 MB, while the main video packets ffmpeg counts add up to 2.72 MB. Take out the Opus track and roughly 2.6 MB is left, which as far as I can tell from the container is the alpha. I haven't walked the bytes to prove it.

The memory also holds on to a mistake

The second clip was "Folkloristic dance in Naples (tammorriata)", also CC0: 1280×720, 25 fps, 12.84 s of a street performance with a handheld camera panning slowly and no cuts. Compatibility model, pure green background. The lead singer stands left of centre, a bit behind the front row, in dark clothes against a crowd and metal barriers. At frame 0 his box is 68% opaque. By 3 s he's a faint shadow at 4%. Around 6 s he comes back, but see-through. From 10 s the left half of the frame holds 0.1% foreground, and from 10.4 s to the end it's zero. The four people in the front row, lighter clothes and clear outlines, stay at 37 to 44% of the right half for the whole clip.

Footage: Wikimedia Commons "Folkloristic dance in Naples (tammorriata)" (CC0).

The pipeline resets the state when it detects a scene cut, and there is no cut here, so nothing forces a fresh look at the singer. My understanding, which again isn't my module, is that the same recurrence that smooths the edge also carries "not foreground" forward once the model has settled on it. I haven't run the quality model or the experimental tier on this clip, so this is one clip on the smallest model, not a statement about the others.

Where that leaves me: temporal matting earns its keep on edges by a margin that's real in the numbers and small on screen, and a per-frame pipeline would at least look at the singer again on every frame, whether or not it then finds him. What I haven't worked out is whether a periodic re-check without a scene cut would bring him back or just add flicker. The page is at https://imging.ai/ if you want to throw your own clip at it.