![ ](https://dev-to-uploads.s3.us-east-2.amazonaws.com/uploads/articles/eMy habit with any video tool has been to look at the first frame of the output, scrub to the middle, and call it done. If the start is right, the model has "found" the person and the rest will follow, or so I assumed. I've been reading up on video matting lately, specifically RVM, which carries a recurrent state from one frame to the next, and that assumption seemed even safer for a model with memory. So I tested one case I expected to break at the start, and one I expected to be hard all the way through. Neither behaved the way I predicted.
The first test was a clip that starts with nobody in it. I don't have footage like that, so I built it: two seconds of an empty street (a crop of road surface from a CC0 Commons clip, scaled up and held still), then a hard cut straight into "Tarun speaking 01", a CC0 clip of a man talking at an event. I spliced it with ffmpeg, so it's an empty shot cutting to a person already standing there, not someone walking into frame. I ran it through the imging video matting page on the Compatible model with a pure green background, and I fully expected the first frames after the cut to be broken, with the model still "warming up".
They weren't. The 120 empty frames came out as solid green, never more than 2 levels off pure #00FF00. On the very first frame after the cut the whole man was there. I compared it with a separate run of the same clip without the empty intro, where the person is there from frame 0. Measured on the edge region, the two differed by 0.028 on average. That's smaller than the 0.042 difference I got between them later in the clip where nothing special is happening, which is mostly noise from estimating the matte back out of a green H.264 frame. Within what I can measure, the blank opening changed nothing. One clip, one model, so I'm not saying more than that.
The second test was a crowd. "Folkloristic dance in Naples (tammorriata)" is 12.84 seconds of street musicians with a crowd behind them and a slowly panning handheld camera, CC0 again, same model, same green. On the first frame the lead singer on the left is there, mostly solid. At 6 seconds he's there too, a bit see-through, which I'd have forgiven. By my old habit of first frame plus a look at the middle, this clip passes. The end of the clip doesn't. He isn't faded there, he's gone, and the left half of the frame, where he's still standing and singing in the source, has no foreground at all.
So now I sample instead of scrubbing. ffmpeg -i out.mp4 -vf fps=1 frames/%03d.png gives one frame for every second, 13 of them for this clip, and I put source and output next to each other in a grid. One thing I didn't know: with fps=1, ffmpeg doesn't hand you the frame at exactly 0 s, 1 s, 2 s. When I checked the timestamps with its showinfo filter, it had picked frames roughly half a second into each second. For eyeballing that doesn't matter, but my first labels on the grid were off by half a second.
Footage: Wikimedia Commons "Folkloristic dance in Naples (tammorriata)" (CC0), Compatible model, green background.
The grid shows a lot more than the two frames I used to check. He starts fading within the first three seconds, is down to a faint shape around 3 s, comes back around 5 to 6 s, and from about 10 s to the end of the clip he's not there at all. Over the same 13 seconds, the front row on the right stays solid in every tile. There's no scene cut anywhere in this clip, only the camera slowly panning.
What I don't understand yet is why he doesn't come back at the end the way he did at 5 seconds. My guess is that the recurrent state, which is what keeps edges steady from frame to frame, can also keep a wrong decision going: once he's treated as background for a while, he stays background, and the state only gets wiped clean on a scene cut, which this clip never has. That's a guess from reading about how RVM works, not something I've verified. I also only ran the small model, so I can't say whether the bigger ones do better on this clip.
What I changed in how I check results is short. The first frame tells me very little, and neither does a random frame in the middle. I sample one frame per second over the whole clip and look at the grid, and for anything important I look at the last few seconds on purpose, since that's where this one broke. A one-per-second grid can still miss something that goes wrong and recovers within a second, and I haven't figured out a cheap way to catch that without looking at every frame.
The matting page I used is at https://imging.ai/.
e448513dtbbhlyoxfiu.png)
Top comments (0)