I spent some time with a small matting evaluation run on 2026-11-03, and the question I got stuck on wasn't which method won. The test: 90 frames of a synthetic matting clip at 540×960, the usual SAD, MSE, gradient and connectivity errors, plus two temporal measures. The natural next step is to set those numbers beside a paper's table. But looking at the MSE column, I couldn't say from the table alone whether it covered the whole frame or only the edges. If I couldn't tell, nobody else reading it could.
So before any number, a matting report has to state three things: the region each metric covers, how each metric is normalized, and where the ground truth came from. On this clip, region alone moved one matte's MSE more than six-fold, while the MSE gap between that matte and the cleanest chroma key was about two-fold. Normalization made a pure-flicker control look like the steadiest result. The third couldn't be measured at all, which is why it belongs there.
The region moved MSE about six times
The "edge band" here is the trimap unknown region. Take the true alpha, mark pixels at or above 0.99 as foreground and at or below 0.01 as background, erode both by 7 pixels with a 15-pixel disk, and what remains is unknown. On this clip that band averaged 81,817 of 518,400 pixels, about 15.8% of the frame.
For the AI matte on the green composite, whole-frame MSE (×1000) was 1.41 and band-only MSE was 9.03. Same matte, same errors, different denominator, because MSE averages over the region. I've argued over 0.5 dB before; here a choice nobody wrote down changed a number more than six-fold. SAD stayed at 2.89 in both, since SAD is a sum and nearly all of that method's errors sit inside the band anyway.
That is also where band-only reporting hides things. A chroma key with similarity pushed to 0.25 removed whole patches of dark clothing and shoes. Its whole-frame SAD was 93.37 and its band SAD 40.65, so more than half of its error lived where a band-only table never looks. The region line needs the thresholds and the erosion size too; "unknown region" without a radius can't be reproduced.
Normalization ranked pure flicker as steadiest
SAD in this evaluation is the summed absolute difference divided by 1000, so it grows with resolution. MSE is a per-pixel mean. The raw JSON output stores MSE unscaled while the table multiplies it by 1000, so 0.0014 in one file and 1.41 in the table are the same value. Copy across without checking and you are off by a factor of a thousand.
The temporal metric is where normalization did the most damage. Here is dtSSD next to a flicker measure the evaluation added, taken only where the true alpha didn't change:
d = (p_t - p_prev) - (g_t - g_prev)
dtSSD = np.sqrt((d**2).sum() / d.size) * 100
static = np.abs(g_t - g_prev) < 0.5 / 255
flicker = np.abs(p_t - p_prev)[static].mean() * 1000
d.size counts every pixel in the frame. The evaluation also built a control: the ground truth plus independent random noise (σ=0.06) on the edge band in every frame, which is pure flicker by construction. Its dtSSD was 1.93, the lowest in the table. Its flicker inside the band was 11.844, tied for the worst. Small noise on 15.8% of the pixels got diluted by the full-frame divisor. That dtSSD was also written from memory of the RVM evaluation code and hasn't been checked line by line, one more reason these numbers shouldn't sit beside a paper's.
The ground truth was a model's output
There were no hand-pulled mattes. The truth is what an RVM ResNet50 model produced for frames 90 to 179 of a real clip, with a waving hand in them. The AI matte being scored is the compatible tier (RVM MobileNetV3) of the Beta video matting in ImgIng https://imging.ai/ , run in a desktop browser. The true foreground and alpha were composited over two generated backgrounds: flat green with σ=2 noise, and 420 random shapes shifted left one pixel per frame.
That truth is a known alpha, not a correct one. If the model got a strand of hair wrong, the truth carries the same mistake. The foreground colours keep green spill from the original cloth, which costs chroma keying at the edges. And the tested model shares a family with the one that made the truth, so a bias toward the AI group can't be ruled out. This setup can't size that bias.
The backgrounds belong in this line too, because the same truth composited over a different background is a different test.
What the header should say
For this test the header reads: synthetic composite, 90 frames, 540×960; region is whole frame or a band from 0.99 / 0.01 thresholds eroded 7 px; SAD summed ÷1000, MSE averaged ×1000, dtSSD over the full frame; truth is model output from the same family as one tested method. Connectivity error follows the step 0.1 and 0.15 cutoff of the Deep Image Matting public code, and hasn't been matched number for number against the Matlab original. With that on top, someone can decide whether this table compares with theirs. Without it, most of the time it doesn't.
The source clip is "Filming with the green screen" by VOA Learning English on Wikimedia Commons, public domain, trimmed and composited (modified). Both backgrounds are generated.
Top comments (0)