DEV Community

orca_forge
orca_forge

Posted on Originally published at forge.workstyle.tech

After Removing the Background, the Characters Became Semi-Transparent Too

#ffmpeg #colorkey

📝 Originally published (in Japanese) at forge.workstyle.tech.

I was investigating a bug where the face and jaw were misaligned. While zooming in on the areas before and after the boundary and lining them up, I noticed something else.

Eve's head was slightly transparent.

You could see the background bleeding through just a bit. The same was true for the grey shirt. Even Dario's stubble had become faint.

When I traced it back, I found that this was happening in every published version.

What I was doing

I generate the lip-sync videos against a solid cream-colored background. When compositing, I remove this background and overlay only the character.

colorkey=0xF4F1E4:0.17:0.08
Enter fullscreen mode Exit fullscreen mode

The second value (similarity) determines how close a color must be to be considered "background." The third value (smoothing) determines how much to blur that boundary.

There are colors within the character that are close to the background color:

  • The top of Aibu's head (a pale skin tone that becomes bright when light hits it)
  • The gray shirt
  • Dario's stubble (pale dots over the skin tone)

A value of 0.17 was wide enough to identify these as "part of the background." They don't disappear completely; they become translucent instead. Because of this, you won't notice it at first glance.

Comparison of the same person composited against a dark background. The left shows the clothes and skin transparent at a threshold of 0.17; the right shows the transparency eliminated by narrowing the threshold to 0.10.

Why the threshold was set widely

The background of the lip-sync video sometimes changes color slightly during generation. To remove the changed part, it was taken broadly.

However, the background color does not need to be determined at one point. It was set to take the average of 4 points, so the change in color can be absorbed by that. There was no need to widen the threshold.

How Far Can We Narrow It Down

We decreased the similarity and observed both "background residue" and "over-extraction of characters" for all 10 characters.

0.17 : 0.08  … Light-colored characters become semi-transparent
0.10 : 0.04  … No residue, characters not extracted (adopted)
0.07 : …     … Background noise remains
Enter fullscreen mode Exit fullscreen mode

The verification method is as follows.

  • Enlarge the faint parts of the characters (bald head, gray clothes, beard) to their original size
  • Also, check the latter half of the generation (where the background is more likely to change)
  • Observe all characters with the same value, rather than changing the value for each character, as this would disrupt management

Why I didn't notice

In the scaled-down list, transparency is not visible. Because the background and the subject share similar colors, there is almost no color difference even when it is see-through.

To confirm something is "not transparent," the fastest way is to overlay it over a color different from the background. If you composite it against a black or primary color background, you will immediately see any parts that are too transparent.

Summary

  • The colorkey threshold should be determined not only by "whether the background is removed" but also by "ens the subject is not removed."
  • To handle background color shifts without widening the threshold, absorb the color by averaging it across multiple points in time.
  • Semi-transparent areas are invisible against backgrounds of similar colors. Verify them by layering them against a background of a different color.

Top comments (0)