Replacing a headshot background is a short function once you have an alpha matte. The failure is almost never the fill. It is the hairline: a green fringe, a sliver of bookshelf left in the flyaways, or a crown that looks clipped.
I measured that failure on one generated head-and-shoulders cutout with a known alpha channel. I composited the subject onto green, white, light gray, and a blurred office, then asked three segmenters to recover the matte and put her on a navy field. The numbers below come from that run. They are not a ranking of every background-removal model, and they are not a claim about a production studio pipeline.
I work on ProfessionalHeadshot, an AI professional headshot generator. The script in the companion folder is a teaching implementation. I did not send these plates through that product.
A known matte, then four plates
The subject is a 1024×1280 transparent PNG of a woman in a navy blazer, with loose hair and visible flyaways. The alpha came with the file. It is not a light-stage capture. Core pixels top out at 254 rather than 255; I stretched that upper range so 254 becomes 255 and left true zeros at zero. I am treating that alpha as ground truth because I need a matte I did not get from the models I am about to score.
Four source plates, all at 1024×1280:
- saturated green, RGB (0, 204, 0)
- white
- light gray, RGB (210, 210, 210)
- a soft office interior, cropped to cover the frame
The replace step is the obvious one. Predict an alpha, then composite:
import numpy as np
def composite(fg_rgb, bg_rgb, alpha):
"""alpha is a float array in [0, 1], shape HxW."""
a = alpha[..., None]
return np.clip(fg_rgb * a + bg_rgb * (1.0 - a), 0, 255).astype(np.uint8)
When I apply a predicted matte, I use the plate itself as the foreground color. That is what you have if you start from a finished photo. It also means a green plate already has green mixed into the hair-edge pixels. Even the ground-truth alpha, applied to that plate and put onto navy, leaves a little green fringe. A production pipeline would de-spill. This one does not, on purpose: the fringe is part of what I am measuring.
Three mattes
I ran two ONNX segmenters from the rembg weight set, plus OpenCV GrabCut as a rectangle-initialized baseline. OpenCV 4.14.0, ONNX Runtime 1.32.0, CPU.
| Method | Weights | Input | Notes |
|---|---|---|---|
| U2-Net | u2net.onnx |
320×320, ImageNet mean/std | rembg's default model family |
| IS-Net general-use | isnet-general-use.onnx |
1024×1024, RGB in [0, 1] | ImageNet normalization produced empty masks on the green and office plates, so I did not use it for this ONNX |
| GrabCut | none | full image, 5 iterations, GC_INIT_WITH_RECT
|
rectangle inset 6% / 2% / 88% / 96%; no use of the ground-truth matte |
I found that IS-Net preprocessing the hard way. With the same ImageNet normalization I used for U2-Net, IS-Net's foreground on the green plate shrank to 0.1% of the pixels and the office plate to 11.6%. The same weights, given a [0, 1] tensor, produced a complete matte on every plate. I am reporting the [0, 1] run, and I am saying what failed first so nobody copies the wrong prep.
What I measured
A pixel is foreground at alpha ≥ 0.5. Against the ground-truth matte I report IoU, then two edge numbers, then two hair numbers.
The edge band is every pixel within six pixels of the ground-truth boundary, inside or outside. Edge MAE is the mean absolute RGB error, in that band, between the predicted composite and the ground-truth composite on the same destination. Leftover source-background is a soft score in the same band: predicted alpha weighted by how close the plate pixel still is to the source background color. It is high when the model keeps old-background pixels as foreground.
The hair region is the ground-truth foreground above the face. I used YuNet to place a face box on the cutout, at score 0.909, and took ground-truth pixels above a line 18% of the way down that box, so the forehead stays with the face and the crown stays with the hair. That region is 152,312 pixels. Hair FN is the fraction of those pixels the model called background: chopped or missing hair. Hair FP is the fraction of an 8-pixel fringe just outside the hair that the model called foreground: leftover background in the flyaways.
Those two hair numbers can trade off. A tight matte chops strands and looks clean in the fringe. A loose matte keeps the strands and also keeps a bit of the room.
The run
IoU, leftover, and the hair rates depend only on the matte, so they are the same for both destinations. Edge MAE below is for the navy destination, where a leftover fringe is easiest to see.
| Source plate | Method | IoU | Edge MAE (navy) | Leftover src-bg | Hair FN | Hair FP |
|---|---|---|---|---|---|---|
| Green | U2-Net | 0.9907 | 12.34 | 0.0063 | 0.0104 (1,578 px) | 0.0425 (417) |
| Green | IS-Net | 0.9873 | 18.70 | 0.0016 | 0.0330 (5,025) | 0.0000 (0) |
| Green | GrabCut | 0.6469 | 28.23 | 0.0185 | 0.0000 (0) | 1.0000 (9,820) |
| White | U2-Net | 0.9940 | 21.95 | 0.0097 | 0.0111 (1,697) | 0.0209 (205) |
| White | IS-Net | 0.9904 | 29.51 | 0.0061 | 0.0225 (3,432) | 0.0017 (17) |
| White | GrabCut | 0.6956 | 51.65 | 0.0000 | 0.0381 (5,805) | 0.0000 (0) |
| Gray | U2-Net | 0.9926 | 21.90 | 0.0185 | 0.0143 (2,171) | 0.0061 (60) |
| Gray | IS-Net | 0.9894 | 26.53 | 0.0124 | 0.0247 (3,757) | 0.0016 (16) |
| Gray | GrabCut | 0.7052 | 41.77 | 0.0075 | 0.0330 (5,030) | 0.0000 (0) |
| Office | U2-Net | 0.9876 | 18.79 | 0.0490 | 0.0047 (713) | 0.3795 (3,727) |
| Office | IS-Net | 0.9887 | 20.02 | 0.0362 | 0.0168 (2,557) | 0.0513 (504) |
| Office | GrabCut | 0.6651 | 38.64 | 0.0395 | 0.0261 (3,978) | 0.3537 (3,473) |
U2-Net and IS-Net are close on IoU, both in the 0.987 to 0.994 range on these plates. GrabCut is not: 0.647 to 0.705. The interesting differences are at the hairline.
On green, GrabCut keeps the entire 9,820-pixel hair fringe as foreground, which is why the close-up has a green cap around the crown. Hair FN is zero: it did not chop the hair, it just failed to let go of the screen. U2-Net leaves a smaller green blob at the parting (hair FP 0.0425, leftover 0.0063) and drops 1.04% of the hair pixels. IS-Net is tighter. Leftover on green is 0.0016, hair FP is zero, and hair FN rises to 3.30%. You can see that in the third frame of the close-up: a cleaner edge, a little less of the finest hair.
White and gray look like the same tradeoff at a smaller scale. U2-Net has the higher IoU and the lower hair FN. IS-Net has the lower leftover score and almost no fringe false positives. GrabCut's edge MAE on white, 51.65, is the worst number in the navy column; the matte is ragged even when the leftover-color score happens to be low.
The office plate is where leftover hair, in the everyday sense, shows up. U2-Net's IoU is still 0.9876, and it chops almost no hair (FN 0.47%). It also keeps 37.95% of the hair fringe as foreground: a smear of shelf and plant sitting in the flyaways. IS-Net's hair FP on the same plate is 5.13%, with a higher FN (1.68%). GrabCut again leaves a chunk of the room (hair FP 0.3537) and a hard cut.
If I had reported only IoU, U2-Net and IS-Net would look interchangeable, and the office plate would not stand out. The hair-fringe rate is the number I would actually look at before I used a matte on a busy background.
That matters for a headshot in a way it does not for a quick product cutout. A LinkedIn or resume portrait is often viewed as a small circle. A leftover plant or shelf in the crown is still there at 400 pixels, and it is still there when the background was supposed to be a clean navy or gray. The office plate is also the case that most resembles a real photo, rather than a studio sweep. A 0.99 IoU can hide a fringe you would not accept on a profile picture.
What I would actually run
For a solid white or gray headshot background, either ONNX matte is usable on this subject, with the usual hairline caveat, and I would still inspect a close crop of the crown. For a plate that already has a room in it, I would not trust IoU alone. I would look at the fringe. GrabCut, initialized with a loose rectangle, is not a substitute for a hair matte.
Choosing a background in the first place is a different problem from keying one. ProfessionalHeadshot is an AI professional headshot generator, and its guide to choosing a headshot background walks through when a white or gray studio, a blurred office, or an outdoor scene is the safer default. I did not use that guide as a source of measurements, and I did not score its outputs here.
If you want the plates, the masks, and out/metrics.json, they are in the companion folder I used for this run. The numbers describe one generated cutout, four backgrounds, and three methods, at the settings above. A different hair type, a different office, or a de-spill step would change the residuals. I would re-measure rather than reuse the table.
Top comments (0)