DEV Community

Cover image for Comparing rembg Models on Edges and Residual Background, With Ground Truth
Jacob Yang
Jacob Yang

Posted on

Comparing rembg Models on Edges and Residual Background, With Ground Truth

A ceramic mug on a patterned tablecloth, with a paper cutout of the same mug standing beside it.

danielgatis/rembg is a small Python library that wraps a set of public background-removal weights and returns a cutout. People usually just call the project rembg. The interesting part is not that it runs. It is that the default package ships more than one official ONNX file, and those files do not fail in the same place.

I ran three of those official weights on fixtures whose alpha I already knew: u2net, silueta, and isnet-general-use. The weights came from the project's GitHub release path, https://github.com/danielgatis/rembg/releases/download/v0.0.0/, and the MD5 checksums matched the values in rembg's session files as they stood on 19 November 2026:

Weight Official MD5 Size
u2net.onnx 60024c5c889badc19c04ad937298a77b 175,997,641 bytes
silueta.onnx 55e59e0d8062d2f5d013f4725ee84782 44,173,029 bytes
isnet-general-use.onnx fc16ebd8b0c10d971d3513d564d01e29 178,648,008 bytes

Preprocessing and postprocessing follow rembg's own session code, fetched the same day. u2net and silueta resize to 320×320 with ImageNet mean and standard deviation. isnet-general-use resizes to 1024×1024 with mean 0.5 and standard deviation 1.0. Each output map is min-max normalized and resized back to the source with LANCZOS. This was a local ONNX Runtime CPU session. It is not a benchmark of a hosted API, and it is not a test of iMideo.

Fixtures with known alpha

I did not use a camera photograph as ground truth, because I would have had to invent a mask for it. The three fixtures are synthetic, and the alpha is exact because I drew it.

The first is a hard-edged cyan disk with a rectangular stem on a magenta field. The mask is binary. It is the easy case: a sharp boundary and a color that never appears in the background.

The second is the case I actually care about. A beige blob was drawn at 2× and downsampled so the body edge is anti-aliased, then one or two pixel fibers were drawn at native resolution, 390 of which are "thin" by a simple neighbor count (foreground pixels with three or fewer foreground neighbors). If a model smooths the silhouette, those fibers are the first thing to go.

The third is a composite, not a photograph: a procedural fabric background and a mug-shaped subject with a 1.2 pixel Gaussian rim. It looks a little more like a product cutout than the first two, but the alpha is still constructed. I am not going to pretend it is a studio pack shot.

All fixtures are 512×512. The code, the PNGs, and the raw numbers are in the folder that accompanies this post.

What I measured

For each model and fixture I recorded:

  • mean absolute error of the predicted alpha against the true alpha
  • intersection-over-union of both maps thresholded at 0.5
  • residual background: mean predicted alpha on pixels whose true alpha is under 0.05, plus the 90th percentile
  • missed foreground: mean of 1 minus predicted alpha on true foreground
  • edge-band MAE, in a 5 pixel dilation-minus-erosion ring around the hard ground-truth mask
  • thin-structure recall at 0.5, when the fixture has thin pixels
  • wall-clock inference on this machine, in seconds, including the resize

I also saved a checkerboard composite and a residual overlay. In the overlay, leftover background (predicted alpha ≥ 0.3 on true background) would be red, and missed foreground would be cyan.

Results

Fixture Model MAE IoU @ 0.5 Residual bg mean Edge MAE (5 px) Thin recall Time (s)
hard_disk u2net 0.004239 0.996215 0.003223 0.078396 n/a (2 px) 0.354
hard_disk silueta 0.005424 0.992428 0.001446 0.101091 n/a (2 px) 0.852
hard_disk isnet-general-use 0.002348 0.999796 0.000987 0.027852 n/a (2 px) 0.869
soft_fibers u2net 0.004782 0.991666 0.002093 0.064224 0.176923 0.415
soft_fibers silueta 0.005827 0.989781 0.000533 0.074804 0.000000 0.612
soft_fibers isnet-general-use 0.003493 0.997170 0.001989 0.031140 0.979487 1.171
composite_mug u2net 0.004534 0.983562 0.000510 0.087099 n/a (0 px) 0.186
composite_mug silueta 0.002759 0.993771 0.000401 0.046489 n/a (0 px) 0.506
composite_mug isnet-general-use 0.002845 0.994359 0.000146 0.046979 n/a (0 px) 0.741

The hard-disk "thin" column is not useful. Only two pixels met the neighbor test, which is a corner artifact, not a fiber. I left the raw counts in the JSON and I am not reading a recall out of two pixels. The mug fixture has no thin pixels at all.

Contact sheet of three synthetic fixtures. Columns are the source, the ground-truth checkerboard, then u2net, silueta, and isnet-general-use composites.

The same three fixtures with residual overlays. Cyan marks missed foreground, including dropped fibers. Red would mark leftover background.

What the files actually show

On the hard disk, all three models are already close. Residual background is low (mean alpha 0.001 to 0.003 on true background, and the 90th percentile is 0.0 in every hard-disk and fiber run). The difference is at the edge. isnet-general-use has an edge-band MAE of 0.027852 and an IoU of 0.999796. u2net is 0.078396 and 0.996215. silueta is 0.101091 and 0.992428. You can stare at the composites and barely see it. The metric is the part that moves.

The fiber fixture is where the models diverge in a way you can see. isnet-general-use kept 0.979487 of the 390 thin pixels at a 0.5 threshold, and its edge-band MAE is 0.031140. u2net kept 0.176923 of those pixels. silueta kept none of them. In the residual overlay the missed fibers show up as cyan lines around the beige body for u2net and silueta, and they are almost gone for IS-Net. That is the result I would not want to bury in a single IoU number: IoU on this fixture is 0.99 for all three, because the fibers are a tiny fraction of the blob.

The mug composite is a middle case. Residual background is very low for all three, lowest for isnet-general-use at a mean of 0.000146. silueta and isnet-general-use are close on MAE and IoU; u2net trails slightly on IoU (0.983562 against 0.993771 and 0.994359) and has the highest edge-band MAE. There are no fibers to lose, so the "thin" test does not apply.

Inference time is only a description of this CPU run. u2net at 320 was the fastest of the three (0.186 to 0.415 s). isnet-general-use at 1024 was the slowest (0.741 to 1.171 s). silueta sits in between and, on the fiber fixture, was the most willing to drop the thin structure. I would not turn those times into a ranking of "production latency." They include Python overhead, and I did not batch, warm up, or run a GPU provider.

Limitations I want to keep visible

These are synthetic fixtures. A real hair photograph, a glass, a transparent object, or a subject that shares color with the background is a different problem, and I do not have ground truth for those here. A high IoU on a blob can hide a total miss on a one-pixel fiber. Residual background was low on this set; that may not hold on a busy photograph. I ran the three official rembg weights I had checksum-verified. I did not run BiRefNet, SAM, or u2netp, and I am not going to guess how they would have scored.

One preprocessing detail is worth repeating if you reimplement this without the rembg package. The official normalize function resizes with LANCZOS, then divides the RGB array by max(np.max(im_ary), 1e-6) before subtracting the mean. On a normal photograph that denominator is 255. On a darker synthetic fixture it can be the actual peak value. I copied that behavior rather than "correcting" it, because the goal was to run rembg's session, not a private variant.

The 90th percentile of residual background alpha was 0.0 on the hard-disk and fiber runs. That means at least nine tenths of the true-background pixels were predicted as empty, which is consistent with the clean magenta and green fields. It is not evidence that the same models will leave a busy photograph equally clean. Mean residual on the mug background was also low, even though the fabric is colorful, but the subject and the fabric do not share a color the way a brown coat would on a brown sofa.

The U-2-Net architecture is described in Qin et al., 2020. I am citing the paper as the source of that model family, not as a score for this reimplementation. The IS-Net weight is rembg's isnet-general-use release file. I did not retrain anything.

I work on iMideo AI, an AI video and image creation platform, which also has a hosted image background remover for people who would rather not install rembg and the ONNX files themselves. The live page, read on 19 November 2026, says you upload a PNG, JPG, or WebP up to 10 MB and get a transparent PNG at the original resolution, and that hair and fur are kept where the source image supports it. I did not send these fixtures through that product. The table above is only the local rembg session.

If you are choosing a rembg weight for a subject with thin structure, the number I would look at first on this set is not IoU. It is the fiber recall, and then the edge-band error. On these fixtures, isnet-general-use kept the fibers. silueta did not. u2net kept some of them and missed the rest.

Top comments (1)

Collapse
 
gautamtalksdev profile image
Gautam K •

Ground truth comparisons are what most rembg write-ups are missing, so this is useful. Edges and leftover background are exactly where the models differ in practice. Which model would you pick as the default for product photos?