6 Background Removal API Tests I Run Before Shipping Video

Dave Gordon — Fri, 21 Aug 2026 13:33:50 +0000

TL;DR:

Real-time background removal is a frame budget problem before it is a quality problem.

Six tests, in this order:

- does it fit the budget on your worst supported phone,
- did the run actually use the accelerator you think it did,
- what happens at the hair line,
- does the mask flicker between frames,
- what breaks when the lighting is real,
- and what does it cost you once you count platforms instead of license fees.

Most shortlists lose half their candidates on tests 1 and 5.

I have swapped segmentation backends three times across two camera apps, and every swap started the same way: the library was fine in the sample project and came apart on a real device with a real user in front of it. So I stopped evaluating these things by looking at output stills.

This is the harness, not the results table. I am not publishing my numbers here, because my numbers are about my pipeline and my support matrix, and neither of those is yours. A frame budget is derived from your capture rate and your effects stack, and a mid range phone in one support range is a flagship in another.

What transfers is the setup and the order the tests run in. Copy the rig, run your own candidates through it, and the numbers you get will actually apply to the thing you are shipping.

The rig

Before the first test, fix these so the results mean something across candidates.

Three device tiers, named, not averaged

A current or previous cycle flagship, a mid range phone around three years old, and the cheapest device with real share in your own analytics. Record the model, OS version and chipset for each. Score every test per device. An average across the matrix hides the phone your median user is holding, which is the only phone the decision actually turns on.

One app state, held constant

Camera open, your effects stack running, screen at full brightness, no other app in the foreground. Candidates measured in different app states are not comparable, and the tempting shortcut of benchmarking inference in isolation produces the number that looks best and matters least.

A warm device, and a stated warm-up

Decide how long the app records before you take the sustained reading, write that duration down, and use the same one every time. Thermal state is the variable that quietly decides half of these comparisons.

One fixed clip set

The same recordings, reused for every candidate: a plain wall clip, a still subject clip, and the lighting cases from Test 5. New footage per candidate means you are scoring the footage.

Pinned versions

SDK or model version for every candidate, written next to every result. Vendors ship model updates that move these numbers, and a result without a version cannot be reproduced or re-checked later.

Why does the frame budget decide this before quality does?

Most background removal comparisons open with mask quality. That is the wrong end. If inference does not fit inside a frame, quality is irrelevant, because you are already dropping frames and the preview stutters.

Work out your own budget before you look at a single demo:

Take your target capture rate. 30 fps for most social and editor video, 60 fps if the feature has to feel like the native camera.
Divide 1,000 ms by that rate. 30 fps gives you 33.3 ms per frame. 60 fps gives you 16.7 ms.
Subtract the rest of the pipeline. Camera capture, color space conversion, your own filters or effects, the render pass, and encoding all take a slice.
What is left is your segmentation budget. Run that subtraction rather than assuming you have the whole frame, because in the pipelines I have worked in, the remainder has never been more than half.
Compare that number against measured inference time on your slowest supported device, not your average one and not the phone on your desk.

Write that number down. Every test below is scored against it.

Test 1: Does it hold the budget on your worst target device?

Run the candidate on the cheapest and oldest tier, in the app state fixed above.

Three things to capture.

Cold inference time

The first frames after model load are always slower. If your feature is triggered mid session, users feel this directly.

Sustained inference time after thermal throttling

This is the number that matters and the one almost nobody publishes. Measure it rather than extrapolating from the cold figure, and expect the two to be far enough apart that the cold figure is useless for capacity planning. If a vendor publishes one inference time without saying whether the device was warm, assume it is the cold one.

Memory ceiling

Segmentation model plus video buffers plus encoder is where low RAM Android devices start getting killed in the background. Watch dumpsys meminfo for your process across a long recording rather than reading a peak off a profiler graph once.

Test 2: Did the run you just measured actually use the accelerator you think it did?

This test exists because Test 1 produces a plausible looking number whether or not it measured the pipeline you intended.

The GPU delegate fails to initialise, the library falls back to CPU without raising anything to your code, the run completes, and you write down a figure that describes a completely different execution path. Nothing in the output tells you. The mask still looks correct, the app does not crash, and the number is wrong in a direction that will not reproduce on the next build.

ERROR: Failed to apply GPU delegate.

If that line is in your logs, every timing you collected in that session is CPU timing. Two habits fix this permanently:

Assert the execution path per run. Log which delegate or backend was actually created, and discard any run where the assertion is missing rather than any run that looks slow. Slow runs are data. Runs of unknown provenance are not.
Fail closed. A benchmark harness that silently completes when initialisation failed is worse than one that crashes, because it produces numbers you will act on.

The same applies to any library that quietly reduces input resolution or frame stride under load. If the candidate has an adaptive mode, either turn it off for the measurement or record what it chose, because otherwise you are comparing two vendors that each decided independently what to sacrifice.

Test 3: What happens at the hair line?

Every segmentation model produces a clean silhouette. The difference between them lives in about forty pixels around the edge.

Use the plain wall clip and score four cases:

Loose or curly hair. Naive segmentation cuts a helmet shape and drops the strands entirely.
Glasses frames, especially thin metal ones, which models frequently classify as background and slice through.
An object held in hand, like a mug or a phone. Person only models will amputate it. Whether that counts as a bug depends on your use case, so decide deliberately rather than finding out in review.
The gap between arm and torso when someone puts a hand on their hip. Soft mask models tend to fill it in.

Score these against a hard edged replacement background, not a blur. Blur hides everything, which is exactly why so many demos use it.

Test 4: Does the mask flicker between frames?

This is the failure mode that gets caught in QA rather than in evaluation, and it is the most expensive one to fix late.

A per frame image segmentation model has no memory of the previous frame. Feed it video and the mask boundary jitters frame to frame even when the subject is completely still. The stills look perfect. The clip looks cheap.

Two ways a library can address it.

Temporal architecture

Robust Video Matting is the reference example, in that it carries state between frames with a recurrent design instead of treating each one independently. Check the license before you get attached to it: the code was re released under GPL-3.0 in September 2021, which for most proprietary mobile apps means you are studying the architecture rather than shipping the repo.

Your own smoothing

You can apply temporal filtering to a per frame mask yourself, but you are trading latency for stability, and that trade comes straight out of the budget you calculated above.

The test is simple. Take the still subject clip, thirty seconds of someone barely moving, then step through the export frame by frame and watch the boundary. If it breathes, you either fix it or you ship it.

Test 5: What breaks when the room does?

Evaluation footage is always shot in good light. Your users are on a couch at 9pm with a window behind them.

Run each candidate through:

Backlight, with a bright window behind the subject. Classic failure: the person is segmented as background.
Low light, where sensor noise destabilises the edge and Test 4 gets much worse.
A busy background with furniture, a doorway, or a poster of a person on the wall.
Two people in frame, where some models pick one, some merge both, and some alternate between the two across frames.
Movement toward and away from the camera, which changes subject scale and is where fixed input resolutions show their limits.

If you automate one part of this harness, automate this one. It is the section that regresses silently when a vendor ships a model update.

Test 6: What does it actually cost you per platform?

The license fee is the visible number and usually not the deciding one. Count the real cost like this:

List every platform you ship on. iOS, Android, web, desktop, and any framework layer such as React Native or Flutter.
Check what the candidate covers natively. Apple's Vision framework offers person segmentation with selectable quality levels, and it is iOS and macOS only, so an Android build needs a second solution and a second set of edge case behaviour to test.
Multiply the platforms you have to solve separately by the integration and QA time for one platform. That multiplier, not the sticker price, is where the budget goes.
Add the ongoing cost. Two segmentation backends means two model update cycles, two sets of device regressions, and visible quality differences between your iOS and Android users.
Check the deployment model, which is where the arithmetic gets decisive.

Can you use a cloud background removal API for live video?

Not realistically. Cloud background removal is priced per image and needs an upload per frame, so a 30 fps preview is 30 requests per second per user with a network round trip on every one. That is the right tool for batch photo processing and the wrong one for a live camera preview, and it fails on arithmetic before you get to quality.

For the free, on device end of the shortlist, MediaPipe Image Segmenter is Apache 2.0, and ML Kit selfie segmentation is a closed source SDK that is free to use. Both run on device across iOS and Android, which makes them the honest baseline. Anything commercial has to beat them on the tests above by enough to justify the line item, and often it does, because Tests 1, 4 and 5 are where a general purpose model gives ground to one tuned for video.

The scorecard

Test	What it measures	Pass condition
1. Frame budget on worst device	Cold and sustained inference time, memory ceiling	Sustained time fits the segmentation budget on the slowest supported device, not the average one
2. Execution path	Whether the accelerator you configured is the one that ran	Delegate asserted in the log on every run, runs without the assertion discarded
3. Hair line	Edge quality on hair, thin frames, held objects, arm to torso gaps	Scored against a hard edged background, not a blur
4. Temporal stability	Mask jitter across frames on a still subject	Boundary holds frame to frame, or the smoothing cost is inside the budget
5. Real conditions	Backlight, low light, busy background, two people, scale change	No condition drops below your quality floor, and the suite is automated
6. Cost per platform	Platforms solved separately, integration and QA multiplier, deployment model	One integration covers your matrix, or the second one is budgeted

Pick your slowest supported device and your worst lighting scenario first, then evaluate. Run these six in order and you will usually eliminate half the shortlist before pricing comes up, and you will eliminate it for reasons you can put in front of a product manager.

If you run this harness on something I have not tried, I would like to hear what broke.

DEV Community: Dave Gordon