DEV Community: Dave Gordon

Where Background Removal APIs Send Your Users' Video

Dave Gordon — Mon, 14 Sep 2026 19:13:00 +0000

Picking a background removal API is an engineering decision right up until it stops being one. The masks hold on hair and glasses, the latency fits, the integration is a two day job, and then a security reviewer at your own company or at your largest customer sends over a questionnaire asking where the frames go and who keeps them afterwards. The API reference does not cover any of that.

TL;DR

Remove.bg puts no duration on how long the images sent to it are kept, and it caps server logs at three months.
Photoroom's model training clause covers photos people send to its app, and the same text excludes API images.
VideoBGRemover publishes a full erasure sequence: live while the job is active, gone on a verified request, production erasure inside 30 days, backups seven days beyond.
Files on the fal CDN are readable by anyone holding the link unless you attach an access control list, and inputs inherit the account default rather than the header on the inference call.
Banuba's Face AR SDK is licensed by a token in .txt format with a published expiry ladder, and the same page never says whether the SDK phones home to check it.
One of the five is sales gated. The other four hand you a key at sign-up.

Everything here was read off vendor pages on 8 September 2026

I ran no tests for this one and there is not a millisecond anywhere below. Every claim comes from a vendor's own live pages as they stood on 8 September 2026, linked next to the sentence it supports. Those pages get edited, often quietly, and any figure that decides a build should come from the vendor in writing, addressed to you.

Does the video frame leave the device?

Four of the five are hosted services, so the media crosses the network before anything happens to it. Remove.bg's privacy policy, last updated 16 July 2025, set out the sequence plainly when I read it on 8 September 2026: images go up securely, get processed, and are provided for download. Photoroom, VideoBGRemover and Veed running on fal.ai work the same way.

Banuba's Face AR SDK is the one product here that is not a hosted service, since the processing happens inside your app rather than on a server you call. That settles the media path and leaves the license path open, which is where its published material stops short.
For the hosted four, the honest questionnaire entry is yes. Everything else is argued after that.

What does each vendor keep, and for how long?

	Banuba Face AR SDK	Remove.bg	Photoroom	VideoBGRemover	Veed on fal.ai
Where processing happens	On the device, inside your app	Vendor servers	Vendor servers	Vendor servers	fal's platform
Does user media leave the device	No	Yes	Yes	Yes	Yes
What is being compared	On-device SDK, licensed by a token in the client app	Hosted REST image service	Hosted REST service plus a consumer app	Hosted video service	A model run on fal's platform
Access	Sales gated: demo and commercial tokens are both issued by a person	Open self-serve with an API key	Open self-serve, custom enterprise pricing	Open self-serve, enterprise DPA available	Open self-serve
How long the vendor holds media	No media reaches a vendor server	"Delete them shortly after", no duration given	Not stated	Live while the account or job is active, erased on job deletion or a verified request	No CDN expiry stated
Tail after deletion, and logs	No media reaches a vendor server. Server logs not applicable	Server logs kept at most three months	Not stated	Production erasure inside 30 days, soft-delete systems and rolling backups seven days beyond	Inputs and outputs kept 30 days by default
Training use	No media reaches a vendor server	Not stated	Training row covers app photos, API images carved out	Does not train or improve models on customer content	Not stated
Named third-party audit	Not named	Not named	SOC 2 Type 2	Not named	Not named
Contract instrument named	Not named, terms go through an account manager	EU and UK representative named (EDPO), rights listed	Not named	DPA incorporating the 2021 Standard Contractual Clauses	Not named
Control you can set yourself	Keep the token on your own server rather than in the app bundle	Not offered	Not offered	Job deletion and verified erasure requests	`X-Fal-Store-IO: 0` on the request, ACL on the upload
Still open on their own pages	Whether the SDK makes a network call to validate the token	How long processed images are actually kept	Retention duration for API media	No named outside audit	CDN file expiry, and what happens to assets at account end

Every cell above is what that vendor's own published pages said when I checked them on 8 September 2026, and an empty-looking cell means I found nothing on the subject, not that the vendor does the opposite. None of it is a measurement.

The first row exists so the first column does not get scanned as like-for-like. An on-device SDK and a hosted video service fail a questionnaire in completely different places, and putting them side by side is only useful if you keep that in view.

Access splits four to one. Remove.bg, Photoroom, VideoBGRemover and fal all issue a working key at sign-up. Banuba is the gated one: a demo credential comes from a sales manager or the website request form, and a commercial credential comes from an account manager after payment. Sales gating is not automatically a problem, but it does mean your evaluation timeline includes somebody else's calendar.

The most specific writing about retention here belongs to VideoBGRemover. Its privacy page says uploaded videos, masks, previews and outputs stay available while the account or job is active and are erased when you delete the job or send a verified request, that production erasure completes within 30 days, and that soft-delete systems and rolling backups may hold copies for seven days beyond that. No other page here puts a tail on the number. Remove.bg's three month ceiling on server logs and the 30 day payload default in fal's media expiration docs are real figures too, though both of those describe logs and JSON rather than the media itself.

On model training, Photoroom and VideoBGRemover arrive at similar comfort from opposite directions. Photoroom carves API traffic out of model improvement in its own legal text. VideoBGRemover rules out training on customer content across the whole service.

Photoroom is the only one of the five naming an outside auditor: its API page states SOC 2 Type 2. If your reviewer's checklist has a certification line on it, that is currently a one horse race here.

For a contract instrument, VideoBGRemover names an enterprise DPA incorporating the European Commission's 2021 Standard Contractual Clauses, and Remove.bg names EDPO as its EU and UK representative alongside the usual list of data-subject rights. The other three leave that conversation to sales.

fal is alone in handing you controls you operate yourself. Everyone else describes their own conduct and asks you to accept the description, which is a weaker artifact to walk into a review with, however honest the description turns out to be.

Can anyone else open the file the API handed back?

On fal, the default is yes. fal's file access control docs state that every object on the fal CDN carries an access control list, and that by default CDN URLs are reachable by anyone who has the link. A long random URL does not make an object private, and links leak through logs, support tickets and screenshots.

Inputs are a separate case on the same page. The SDKs push them to the CDN before the inference call runs, so they are not governed by the ACL header you set on that call and instead inherit the account default. To constrain an input you set the ACL on the upload itself.

To size your own exposure, with named inputs you can redo on paper:

Start with clips per user per day. Mine here is one, and it is yours to substitute.
Multiply by daily active users who touch the feature. I am using 500, an illustrative figure to replace with your own.
500 x 1 = 500 input objects landing on the CDN each day.
CDN URLs are public by default, so unless an ACL is attached at upload time, those are 500 publicly reachable URLs added per day.
The JSON payloads default to 30 days, so 500 x 30 = 15,000 request records held in the history at any steady state moment.

In elapsed terms, that history never empties while the feature is switched on, because the oldest record ages out 30 days after it was created and a fresh one arrives the same day.

Are my users' images used to train the vendor's models?

Photoroom's privacy policy carries a retention table with a row titled "Training our AI model", covering photos and images that people sent to Photoroom, on a legitimate interest basis, held for as long as training requires. The same document then states that this model improvement does not apply to images processed through the API. That clause is about app users, not API traffic, and getting the distinction backwards in a questionnaire will cost you a week of email.

VideoBGRemover takes the other route and states flatly that it does not use content people send it, or content it generates, to train or improve machine learning models. That single sentence covers the whole service.

Remove.bg's policy, fal's legal pages and the Banuba licensing page say nothing either way, going by what was up on 8 September 2026. That absence does not amount to a denial, and it stays open until the vendor answers in email.

Who else on my team can see the requests?

fal's privacy policy is unusually direct about this. If you are a member of a Team Account, it says, other members of that Team Account may view billing information, API keys, and AI model requests, including any input and output data.

Think that through before you invite the contractor. Whoever sits in the account can read the media your users sent, and your key too.

The other four say nothing about who inside a customer account can read request bodies, in anything they published as of 8 September 2026. If the question comes up, and it always has for me, you will be relaying an email rather than pointing at a URL.

What happens to the feature when the license lapses?

Banuba is the only one of the five that publishes what happens after the license runs out. Its token management documentation describes the token as an automatically generated set of characters in .txt format, unique to each client, that activates the licensed SDK functionality in the client app.

The expiry ladder on that page has three rungs. A credential that has not expired means the SDK works as expected. One expired by a month or less still works, with a watermark displayed. Past a month, the functionality will not work.

The same page recommends keeping the credential on your own server instead of in the app bundle, and the reasoning is operational: ship it inside the binary and every renewal means a new build queued for App Store or Play Store approval.

What the page does not state is whether the SDK contacts Banuba at runtime to validate the credential. The expiry ladder is written down. Whether a network call sits behind it is left open, so a reviewer asking about outbound traffic needs that confirmed by the vendor.

Among the hosted four, only VideoBGRemover ties asset availability to account state in its published terms, saying that videos, masks, previews and outputs stay available while the account or job is active. For Remove.bg, Photoroom and fal, I found nothing published about what becomes of stored assets once an account ends.

What will the vendor put in writing?

Photoroom offers an attestation from an outside auditor plus an explicit carve-out for API traffic. VideoBGRemover offers a DPA built on the 2021 Standard Contractual Clauses for enterprise customers under GDPR. Remove.bg names a representative for EU and UK data subjects and enumerates the rights those people hold. fal offers something different in kind, two controls you operate yourself: the store header on the request and the ACL on the upload. Banuba issues its commercial license through an account manager after payment, so the terms conversation happens with a person and whatever you need has to end up in that thread.

Two of the five publish noticeably less than the rest. Remove.bg gives no duration whatsoever for the images sent to it, only that they go "shortly after", and shortly is not a number your reviewer can log. Banuba's material covers the license credential thoroughly and stops before the SDK's runtime network behavior, which leaves the most basic question about an on-device product unresolved on the public web.

Ask for the four answers in writing before you sign

Four items close most of the gaps above, and they fit in one email.

A number, in days, for how long processed media is held, and a second number for the backup tail after removal.
Whether traffic through the API is excluded from model training, stated separately from whatever applies to their consumer app.
A named list of who inside your own account can read request inputs and outputs.
What becomes of stored assets when the contract or license ends.

Each of the five publishes some of those, and none of the five had all four on the pages I went through on 8 September 2026. Whichever come back thin are the ones to raise before the integration lands in a sprint, since they get harder to ask once the feature ships.

5 Places a Real-Time Camera Filter Drops Frames

Dave Gordon — Thu, 03 Sep 2026 11:42:31 +0000

A real-time camera filter has about 33 milliseconds per frame at 30 FPS. When a filter that looked fine in a demo stutters on a real handset, the loss is almost always in one of five places: format conversion, frame backpressure, where inference actually executes, synchronous readback, or sustained thermal load. Here is how I find which one.

TL;DR

The budget is arithmetic you can do in your head: 1000 / 30 = 33.3 ms per frame, and 1000 / 60 = 16.7 ms.
Converting a camera frame to RGBA before anything else executes is the most common self-inflicted cost, and a byte count exposes it before you open a profiler.
If you never chose a backpressure strategy you already have the non-blocking one, which puts a leaked frame reference ahead of the strategy on the list of suspects.
A model that reports a plausible inference time can still be executing somewhere you did not pick, which makes the number describe a different pipeline from the one you are shipping.
Reading pixels back synchronously drains the queue you spent the whole frame filling.
Thermal decay does not show up in a 30 second run, which is why most of these pipelines pass their own tests.

No device numbers in this piece

I have not benchmarked anyone's SDK here and there are no handset figures below. A number without a named device, an OS build and a thermal state cannot be reused by anyone reading it. What follows is the order I work through when a camera pipeline misses its budget, plus the arithmetic that tells you whether a given stage can possibly fit before you go looking for it in a trace.

What I did do is verify every mechanism against primary documentation and link it: the image analysis contract and frame format from Android's own docs, the delegate behaviour from the TensorFlow Lite repository and its issue tracker, the profiling tools from Google and Apple, and the readback stall from a handset vendor's own optimisation guidance. The arithmetic is reproducible on paper, two divisions and a multiplication, so if any of it is wrong you can check it in about a minute.

The five places below are ordered by how often they turn out to be the answer in my own work. Ranking them by theoretical cost would give a different order.

The budget, before you profile anything

Two divisions set the ceiling for everything else.

At 30 FPS you have 1000 / 30 = 33.3 ms to receive a frame, convert it, run whatever model you are running, render, and composite. At 60 FPS that halves to 1000 / 60 = 16.7 ms. Each stage you add spends part of that, and the preview is only as smooth as its slowest frame.

Resolution is the other multiplier, and it is worth doing on paper first. A 1080p frame is 1920 x 1080 = 2,073,600 pixels. A 720p frame is 1280 x 720 = 921,600 pixels. That is a factor of 2.25, so any per-pixel stage costs 2.25 times more at 1080p than at 720p before you change a line of code.

I start here because the arithmetic frequently ends the investigation on its own. If a stage has to touch every pixel of a 1080p frame and you have four such stages, the budget was gone before the model loaded.

Do I need 60 FPS, or is 30 enough?

For a camera filter that a person is looking at while moving their own face, 30 is usually enough and 60 is usually not worth what it costs, because halving the budget to 16.7 ms tends to force a resolution cut that is more visible than the smoothness gain.

The exception is anything the user physically tracks with their hand or head, where the extra samples do register. I would decide this by shipping 30 and instrumenting it before assuming the answer, since the 95th percentile at 30 FPS is a better predictor of perceived quality than the target rate.

1. The format conversion you did not know you were paying for

Camera hardware on Android hands you YUV_420_888, a planar YCbCr format with 8 bits per sample and chroma subsampled 4:2:0. Most image processing code, and most tutorial code, wants RGBA instead. So a conversion gets inserted, often before anyone has decided whether it is needed.

The cost is a byte count you can compute without a profiler.

At 4:2:0, a pixel averages 1.5 bytes. A 1080p YUV frame is 2,073,600 x 1.5 = 3,110,400 bytes, roughly 3.11 MB. The same frame as RGBA_8888 at 4 bytes per pixel is 2,073,600 x 4 = 8,294,400 bytes, roughly 8.29 MB. Running that conversion at 30 FPS means writing 8.29 x 30 = 248.7 MB per second, versus 93.3 MB/s if you had left the frame in its native layout.

That traffic is not free on a mid-range memory bus, and it happens before your actual effect has done anything.

What I look at: whether the conversion is needed at all. A shader can sample the Y and UV planes directly and do the color transform on the GPU as part of a pass you are already running. CameraX will also hand ImageAnalysis an RGBA output natively if you genuinely need RGBA, which at least moves the conversion out of your own loop.

2. Why does my camera preview freeze after a few seconds on Android?

The cause is usually more boring than a slow filter.

If you are using CameraX ImageAnalysis, the analyzer contract puts the responsibility on you to close each image. Google's documentation is explicit that if images are not closed, they may block further images from being produced, which stalls the preview, or get dropped according to whichever backpressure strategy is configured. Earlier versions closed the image for you; current ones do not, so that multi-frame analysis stays possible. Close the ImageProxy, not the Media.Image it wraps, because closing the wrapped image directly breaks the image sharing mechanism inside CameraX.

So a freeze that arrives a second or two in, instead of immediately, usually means a leaked frame reference. Filter speed is a red herring in that case.

The second half of this is the strategy itself, and it is worth knowing what you already have. Non-blocking is the default: STRATEGY_KEEP_ONLY_LATEST caches only the newest frame in a buffer one deep and overwrites it while you are still working on the previous one. That is almost always what a live filter wants, because a queued frame is a stale frame by the time it renders. So if you never set a strategy, you are already on the non-blocking one, which means a freeze is far more likely to be a frame you did not close than a strategy you did not pick.

The blocking variety has to be chosen deliberately. STRATEGY_BLOCK_PRODUCER queues images and starts dropping only once the queue is full, and Android's documentation notes that the blocking occurs across the entire camera device scope: if several use cases are bound to that camera, all of them stall while CameraX works through the queue. That is the mechanism that turns a slow analyzer into a frozen preview instead of a dropped frame or two.

// androidx.camera:camera-core:1.3.4
val analysis = ImageAnalysis.Builder()
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .build()

analysis.setAnalyzer(executor) { image: ImageProxy ->
    try {
        frameClock.mark()      // see the timing helper below
        renderEffect(image)    // your pipeline
    } finally {
        image.close()          // ImageProxy.close(), required including on the failure path
    }
}

// When the screen goes away, including when the user backgrounds the app
// mid-session rather than pressing a stop button:
analysis.clearAnalyzer()
cameraProvider.unbind(analysis)

The finally block matters more than it looks. One exception on one frame, with close() sitting on the happy path only, produces the symptom above and nothing in the log that points at it.

3. The model may not be running where you think

A GPU delegate that reports a believable inference time is not proof that inference ran on the GPU. TensorFlow Lite's GPU delegate documentation covers which operations the backend supports, and unsupported ops may fall back to CPU depending on version and configuration. Reported behavior is not uniform: developers have filed cases where the expected fallback did not happen and produced a runtime error instead, and cases where it silently did.

Either way, a plausible millisecond figure comes back. It may be describing an execution path other than the one you intend to ship.

That picture comes from the documentation and the issue tracker, so verify it against the version you actually build with before you rely on it.

What I look at: the execution path in a trace, instead of trusting the timing. On Android that means Perfetto, which has shipped in Android system images since Android 9 and has been enabled by default on most devices since Android 11. On iOS the Metal debugger and Metal System Trace in Xcode show the parallel CPU and GPU timeline, which is where a stage that claimed the GPU but ran elsewhere becomes obvious.

I have not tried to name the exact log line for a fallback here. The strings vary across versions and I could not verify a current one while writing this, so I have left it out. The symptom is the reliable signal: GPU utilization that stays flat while inference time stays constant.

4. Why does my frame rate drop when I read pixels back from the GPU?

If any stage pulls pixels back from the GPU with a blocking read, the pipeline empties.

The mechanism is well understood and worth stating precisely, because it explains why the cost never shows up where you look for it. A blocking readback cannot return until every queued draw command ahead of it has completed, so the CPU waits for the GPU to drain. The call itself is cheap. The stall is what costs you, and it lands on whichever line touches the data.

Samsung's own OpenGL ES guidance, which is a handset maker writing about its own hardware, is one of the clearer public writeups of the pattern. The standard remedy is a pixel buffer object: bind the buffer, issue the read, and it returns immediately while the copy happens asynchronously, then use a fence to check for completion a frame or two later.

The question I ask first: is the readback needed on every frame at all? Histogram, autoexposure and face-region logic often tolerate a fraction of the preview rate, and dropping one of those to every fifth frame is usually a larger win than optimizing the read itself.

5. The run was too short to fail

A pipeline that holds its budget for 30 seconds tells you very little, because a phone under sustained camera load heats up and the governor responds by lowering clocks. The camera sensor, the GPU and the neural accelerator all work at once, which is close to a worst case for sustained power draw.

What I look at: I run one scene for several minutes, log per-frame intervals throughout, and compare the last minute with the first. If the tail is materially worse than the head, the problem is thermal and no amount of per-stage optimization will move it. Screen brightness, whether the device is charging, and case material all change the result, which is why this one has to run on real hardware in a realistic state, never on a bench.

It is also the step teams skip most often, since it is slow and it needs a person holding a phone.

Here is the timing helper referenced in the snippet above. It keeps the full interval history, since averaging smooths away the exact spikes you are hunting.

class FrameClock(private val capacity: Int = 600) {
    private val intervalsMs = ArrayDeque<Long>(capacity)
    private var lastNs = 0L

    fun mark() {
        val nowNs = System.nanoTime()
        if (lastNs != 0L) {
            if (intervalsMs.size == capacity) intervalsMs.removeFirst()
            intervalsMs.addLast((nowNs - lastNs) / 1_000_000)
        }
        lastNs = nowNs
    }

    /** p is 0.0 to 1.0. Returns null until at least one interval is recorded. */
    fun percentileMs(p: Double): Long? {
        if (intervalsMs.isEmpty()) return null
        val sorted = intervalsMs.sorted()
        val index = Math.round(p * (sorted.size - 1)).toInt()
        return sorted[index]
    }
}

Read percentileMs(0.95). A pipeline averaging 28 ms with a 95th percentile of 70 ms is visibly janky, and the average on its own will never say so.

The budget sheet I actually fill in

I keep this per device, per resolution. It is the artifact I hand to whoever asks why the filter is slow.

The budget sheet I actually fill in

Stage	The budget I start from	How I confirm it	Cheapest fix
Frame delivery and format	2 to 4 ms	Byte count on paper, then a trace	Sample YUV planes in the shader, skip the conversion
Backpressure and queueing	0 ms if correct	Frame interval history at p95	STRATEGY_KEEP_ONLY_LATEST plus close() in finally
Inference	Largest single share	Trace the execution path itself	Confirm the backend, then lower input resolution
Render and composite	4 to 8 ms	GPU timeline in Perfetto or Metal System Trace	Merge passes, avoid full-frame intermediates
Readback	0 ms if asynchronous	Look for a CPU wait beside a drained GPU queue	Pixel buffer object plus a fence, or run it less often
Thermal headroom	Whatever is left	Multi-minute run, last minute versus first	Lower resolution or frame rate under sustained load

Those figures are my own starting budgets. I have yet to see a device leave all six of them intact, which is the point of filling in the measured column beside them. The gap between the two columns is the article you should be writing about your own app.

Where this usually lands

In most stuttering camera pipelines I have looked at, the effect itself was cheap. The time went to a frame copied into a format nothing needed, or queued when it should have been dropped, or read back synchronously, or executed somewhere nobody checked, and none of it surfaced in a 30 second run on a cool phone.

Do the two divisions and the byte count before you open a profiler. They cost a minute and they routinely tell you the answer, or at least tell you which of the five places to look in first.

6 Background Removal API Tests I Run Before Shipping Video

Dave Gordon — Fri, 21 Aug 2026 13:33:50 +0000

TL;DR:

Real-time background removal is a frame budget problem before it is a quality problem.

Six tests, in this order:

- does it fit the budget on your worst supported phone,
- did the run actually use the accelerator you think it did,
- what happens at the hair line,
- does the mask flicker between frames,
- what breaks when the lighting is real,
- and what does it cost you once you count platforms instead of license fees.

Most shortlists lose half their candidates on tests 1 and 5.

I have swapped segmentation backends three times across two camera apps, and every swap started the same way: the library was fine in the sample project and came apart on a real device with a real user in front of it. So I stopped evaluating these things by looking at output stills.

This is the harness, not the results table. I am not publishing my numbers here, because my numbers are about my pipeline and my support matrix, and neither of those is yours. A frame budget is derived from your capture rate and your effects stack, and a mid range phone in one support range is a flagship in another.

What transfers is the setup and the order the tests run in. Copy the rig, run your own candidates through it, and the numbers you get will actually apply to the thing you are shipping.

The rig

Before the first test, fix these so the results mean something across candidates.

Three device tiers, named, not averaged

A current or previous cycle flagship, a mid range phone around three years old, and the cheapest device with real share in your own analytics. Record the model, OS version and chipset for each. Score every test per device. An average across the matrix hides the phone your median user is holding, which is the only phone the decision actually turns on.

One app state, held constant

Camera open, your effects stack running, screen at full brightness, no other app in the foreground. Candidates measured in different app states are not comparable, and the tempting shortcut of benchmarking inference in isolation produces the number that looks best and matters least.

A warm device, and a stated warm-up

Decide how long the app records before you take the sustained reading, write that duration down, and use the same one every time. Thermal state is the variable that quietly decides half of these comparisons.

One fixed clip set

The same recordings, reused for every candidate: a plain wall clip, a still subject clip, and the lighting cases from Test 5. New footage per candidate means you are scoring the footage.

Pinned versions

SDK or model version for every candidate, written next to every result. Vendors ship model updates that move these numbers, and a result without a version cannot be reproduced or re-checked later.

Why does the frame budget decide this before quality does?

Most background removal comparisons open with mask quality. That is the wrong end. If inference does not fit inside a frame, quality is irrelevant, because you are already dropping frames and the preview stutters.

Work out your own budget before you look at a single demo:

Take your target capture rate. 30 fps for most social and editor video, 60 fps if the feature has to feel like the native camera.
Divide 1,000 ms by that rate. 30 fps gives you 33.3 ms per frame. 60 fps gives you 16.7 ms.
Subtract the rest of the pipeline. Camera capture, color space conversion, your own filters or effects, the render pass, and encoding all take a slice.
What is left is your segmentation budget. Run that subtraction rather than assuming you have the whole frame, because in the pipelines I have worked in, the remainder has never been more than half.
Compare that number against measured inference time on your slowest supported device, not your average one and not the phone on your desk.

Write that number down. Every test below is scored against it.

Test 1: Does it hold the budget on your worst target device?

Run the candidate on the cheapest and oldest tier, in the app state fixed above.

Three things to capture.

Cold inference time

The first frames after model load are always slower. If your feature is triggered mid session, users feel this directly.

Sustained inference time after thermal throttling

This is the number that matters and the one almost nobody publishes. Measure it rather than extrapolating from the cold figure, and expect the two to be far enough apart that the cold figure is useless for capacity planning. If a vendor publishes one inference time without saying whether the device was warm, assume it is the cold one.

Memory ceiling

Segmentation model plus video buffers plus encoder is where low RAM Android devices start getting killed in the background. Watch dumpsys meminfo for your process across a long recording rather than reading a peak off a profiler graph once.

Test 2: Did the run you just measured actually use the accelerator you think it did?

This test exists because Test 1 produces a plausible looking number whether or not it measured the pipeline you intended.

The GPU delegate fails to initialise, the library falls back to CPU without raising anything to your code, the run completes, and you write down a figure that describes a completely different execution path. Nothing in the output tells you. The mask still looks correct, the app does not crash, and the number is wrong in a direction that will not reproduce on the next build.

ERROR: Failed to apply GPU delegate.

If that line is in your logs, every timing you collected in that session is CPU timing. Two habits fix this permanently:

Assert the execution path per run. Log which delegate or backend was actually created, and discard any run where the assertion is missing rather than any run that looks slow. Slow runs are data. Runs of unknown provenance are not.
Fail closed. A benchmark harness that silently completes when initialisation failed is worse than one that crashes, because it produces numbers you will act on.

The same applies to any library that quietly reduces input resolution or frame stride under load. If the candidate has an adaptive mode, either turn it off for the measurement or record what it chose, because otherwise you are comparing two vendors that each decided independently what to sacrifice.

Test 3: What happens at the hair line?

Every segmentation model produces a clean silhouette. The difference between them lives in about forty pixels around the edge.

Use the plain wall clip and score four cases:

Loose or curly hair. Naive segmentation cuts a helmet shape and drops the strands entirely.
Glasses frames, especially thin metal ones, which models frequently classify as background and slice through.
An object held in hand, like a mug or a phone. Person only models will amputate it. Whether that counts as a bug depends on your use case, so decide deliberately rather than finding out in review.
The gap between arm and torso when someone puts a hand on their hip. Soft mask models tend to fill it in.

Score these against a hard edged replacement background, not a blur. Blur hides everything, which is exactly why so many demos use it.

Test 4: Does the mask flicker between frames?

This is the failure mode that gets caught in QA rather than in evaluation, and it is the most expensive one to fix late.

A per frame image segmentation model has no memory of the previous frame. Feed it video and the mask boundary jitters frame to frame even when the subject is completely still. The stills look perfect. The clip looks cheap.

Two ways a library can address it.

Temporal architecture

Robust Video Matting is the reference example, in that it carries state between frames with a recurrent design instead of treating each one independently. Check the license before you get attached to it: the code was re released under GPL-3.0 in September 2021, which for most proprietary mobile apps means you are studying the architecture rather than shipping the repo.

Your own smoothing

You can apply temporal filtering to a per frame mask yourself, but you are trading latency for stability, and that trade comes straight out of the budget you calculated above.

The test is simple. Take the still subject clip, thirty seconds of someone barely moving, then step through the export frame by frame and watch the boundary. If it breathes, you either fix it or you ship it.

Test 5: What breaks when the room does?

Evaluation footage is always shot in good light. Your users are on a couch at 9pm with a window behind them.

Run each candidate through:

Backlight, with a bright window behind the subject. Classic failure: the person is segmented as background.
Low light, where sensor noise destabilises the edge and Test 4 gets much worse.
A busy background with furniture, a doorway, or a poster of a person on the wall.
Two people in frame, where some models pick one, some merge both, and some alternate between the two across frames.
Movement toward and away from the camera, which changes subject scale and is where fixed input resolutions show their limits.

If you automate one part of this harness, automate this one. It is the section that regresses silently when a vendor ships a model update.

Test 6: What does it actually cost you per platform?

The license fee is the visible number and usually not the deciding one. Count the real cost like this:

List every platform you ship on. iOS, Android, web, desktop, and any framework layer such as React Native or Flutter.
Check what the candidate covers natively. Apple's Vision framework offers person segmentation with selectable quality levels, and it is iOS and macOS only, so an Android build needs a second solution and a second set of edge case behaviour to test.
Multiply the platforms you have to solve separately by the integration and QA time for one platform. That multiplier, not the sticker price, is where the budget goes.
Add the ongoing cost. Two segmentation backends means two model update cycles, two sets of device regressions, and visible quality differences between your iOS and Android users.
Check the deployment model, which is where the arithmetic gets decisive.

Can you use a cloud background removal API for live video?

Not realistically. Cloud background removal is priced per image and needs an upload per frame, so a 30 fps preview is 30 requests per second per user with a network round trip on every one. That is the right tool for batch photo processing and the wrong one for a live camera preview, and it fails on arithmetic before you get to quality.

For the free, on device end of the shortlist, MediaPipe Image Segmenter is Apache 2.0, and ML Kit selfie segmentation is a closed source SDK that is free to use. Both run on device across iOS and Android, which makes them the honest baseline. Anything commercial has to beat them on the tests above by enough to justify the line item, and often it does, because Tests 1, 4 and 5 are where a general purpose model gives ground to one tuned for video.

The scorecard

Test	What it measures	Pass condition
1. Frame budget on worst device	Cold and sustained inference time, memory ceiling	Sustained time fits the segmentation budget on the slowest supported device, not the average one
2. Execution path	Whether the accelerator you configured is the one that ran	Delegate asserted in the log on every run, runs without the assertion discarded
3. Hair line	Edge quality on hair, thin frames, held objects, arm to torso gaps	Scored against a hard edged background, not a blur
4. Temporal stability	Mask jitter across frames on a still subject	Boundary holds frame to frame, or the smoothing cost is inside the budget
5. Real conditions	Backlight, low light, busy background, two people, scale change	No condition drops below your quality floor, and the suite is automated
6. Cost per platform	Platforms solved separately, integration and QA multiplier, deployment model	One integration covers your matrix, or the second one is budgeted

Pick your slowest supported device and your worst lighting scenario first, then evaluate. Run these six in order and you will usually eliminate half the shortlist before pricing comes up, and you will eliminate it for reasons you can put in front of a product manager.

If you run this harness on something I have not tried, I would like to hear what broke.