DEV Community: Dave Gordon

8 React Native Video Editor Options and the Code You Inherit

Dave Gordon — Fri, 23 Oct 2026 12:54:40 +0000

TL;DR

In React Native the editing runs in native code, so what you pick is the bridge layer.
Four of the eight publish a current React Native package. Only two of those declare a codegen spec, and one of the three without a spec shipped this month.
Two never run on the device: both render on a server, and the one browser preview among them is desktop-only.
FFmpegKit is retired. Repository archived 2 July 2026, last React Native release September 2023.
Package-and-docs screen, no device runs.

If you are picking a React Native video editor, start with the layer you will actually own: the JavaScript package and its native bridge.

Read who publishes it, under which license, whether it declares a codegen spec, when it last shipped, and where the editing work runs. Those five answers narrow the field faster than any feature list.

This is the screening pass. I read packages and documentation to decide which options are worth putting on a device at all. The device measurements, with real export runs, come in a later post.

Where these facts come from

I went through the npm metadata for each package: the license field, the codegenConfig block, the published date, and the peer and dev dependency pins. Then each project's own documentation. Everything below was read in October 2026 unless a sentence says otherwise.

You can redo the package half of this in one line, which is the whole reason the screen is built on metadata rather than on feature lists:

npm view react-native-video-trim version license time.modified codegenConfig

That prints checks 1, 2, and 3 for any package in this post, and for any package that gets published after it. The numbers below will age. The command will not.

Nothing here was installed or run. I did no exports, and I timed nothing, so there is no device, no OS version, no chipset, and no per-device result anywhere in this post.

Export timings for these SDKs do exist, published as benchmarks by the companies selling them. I produced none of them.

For anything that is not on npm, the vendor documentation is the source, and a documentation page is a snapshot of whenever someone last looked. Before any of it decides a build, get the claim from the vendor in a document with a date and a name on it.

Check 1: Who publishes the React Native package, and what license does it carry?

Look at the npm author field, the maintainer list, and the license field.

An author that is a company means a support contract is possible and a bill is coming. A single named maintainer means your escalation path is one person's inbox.

IMG.LY publishes @imgly/editor-react-native under the author "IMG.LY GmbH", maintained by company accounts rather than one individual. Its license field reads "SEE LICENSE IN LICENSE.md", which is a commercial license. That is a complete first-party React Native story: the vendor writes the bridge and signs for it.

Licensing is per target platform for React Native or Flutter, monthly or yearly, with one license covering several app identifiers such as staging, beta, and production, as IMG.LY's video SDK page and its pricing page read. Access is commercial and sales-led, full white-labeling is offered, and enterprise terms include an air-gapped option.

Those two pages are the one exception to the October reading above. I last went through them in June, so of everything here, they are the most likely to have moved.

Banuba publishes video-editor-react-native, authored by Banuba, against the repository at GitHub, with one company maintainer account.

Its npm license field reads MIT, and that one needs reading slowly. The SDK is commercial and will not initialise without a token, so the MIT covers the bridge code in the package and not the engine underneath it. IMG.LY's field says "SEE LICENSE IN LICENSE.md" and tells you that in the field itself. If you run check 1 the way I just described it, on this listing you get the wrong answer.

react-native-video-trim is MIT, authored by Duc Trung Mai, with one maintainer on the package.

MIT is the cleanest license outcome in this set: no per-platform fee, no approval gate, and the whole module is small enough to read end to end, since its repository scopes itself as "Video trimmer for your React Native app".

react-native-video-processing is also MIT, authored by Shahen Hovhannisyan, one maintainer, described as "Native Video processing/editing for React-Native" on its repository. MIT plus a dormant codebase means you own the fork outright, with no license conversation to have.

FFmpegKit's React Native library is LGPL v3.0 by default, and GPL v3.0 when the build enables GPL-licensed libraries, through the --enable-gpl flag or a prebuilt package with a -gpl suffix, as its repository documents.

Its own patents section states the belief that FFmpeg, kvazaar, x264 and x265 include algorithms subject to software patents, and recommends legal advice. It also gives format coverage no vendor SDK in this group matches, with no license fee attached.

Shotstack's Studio SDK is published as @shotstack/shotstack-studio under PolyForm Shield 1.0.0, which Shotstack describes as usable "for any application, commercial or non-commercial, as long as it doesn't directly compete with Shotstack".

That is a source-available license. It is not an open-source license, and the distinction decides things if your own product is a video tool.

Creatomate publishes @creatomate/preview at version 1.6.1, published 3 September 2026, under MIT, with devDependencies only and no runtime dependencies at all.

For a browser bundle that is the lowest dependency surface in this group, and MIT on the client library is worth noting next to a cloud service you pay for by output.

Meishe publishes no React Native package at all, so there is nothing to read for checks 1 to 3.

Its page title reads "Video Editor SDK for React Native & Flutter" and the body states that it "supports React Native, Flutter and other high-quality application development frameworks". No package, no installation page and no sample exists on the site or the docs.

Pricing is not publicly documented, which puts access somewhere between approval-gated and enterprise-only without saying which.

Check 2: Is there a codegen spec, or is it still on the old bridge?

Open the package and search for codegenConfig.

Its presence means the module was built as a turbo-module against the New Architecture. Its absence means an old-architecture native module, and a migration that lands on you.

IMG.LY ships a codegenConfig block with the spec name IMGLYEditorModuleSpec and the Android package ly.img.editor.reactnative.module. react-native-video-trim declares spec VideoTrimSpec with Android package com.videotrim and was scaffolded as a turbo-module. Those are the two New Architecture entries here.

Banuba's package has no codegenConfig. Its peer dependencies accept any React Native version, and its devDependencies pin react-native 0.83.0 with react 19.2.0, which is the newest pair anyone in this group builds against.

That combination is worth sitting with: current on the React Native version, old on the architecture. You get a package built against this autumn's React Native and an interop path you maintain yourself.

react-native-video-processing has no codegenConfig either. Treat it as a legacy bridge module and budget the port before you budget features.

For FFmpegKit, the React Native library provided a JavaScript API with TypeScript definitions, and nothing in the archived repository addresses codegen. Shotstack and Creatomate publish browser JavaScript, so this check does not apply to either, and Meishe publishes nothing to check.

The same pattern shows up on the other side of the cross-platform fence, which I went through in a post on Flutter video editor packages wrapping native engines.

Check 3: When was the React Native package last published, and against which React Native version?

Read the published date, then read the dev dependency pins. The pins tell you which React Native the maintainer last built against, which is a better signal than a README badge.

IMG.LY's package is at version 1.84.0, published 15 October 2026. react-native-video-trim is at 8.2.4, published 5 October 2026, with devDependencies pinning react-native 0.79.2 and react 19.0.0. Both are current enough that your own upgrade cadence sets the pace.

Banuba's package is at 0.55.2, published 9 October 2026. The four releases before it land on 6 October, 12 September, 8 September, and 18 August, which is the most regular cadence of anything in this group.

The version number deserves a glance of its own. A 0.x on a product that has been selling for years usually means the package is much newer than the engine behind it, and that the bridge is where the churn is.

react-native-video-processing is at 1.7.2, published 8 September 2022, pinning react-native ^0.49.5 and react 16.0.0-beta.5. Four years of React Native releases have landed since.

Beyond that package metadata the project publishes very little, so the version gap is most of what there is to read on it, and I would rather say that than score it down quietly.

FFmpegKit's last React Native release in the repository's own version table is 6.0.2, built on FFmpeg 6.0, released 19 September 2023. Shotstack, Creatomate and Meishe publish no React Native package, so there is no date to read for any of the three.

Check 4: Does the editing happen on the device or on someone's server?

This check decides your architecture, your offline behavior and where your users' footage goes. Read whether the project documents on-device processing, a hosted render path, or both.

IMG.LY runs client-side, and IMG.LY also publishes a Node.js package that runs the same engine headlessly on a server for batch rendering. That headless server path is the only documented one of its kind among these eight, which matters when you need the same output from a phone and from a queue.

Banuba runs on device, and its package description states that it works offline. No server render path is published alongside it, so if you later need the same project rendered in a queue, that is a second engine rather than a flag.

Shotstack is the opposite shape. There is no React Native SDK, no Flutter SDK, no Unity SDK, no native iOS or Android SDK and no on-device processing path.

What exists is the Edit API: hosted, JSON in and rendered video out, priced per minute of output, with server-side client libraries for Node.js, Python, PHP and Ruby, plus the browser Studio SDK whose export path uses FFmpeg.

Taking the whole render off the device is the point of it, and the same Edit JSON the browser editor loads is what the Edit API renders. Sign-up is open and the per-minute price is published.

Creatomate splits the same way, and its own Preview SDK installation page is direct about why. The preview "renders video in real time on the user's device, which takes more performance than most phones and tablets have. It therefore runs in desktop browsers only", best on Chrome, Edge and Opera, with Safari and Firefox supported and playback possibly less smooth.

The page tells you to detect the device before loading the SDK and, for phones and tablets, to "show them a video rendered through the API instead of the live preview", at low resolution through a render_scale parameter. On the final file: "Rendering an MP4 in the browser is too unreliable across devices." No React Native path is published.

The strength here is consistency: the same template and the same changes drive both the preview and the server render, so what the user sees is what the API produces. Sign-up is open.

FFmpegKit, react-native-video-trim and react-native-video-processing are all on-device by construction, and all three install from npm with no approval step.

Meishe publishes the widest platform list of the eight: iOS, Android, Web, PC and HarmonyOS Next, with offline SDK operation and a privatized deployment option, plus 8K image processing, HDR export and multi-track editing.

The documented integration path is an Objective-C API and a Java API against a class called NvsStreamingContext, and a web build loaded from a CDN script tag rather than an npm package. The SDK is split into EffectSDK and StreamingSDK.

Check 5: What happens if the package stops being maintained?

FFmpegKit is the worked case, because it already happened.

The arthenica/ffmpeg-kit repository page states: "This repository was archived by the owner on Jul 2, 2026. It is now read-only." The README's "Update (July 2026)" section states that "FFmpegKit has been officially retired" and that development continues as FFmpegKitNext, which "is distributed as source only".

Read what that sentence costs you. Source-only distribution means you build the native binaries for both platforms yourself, in CI, for every FFmpeg configuration flag your feature set needs, and you keep doing it.

The archived README now points readers to community-maintained forks through the package managers, so your other option is trusting a fork whose maintainer you have not met. Both paths start at the successor repository.

Ask the same question of every other option before you commit, because the answer differs by class.

A commercial package with company maintainers, which covers IMG.LY, Banuba and Meishe, puts you in a contract conversation. That is the easiest of these to plan for.

An MIT package with a single maintainer leaves you holding a fork. Survivable while the scope is a trimmer, expensive once the scope is a full editor.

The cloud APIs expose you to an outage you cannot patch, and eventually to migrating your JSON format somewhere else.

With a source-available license your legal reviewer inherits the question of what "directly compete" covers for your product, and that review is worth starting before the integration rather than after it.

The eight side by side

Option	What it is	RN package and license	Codegen	Last RN publish	Where processing runs	Access
IMG.LY CreativeEditor SDK	Vendor SDK with an RN package	@imgly/editor-react-native, "SEE LICENSE IN LICENSE.md" (commercial)	Yes, IMGLYEditorModuleSpec	1.84.0, 15 Oct 2026	On device, plus a headless Node.js server path	Commercial, per target platform, white-labeling and air-gapped option
Banuba Video Editor SDK	Vendor SDK with an RN package	video-editor-react-native, npm field reads MIT, engine is commercial	No	0.55.2, 9 Oct 2026	On device, offline, no server path published	Commercial, token required
Meishe SDK	Vendor SDK, native only	None published	None to read	None	On device, offline operation, privatized deployment	Pricing not publicly documented
FFmpegKit	Community RN module, retired	ffmpeg-kit-react-native, LGPL v3.0, GPL v3.0 with GPL builds	Not addressed	6.0.2, 19 Sep 2023	On device	Open, archived 2 Jul 2026, successor is source only
react-native-video-trim	Community RN module	MIT	Yes, VideoTrimSpec	8.2.4, 5 Oct 2026	On device	Open, npm install
react-native-video-processing	Community RN module	MIT	No	1.7.2, 8 Sep 2022	On device	Open, npm install
Shotstack	Cloud render API plus browser editor	No RN package; @shotstack/shotstack-studio is PolyForm Shield 1.0.0 (source-available)	Not applicable	None	Server, priced per minute of output	Open sign-up, price published
Creatomate	Cloud render API plus browser preview	No RN package; @creatomate/preview (MIT) for desktop browsers	Not applicable	None	Server, preview in desktop browser only	Open sign-up

Every row in that table is stated in the text above, so nothing lives only in the table.

These are the values each project published about itself when I read them. I measured none of them, and the npm command above re-reads four of these columns for you.

Which React Native video editor would I start with?

Feature depth is a separate question from maintenance, and I went through the feature side of these tools in an earlier roundup of mobile video editor SDKs. What follows is the maintenance answer.

A short-form camera app where users trim clips before upload: start with react-native-video-trim. MIT plus a current New Architecture turbo-module plus a scope you can read in one sitting means the fork risk is bounded even with one maintainer.

A multi-track editor inside a consumer app, with a vendor who signs for the bridge: shortlist IMG.LY and Banuba, and let the architecture split them.

IMG.LY declares a codegen spec and publishes a headless Node.js path, so the New Architecture is handled and the same project can render server-side later. Banuba ships more often, builds against a newer React Native, and runs offline, but has no codegen spec and no server render, so you carry the interop path and you keep rendering on the phone.

A template-driven product where users fill in text and media and a finished video comes out: start with Shotstack or Creatomate. The render never touches the phone, both bill against output, and your React Native app only posts JSON and polls for a URL.

A fixed FFmpeg pipeline you already understand, with CI engineers to hand: start from the FFmpegKitNext source. You accept a build burden in exchange for format coverage and no license fee, having read the patents note first.

HarmonyOS Next or desktop alongside iOS and Android: start a conversation with Meishe, and go in expecting to write the React Native bridge yourself against NvsStreamingContext.

Two packages pass all five checks, and both shipped in October

The screen sorts these options by what each project publishes about its React Native layer.

IMG.LY and react-native-video-trim give you a current turbo-module to read. Banuba gives you the most frequent releases and the newest React Native pin, with no codegen spec behind them. react-native-video-processing gives you MIT source and a four-year-old bridge.

FFmpegKit gives you an archive notice and a source-only successor. Meishe names React Native in its page title and publishes nothing for it. Shotstack and Creatomate move the work to a server, which changes your architecture more than your feature list.

Before the device post, the next thing I open is node_modules for the two packages that declare a codegen spec.

Questions I get asked about this

Does a missing codegenConfig mean the package will not work on the New Architecture?

It can still work through the interop layer, but you are relying on a compatibility path rather than a generated spec, and nobody upstream is testing your case.

For react-native-video-processing, with react-native pinned at ^0.49.5, assume you are the one maintaining that path. For Banuba, pinned at 0.83.0, the package is current enough that the interop path is at least being exercised against a recent React Native.

Can I use FFmpegKit's last release and deal with it later?

You can install 6.0.2 from September 2023 today, and it will keep building until something in your toolchain moves.

The cost arrives on the next iOS or Android SDK bump, when no upstream release exists and you are either patching the archive yourself or moving to FFmpegKitNext and its source-only builds.

Is PolyForm Shield 1.0.0 open source?

No, it is source-available. You can read and use the code, including commercially, under Shotstack's stated condition that your application does not directly compete with Shotstack.

That condition is the part to send to your legal reviewer before you build on it.

Why does an npm package say MIT when the SDK costs money?

Because the field describes the package, not the engine it loads. A vendor bridge can be MIT while the native binaries behind it need a commercial token, and the npm field has no way to say so.

IMG.LY handles this by putting "SEE LICENSE IN LICENSE.md" in the field. When a vendor package shows a bare MIT, check whether the thing you are licensing is the wrapper or the product.

Why not just use a cloud API and skip native code entirely?

That is a real option if your editing is template-driven and your users tolerate a round trip, which is how Shotstack's Edit API and Creatomate's REST API are shaped.

It stops being an option when you need frame-accurate scrubbing with no network, since Creatomate's installation page puts the Preview SDK in desktop browsers only and Shotstack documents no on-device path.

Flutter AR Plugins in 2026: 7 Packages Screened Before a Device Test

Dave Gordon — Thu, 22 Oct 2026 08:50:42 +0000

TL;DR

Seven Flutter AR packages from pub.dev, read in October 2026. No builds and no device runs.
Three track faces only. Four wrap ARKit or ARCore and track the room, and three of those four also expose a face mode.
None of the seven gives you face tracking in a Flutter web build.
deepar_flutter is DeepAR's own package, last uploaded in January 2023, and its permission_handler pin clashes with apps already on version 11 or newer.
The ML Kit face mesh package carries an iOS tag on pub.dev, while its README says Android only.

Flutter AR always runs in native code. ARKit, ARCore, Google's ML Kit or a vendor SDK does the tracking, and the Dart package you add to pubspec.yaml is the bridge to it, so choosing a Flutter AR package mostly means choosing which native engine you inherit and how much setup it leaves on your desk.

I screen for that before any package earns time on a phone. This post is the screen for seven packages, run as five numbered checks. The native SDKs underneath are a separate question, and I went through six of those from their own documentation in 6 face AR SDKs compared by what their docs publish.

How I ran this screen

Sources for this round, read in October 2026: each package's pub.dev listing (version, upload date, publisher, license, platform tags, dependencies), its README, and the API reference where the README was thin. For two native face SDKs that skip pub.dev, Agora's integration guide and each vendor's own site.

Nothing was built, installed or run for this post, so it carries no device model, OS version, chipset or frame time. It is a filter that decides what deserves device time, and the device round is a separate post.

One rule for this round was self-serve only. A package qualified if a developer can install it and initialize it without talking to a salesperson. DeepAR's license keys come from a project you create on its developer site, so it qualified. Three vendor SDKs did not, and they are at the end.

If one of these facts ends up deciding your build, confirm it with the publisher in writing, because a pub.dev listing changes with every upload.

The seven packages

Four wrap a platform AR framework and are built around the room: ar_flutter_plugin_plus, arkit_plugin, arcore_flutter_plugin and augen.

Three track faces only: google_mlkit_face_detection, google_mlkit_face_mesh_detection and deepar_flutter.

Check 1: does the package track a face, or the room around you?

This is the split people skip, and it decides how much rendering code you write.

ar_flutter_plugin_plus wraps ARKit on iOS and ARCore on Android. Its eight example apps cover debug options, local and online objects, anchors on planes, transformation gestures, screenshots, cloud anchors, external model management and image tracking. None of the eight is a face sample, so treat it as a world-tracking package.

arkit_plugin wraps ARKit and lists a Face Tracking example, described as a face mask sample, next to plane detection, image detection and body tracking.

arcore_flutter_plugin wraps ARCore on Android and ships an ArCoreFaceView widget. Its enableAugmentedFaces flag defaults to false, so face mode is something you switch on.

augen wraps ARCore on Android and RealityKit with ARKit on iOS. Its feature list puts face tracking "with facial landmarks and expressions" beside plane detection, image tracking, cloud anchors and physics.

The other three only do faces, and they split again. google_mlkit_face_detection returns a bounding box, head rotation angles, landmarks for the mouth, ears, eyes, cheeks and nose, a smile probability when classification is on, and a tracking ID when tracking is on. google_mlkit_face_mesh_detection returns a 468-point 3D mesh. Both hand geometry back to Dart, and drawing an effect on top of it is your job.

deepar_flutter sits at the other end. Its feature list is live AR preview, screenshots, video recording, camera flip and flash, so the vendor's renderer draws the effect and you place a preview widget.

Here is the ML Kit side in practice, adapted from the face mesh package's README:

// FaceMeshDetector and FaceMesh come from google_mlkit_face_mesh_detection.
// inputImage: one camera frame wrapped as an InputImage (google_mlkit_commons).
Future<void> readFaceMesh(InputImage inputImage) async {
  final meshDetector =
      FaceMeshDetector(option: FaceMeshDetectorOptions.faceMesh);
  final List<FaceMesh> meshes = await meshDetector.processImage(inputImage);
  for (final FaceMesh mesh in meshes) {
    final points = mesh.points;       // the 468 3D points
    final triangles = mesh.triangles; // topology for your own renderer
  }
  meshDetector.close();
}

Every point and triangle in that loop is input for a painter or shader you still have to write.

One number to keep straight if you are comparing docs: ML Kit's face mesh returns 468 points, while Google's MediaPipe Face Landmarker returns 478. They are different products from the same company, and the counts are not a typo in either place.

Check 2: who publishes it, under which license, and when did it last ship?

pub.dev shows a verified publisher as a domain next to the package name. Six of the seven have one:

ar_flutter_plugin_plus: xinix.tech, version 1.1.3, uploaded 28 January 2026. It is a fork of Lars Carius's original ar_flutter_plugin, whose newest release, 0.7.3, dates from 19 November 2022.
arkit_plugin: leushchenko.com, version 1.5.0, uploaded 3 August 2026.
arcore_flutter_plugin: gdifrancesco.dev, version 0.1.0, uploaded 7 February 2023, plus a 0.2.0-alpha prerelease from 20 February 2023.
google_mlkit_face_detection 0.15.1 and google_mlkit_face_mesh_detection 0.6.1: flutter-ml.dev, both uploaded 17 August 2026. Both READMEs state that the plugin is "not sponsored or maintained by Google".
deepar_flutter: deepar.ai, version 0.0.5, uploaded 29 January 2023.

augen is the one unverified uploader in the set, and also the newest upload, version 1.5.0 from 1 October 2026.

All seven list MIT on pub.dev. That license covers the bridge code in each package. The engine underneath is a separate dependency, and DeepAR's will not initialize without a license key for each platform.

Does DeepAR support Flutter?

Yes, through a package DeepAR publishes itself. pub.dev lists deepar_flutter under the verified publisher deepar.ai with the description "Official Flutter SDK for DeepAR", for Android SDK 23+ and iOS 13.0+. The catch is age: 0.0.5, uploaded 29 January 2023, is the newest of its five versions.

The confusion has a source. DeepAR's documentation, as it read on 17 October 2026, names four supported platforms (iOS, macOS, Android and Web) and has no Flutter page, so research that stops at the docs comes back with the wrong answer.

The age shows up in the dependency list. deepar_flutter 0.0.5 depends on permission_handler ^10.0.0, which in Dart's caret syntax means any version from 10.0.0 up to, but not including, 11.0.0. If your app or another package in it needs permission_handler 11 or newer, pub cannot resolve both without a dependency_overrides entry, and the same Dart page warns that overriding outside a package's declared range "might break your application". ar_flutter_plugin_plus 1.1.3, for example, asks for permission_handler ^12.0.1, so those two packages cannot sit in one pubspec without an override.

Access is self-serve. You create a project on DeepAR's developer site and generate the keys there. The native SDK is free for development and testing with a watermark. DeepAR's pricing page, last checked on 17 October 2026, listed free use up to 10 monthly active users, paid tiers from $25 a month for up to 1,000 MAU to $1,000 a month for up to 100,000 MAU, and sales above that. Treat those figures as approximate, not a quote.

Check 3: does the pub.dev platform tag match what actually works?

Read the tags and the README together, because they do not always agree.

google_mlkit_face_mesh_detection is tagged Android and iOS. Its README opens with a note that the feature "is still in Beta, and it is only available for Android".

augen is tagged Android, iOS and web. Its web table shows marker tracking from PNG or JPG templates working in the browser, while face tracking, plane detection and cloud anchors are marked "Mobile only". On iOS it adds one more limit: RealityKit cannot read GLTF or GLB, so those models render as an orange placeholder box and the README tells you to ship USDZ for iOS.

arkit_plugin is iOS only, and its FAQ answers the Android question with "No, as ARKit is not available on Android". arcore_flutter_plugin is Android only.

google_mlkit_face_detection covers Android and iOS, and its README states that web "or any other platform is not supported". deepar_flutter lists "Android & iOS", even though DeepAR's native SDK also runs on the web.

Put together, none of the seven gives a Flutter web build face tracking.

Can I use ML Kit face mesh detection on iOS in Flutter?

Not with google_mlkit_face_mesh_detection 0.6.1, whose README limits the feature to Android while it is in Beta. On iOS the closest option in this set is google_mlkit_face_detection, which gives you contours and landmarks but no 468-point mesh.

Check 4: what do you set up outside pubspec.yaml?

None of the following happens on flutter pub get, so it belongs in the estimate.

deepar_flutter on Android needs compileSdkVersion 33 or more and minSdkVersion 23 or more, plus the native library itself: you download it from DeepAR and place it at android/app/libs/deepar.aar. Its README adds ProGuard keep rules, "else app might crash in release build". On iOS it wants a 13.0 deployment target and Podfile preprocessor definitions for camera and microphone permissions.

google_mlkit_face_detection on iOS needs a 15.5 minimum deployment target, Xcode 15.3.0 or newer, and armv7 excluded in Xcode, because ML Kit supports 64-bit architectures only.

augen on Android declares android.hardware.camera.ar as required and sets the com.google.ar.core meta-data to required, with minSdkVersion 24. On iOS it needs NSCameraUsageDescription, arkit under UIRequiredDeviceCapabilities, and a 13.0 deployment target. It also asks for Dart 3.9.2 or newer, which matters if your app is pinned to an older toolchain.

How do I turn on face tracking in arkit_plugin?

You opt in, because the TrueDepth and face tracking code is not compiled in by default. On Flutter 3.44 and later, which builds plugins with Swift Package Manager, add this key to ios/Runner/Info.plist:

<key>ARKitFaceTrackingEnabled</key>
<true/>

Then clear the SwiftPM manifest cache and DerivedData:

rm -rf ~/Library/Caches/org.swift.swiftpm/manifests
rm -rf ~/Library/Developer/Xcode/DerivedData

The README is explicit that flutter clean and Xcode's Clean Build Folder are not enough on their own, because SwiftPM caches the evaluated Package.swift.

If your repository does not put Info.plist where the plugin expects it, which is the normal situation in a monorepo, the README gives an environment variable instead: set ARKIT_FACE_TRACKING_ENABLED=1, with export for terminal builds and launchctl setenv for builds started from the Xcode GUI.

On older Flutter versions that still build with CocoaPods, the same switch is a post_install block in ios/Podfile that adds -DENABLE_TRUEDEPTH_API to OTHER_SWIFT_FLAGS on the arkit_plugin target. If you have set up SPM, skip the Podfile step.

Why is ARCore tracking choppy in my debug build?

ar_flutter_plugin_plus warns about a bug in the ARCore library that interferes with tracking on some Android devices, and its workaround is to build with debuggable false in build.gradle. The README links the ARCore issue on GitHub. If tracking only looks rough in debug, rule that out before you blame the plugin. Its suggested Podfile also raises the iOS deployment target to 15.0.

"can only be launched from Flutter tooling" on an iOS debug build

augen's troubleshooting section covers this one: debug builds must be started by flutter run or Xcode, not from the home screen. If flutter run hangs waiting for the VM Service, it suggests closing Xcode and running flutter config --enable-lldb-debugging, or using flutter run --release.

Check 5: what does the phone have to have?

arkit_plugin notes that ARKit needs an A9 or later processor (iPhone 6s/7/SE/8/X, iPad 2017/Pro) on iOS 11 or newer, that some features need iOS 12, and that the plugin's own minimum is iOS 13.0. For faces, augen's iOS table maps face tracking to ARFaceTrackingConfiguration on the TrueDepth front camera.

On Android, augen's troubleshooting list starts with two device checks that apply to every ARCore wrapper here: the device has to support ARCore, and Google Play Services for AR has to be installed.

The ML Kit face mesh wants faces within about 2 meters of the camera, facing it, with at least half of the face visible. On iOS, ML Kit runs on 64-bit devices only.

deepar_flutter sets its floor at Android SDK 23 and iOS 13.0.

For arcore_flutter_plugin I read the versions list and the API reference only, so it has the thinnest entry in this post. That reflects my reading list and says nothing about the package's quality.

The seven side by side

Package	Tracks	Publisher on pub.dev	Version I read (upload date)	Platforms that work	Setup to plan for	Access
ar_flutter_plugin_plus	Planes, anchors, images, cloud anchors	xinix.tech (verified)	1.1.3 (28 Jan 2026)	Android, iOS	debuggable false for the ARCore bug, iOS 15.0 in Podfile	Open, MIT
arkit_plugin	World, face (opt-in), body	leushchenko.com (verified)	1.5.0 (3 Aug 2026)	iOS	TrueDepth flag plus SwiftPM cache clear	Open, MIT
arcore_flutter_plugin	World, Augmented Faces	gdifrancesco.dev (verified)	0.1.0 (7 Feb 2023)	Android	enableAugmentedFaces: true	Open, MIT
augen	World, images, face	Unverified uploader	1.5.0 (1 Oct 2026)	Android, iOS; web for markers only	ARCore required in manifest, USDZ on iOS, Dart 3.9.2+	Open, MIT
google_mlkit_face_detection	Face box, landmarks, contours	flutter-ml.dev (verified)	0.15.1 (17 Aug 2026)	Android, iOS	iOS 15.5, exclude armv7	Open, MIT
google_mlkit_face_mesh_detection	468-point face mesh	flutter-ml.dev (verified)	0.6.1 (17 Aug 2026)	Android only, despite the iOS tag	minSdkVersion 21	Open, MIT
deepar_flutter	Face, with vendor rendering	deepar.ai (verified)	0.0.5 (29 Jan 2023)	Android, iOS	Manual deepar.aar, ProGuard rules, permission_handler ^10	Self-serve keys, paid by MAU above 10

Every row in that table is stated in the text above, so nothing lives only in the table. The version column is what each pub.dev listing showed when I read it in October 2026, and every cell is a listing value I copied. Nothing in the table was measured.

What about the vendor SDKs that did not qualify?

Three face SDKs are missing from the table, all for the same reason: you cannot get started on them without talking to someone. They fail differently, so they are worth separating.

FaceUnity has no pub.dev package that I found. Its own effects page, as it read in July 2026, lists Flutter among its platforms, and the documented Flutter route I found runs through Agora: FaceUnity's AR Filter extension for the Agora RTC SDK. You add the native Android .aar files through Gradle and the iOS .framework files through CocoaPods, convert an authpack certificate to Dart, and enable it with enableExtension(provider: "FaceUnity", extension: "Effect", enable: true). The guide asks for Flutter 2.0.0 or higher and Dart 2.15.1 or higher. That route fits apps that already run Agora video calls, and it is a lot of plumbing for a camera filter outside a call. FaceUnity publishes no price list, and access starts with a trial application.

Visage Technologies documents no Flutter or React Native wrapper on its pages as they read in June 2026, with Unity as the only engine it names, so the Flutter side would be platform channels you write yourself. Its pricing page at that time listed a free FaceTrack LITE license for one app up to 10,000 users and technical support billed separately at 1,500 euros per year.

Banuba is the opposite case, and the one worth knowing about if the January 2023 date on deepar_flutter is what rules DeepAR out for you. It does publish a Flutter package, banuba_sdk, under the verified publisher banuba.com, and it has shipped through 2026, with 3.1.4 uploaded on 10 July 2026 and 3.1.7 current when I read the listing in October 2026. Two things the listing does not give you: a license, which shows as unknown rather than MIT, and the platform tags, which were blank on the version page I opened. The access gate is what keeps it out of the screen: the 14-day demo token comes through a website form or a sales manager, not from a developer console, so you cannot get from pub.dev to a running app on your own.

Which Flutter AR package fits which app?

If you are building a face filter app on both platforms and want a vendor renderer, start with deepar_flutter, because it is the only self-serve package here built around switching prepared face effects (switchEffect) on a vendor renderer. Plan to own its Dart layer, since nothing has shipped since January 2023, and check the permission_handler pin against your app first.

If you are building your own overlay or analytics on face geometry, use google_mlkit_face_detection, because it covers Android and iOS from a verified publisher and shipped in August 2026. Add google_mlkit_face_mesh_detection only for an Android-only mesh feature.

If your face feature is iOS only, use arkit_plugin, because it shipped in August 2026 and exposes ARKit face tracking once you turn on the TrueDepth flag.

If you need world AR on both platforms (plane anchors, placed models, cloud anchors), use ar_flutter_plugin_plus over the original ar_flutter_plugin, because the fork shipped in 2026 and the original has not had a release since November 2022.

If you want one Dart API over ARCore and RealityKit with faces included, try augen, the newest upload in the set, and weigh that against an unverified uploader and a Dart 3.9.2 minimum.

If you need face effects in a Flutter web build, none of these seven covers it, so plan a separate web path from day one.

Face tracking in Flutter still means a mobile build and native setup

Across the seven, the Dart API was the easy part. Every face option is mobile only, every one asks for native configuration that flutter pub get does not do, and the upload dates run from October 2026 back to January 2023.

Two findings surprised me: the face mesh package's iOS tag, and the permission_handler pin in DeepAR's own package. Both are visible on the listing pages before anyone writes a line of code, which is the argument for running this screen at all. The device round takes the survivors onto hardware next.

Virtual Try On Glasses: 7 Checks Before You Trust a Demo

Dave Gordon — Wed, 14 Oct 2026 13:15:28 +0000

TL;DR:
A virtual try on glasses demo is a best-case recording. I run seven checks on a live camera before I believe one: real-world scale, temple occlusion, head turns, lens rendering, jitter, bad light, and how a frame gets into the catalog. This post is a checklist. I have not published measurements here.

I look at a lot of virtual try on glasses demos for client projects. Most of them look fine in the first ten seconds, because the demo is a frontal face, good light and one hand-tuned frame. Shoppers do not behave like that. They tilt the phone, sit by a window and try thirty frames in a row.

So I test the way a shopper breaks things. Each check below is something you can run yourself in under a minute, on the device you actually care about.

What did I actually measure for this post?

No numbers, and I want that stated before the checks. This is step zero: the method. If I publish measurements later they will be per device, never averaged, with the phone model, OS version and chipset written next to every number. For now, if you run the checks yourself, you will know what to ask any vendor to show you.

How do you test virtual try on glasses on a real face?

Run the seven checks in this order. The early ones fail fastest.

1. Is the frame the right size, in millimeters?

A try-on that looks good but renders every frame at one default width is a styling toy.

Take a physical frame and read the numbers printed on the inner arm, for example 52-18-140. Those are lens width, bridge width and temple length, in millimeters. Note what they are not: 140 is the length of the arm that runs to your ear, not how wide the frame is. The front of that frame measures roughly 2 x 52 + 18 = 122 mm across, plus a millimeter or two of hinge. If a vendor quotes you the third number as a width, that alone tells you something.

Here is the arithmetic you can redo. Say the overlay renders that 122 mm front 2 mm too narrow. That is 2 / 122, about 1.6%. Small on its own, and you probably will not see it.

Now add the pupillary distance. Adult PD runs roughly 54 to 74 mm, with 63 mm a common middle. An estimate that is 3 mm off on a 63 mm PD is a 4.8% error, and it puts each lens center about 1.5 mm away from the eye it is supposed to sit in front of. Combine a 1.6% scale error with a 4.8% PD error and the frame stops sitting where the eyes say it should, even though neither number sounds alarming on its own. Illustrative numbers, so swap in yours.

What I look for: does the vendor use a stated scale reference, and does the same frame in two sizes render at two sizes?

2. Do the temples disappear behind the ears?

Turn your head slowly to the left. On a real face the far temple arm goes behind the ear and the near one stays visible. If the arms float in front of the cheek, the occlusion is missing. This is the single most common tell I see in cheap overlays.

3. What happens at 45 degrees and beyond?

Sweep from frontal to a hard yaw, then pitch up and down. Tracking that holds at 10 degrees and snaps at 40 loses the shopper exactly when they check the side profile. Landmark models tend to be happiest on near-frontal faces, which is why the sweep is worth doing. For reference, Google's MediaPipe Face Landmarker documents 478 landmarks per face, and even a model with that many points can lose them when half the face leaves the frame.

4. Do the lenses look like lenses?

Tinted, mirrored and clear lenses need different treatment. Clear lenses should not turn the eyes into a gray rectangle. Look for glare that moves when you turn, and for frame edges that stay crisp against hair and skin. If every material looks like matte plastic, the asset pipeline is flat.

5. Is the frame steady when you hold still?

Hold the phone against a stack of books and watch for the frame swimming a pixel or two on a face that is not moving. Jitter is cheap to hide in a recording and hard to hide live. I count it as a fail when I notice it without looking for it.

6. Does it survive a window behind you, and your own glasses?

Backlight, a dim room and a face already wearing glasses are three separate tests. Many tracking stacks quietly assume bare faces. If the shopper's own pair confuses the overlay, the first thing they will do is take them off, and many will not.

7. How does a new frame get into the catalog?

This is where projects go over budget, and it is the check almost nobody runs because it is not visual.

Ask four things, in this order:

What does the merchant supply per SKU? Product photographs, a studio capture session, or a 3D model someone has to build. These are three different budgets.
How many images per frame, and from which angles? The answer separates a flow you can run on the photos already in your product feed from one that needs a turntable.
Do colorways need separate capture? A catalog of 400 frames in three colors each is 1,200 SKUs or 400, depending on the answer.
How long does a collection take to go live, and who did the work in the number you were quoted? "Two days" means something different when the vendor's team did it. A photo-based flow and a 3D-modeling flow have very different costs, and a try-on that works for twenty hero frames and breaks at two thousand SKUs only works as a lookbook.

How do I score what I see?

I use a plain pass / partial / fail per check, written next to the device it ran on. The table is my own scoring sheet. The columns and the thresholds are mine, not an industry standard.

Check	What I look at	Fail looks like
1. Scale	Printed frame size vs on-screen size	Every frame renders the same width
2. Temples	Far arm during a slow head turn	Arms float in front of the cheek
3. Head turns	Yaw and pitch sweep	Overlay snaps or detaches
4. Lenses	Clear, tinted, mirrored	Everything looks matte
5. Jitter	Phone still against a stack of books	Visible swim on a still face
6. Light and own glasses	Backlight, dim room, glasses on	Frame lost or misplaced
7. Onboarding	Images and time per new SKU	Only hero frames are available

Every row in that table is stated in the text above, so nothing lives only in the table.

Should I test on my own phone or the vendor's?

Yours, plus the oldest mid-range phone in your test pool. A flagship is the best case for every one of these checks, and that is the device the demo was recorded on. I wrote up the same habit for face AR SDKs in 8 Face AR SDK Checks I Run Before Signing a Contract.

Why does color matter for glasses at all?

It matters less than for makeup, but it still shows up in tinted lenses and acetate frames under warm light. The long version of why rendering never matches the real thing is in 6 reasons virtual makeup rendering never matches the real shade.

What this does not cover

This checklist says nothing about purchase conversion, return rates or pricing. It also says nothing about which product to buy, because I have not run these checks against named products for this post. Treat any vendor claim about fit accuracy as a claim until you have run checks 1 to 3 yourself on a few real faces.

If you only run three

Run 2, 3 and 7. Temple occlusion and a hard head turn expose weak tracking in under a minute, and the onboarding question exposes weak economics before you sign anything. Everything else is polish you can negotiate.

Where Background Removal APIs Send Your Users' Video

Dave Gordon — Mon, 14 Sep 2026 19:13:00 +0000

Picking a background removal API is an engineering decision right up until it stops being one. The masks hold on hair and glasses, the latency fits, the integration is a two day job, and then a security reviewer at your own company or at your largest customer sends over a questionnaire asking where the frames go and who keeps them afterwards. The API reference does not cover any of that.

TL;DR

Remove.bg puts no duration on how long the images sent to it are kept, and it caps server logs at three months.
Photoroom's model training clause covers photos people send to its app, and the same text excludes API images.
VideoBGRemover publishes a full erasure sequence: live while the job is active, gone on a verified request, production erasure inside 30 days, backups seven days beyond.
Files on the fal CDN are readable by anyone holding the link unless you attach an access control list, and inputs inherit the account default rather than the header on the inference call.
Banuba's Face AR SDK is licensed by a token in .txt format with a published expiry ladder, and the same page never says whether the SDK phones home to check it.
One of the five is sales gated. The other four hand you a key at sign-up.

Everything here was read off vendor pages on 8 September 2026

I ran no tests for this one and there is not a millisecond anywhere below. Every claim comes from a vendor's own live pages as they stood on 8 September 2026, linked next to the sentence it supports. Those pages get edited, often quietly, and any figure that decides a build should come from the vendor in writing, addressed to you.

Does the video frame leave the device?

Four of the five are hosted services, so the media crosses the network before anything happens to it. Remove.bg's privacy policy, last updated 16 July 2025, set out the sequence plainly when I read it on 8 September 2026: images go up securely, get processed, and are provided for download. Photoroom, VideoBGRemover and Veed running on fal.ai work the same way.

Banuba's Face AR SDK is the one product here that is not a hosted service, since the processing happens inside your app rather than on a server you call. That settles the media path and leaves the license path open, which is where its published material stops short.
For the hosted four, the honest questionnaire entry is yes. Everything else is argued after that.

What does each vendor keep, and for how long?

	Banuba Face AR SDK	Remove.bg	Photoroom	VideoBGRemover	Veed on fal.ai
Where processing happens	On the device, inside your app	Vendor servers	Vendor servers	Vendor servers	fal's platform
Does user media leave the device	No	Yes	Yes	Yes	Yes
What is being compared	On-device SDK, licensed by a token in the client app	Hosted REST image service	Hosted REST service plus a consumer app	Hosted video service	A model run on fal's platform
Access	Sales gated: demo and commercial tokens are both issued by a person	Open self-serve with an API key	Open self-serve, custom enterprise pricing	Open self-serve, enterprise DPA available	Open self-serve
How long the vendor holds media	No media reaches a vendor server	"Delete them shortly after", no duration given	Not stated	Live while the account or job is active, erased on job deletion or a verified request	No CDN expiry stated
Tail after deletion, and logs	No media reaches a vendor server. Server logs not applicable	Server logs kept at most three months	Not stated	Production erasure inside 30 days, soft-delete systems and rolling backups seven days beyond	Inputs and outputs kept 30 days by default
Training use	No media reaches a vendor server	Not stated	Training row covers app photos, API images carved out	Does not train or improve models on customer content	Not stated
Named third-party audit	Not named	Not named	SOC 2 Type 2	Not named	Not named
Contract instrument named	Not named, terms go through an account manager	EU and UK representative named (EDPO), rights listed	Not named	DPA incorporating the 2021 Standard Contractual Clauses	Not named
Control you can set yourself	Keep the token on your own server rather than in the app bundle	Not offered	Not offered	Job deletion and verified erasure requests	`X-Fal-Store-IO: 0` on the request, ACL on the upload
Still open on their own pages	Whether the SDK makes a network call to validate the token	How long processed images are actually kept	Retention duration for API media	No named outside audit	CDN file expiry, and what happens to assets at account end

Every cell above is what that vendor's own published pages said when I checked them on 8 September 2026, and an empty-looking cell means I found nothing on the subject, not that the vendor does the opposite. None of it is a measurement.

The first row exists so the first column does not get scanned as like-for-like. An on-device SDK and a hosted video service fail a questionnaire in completely different places, and putting them side by side is only useful if you keep that in view.

Access splits four to one. Remove.bg, Photoroom, VideoBGRemover and fal all issue a working key at sign-up. Banuba is the gated one: a demo credential comes from a sales manager or the website request form, and a commercial credential comes from an account manager after payment. Sales gating is not automatically a problem, but it does mean your evaluation timeline includes somebody else's calendar.

The most specific writing about retention here belongs to VideoBGRemover. Its privacy page says uploaded videos, masks, previews and outputs stay available while the account or job is active and are erased when you delete the job or send a verified request, that production erasure completes within 30 days, and that soft-delete systems and rolling backups may hold copies for seven days beyond that. No other page here puts a tail on the number. Remove.bg's three month ceiling on server logs and the 30 day payload default in fal's media expiration docs are real figures too, though both of those describe logs and JSON rather than the media itself.

On model training, Photoroom and VideoBGRemover arrive at similar comfort from opposite directions. Photoroom carves API traffic out of model improvement in its own legal text. VideoBGRemover rules out training on customer content across the whole service.

Photoroom is the only one of the five naming an outside auditor: its API page states SOC 2 Type 2. If your reviewer's checklist has a certification line on it, that is currently a one horse race here.

For a contract instrument, VideoBGRemover names an enterprise DPA incorporating the European Commission's 2021 Standard Contractual Clauses, and Remove.bg names EDPO as its EU and UK representative alongside the usual list of data-subject rights. The other three leave that conversation to sales.

fal is alone in handing you controls you operate yourself. Everyone else describes their own conduct and asks you to accept the description, which is a weaker artifact to walk into a review with, however honest the description turns out to be.

Can anyone else open the file the API handed back?

On fal, the default is yes. fal's file access control docs state that every object on the fal CDN carries an access control list, and that by default CDN URLs are reachable by anyone who has the link. A long random URL does not make an object private, and links leak through logs, support tickets and screenshots.

Inputs are a separate case on the same page. The SDKs push them to the CDN before the inference call runs, so they are not governed by the ACL header you set on that call and instead inherit the account default. To constrain an input you set the ACL on the upload itself.

To size your own exposure, with named inputs you can redo on paper:

Start with clips per user per day. Mine here is one, and it is yours to substitute.
Multiply by daily active users who touch the feature. I am using 500, an illustrative figure to replace with your own.
500 x 1 = 500 input objects landing on the CDN each day.
CDN URLs are public by default, so unless an ACL is attached at upload time, those are 500 publicly reachable URLs added per day.
The JSON payloads default to 30 days, so 500 x 30 = 15,000 request records held in the history at any steady state moment.

In elapsed terms, that history never empties while the feature is switched on, because the oldest record ages out 30 days after it was created and a fresh one arrives the same day.

Are my users' images used to train the vendor's models?

Photoroom's privacy policy carries a retention table with a row titled "Training our AI model", covering photos and images that people sent to Photoroom, on a legitimate interest basis, held for as long as training requires. The same document then states that this model improvement does not apply to images processed through the API. That clause is about app users, not API traffic, and getting the distinction backwards in a questionnaire will cost you a week of email.

VideoBGRemover takes the other route and states flatly that it does not use content people send it, or content it generates, to train or improve machine learning models. That single sentence covers the whole service.

Remove.bg's policy, fal's legal pages and the Banuba licensing page say nothing either way, going by what was up on 8 September 2026. That absence does not amount to a denial, and it stays open until the vendor answers in email.

Who else on my team can see the requests?

fal's privacy policy is unusually direct about this. If you are a member of a Team Account, it says, other members of that Team Account may view billing information, API keys, and AI model requests, including any input and output data.

Think that through before you invite the contractor. Whoever sits in the account can read the media your users sent, and your key too.

The other four say nothing about who inside a customer account can read request bodies, in anything they published as of 8 September 2026. If the question comes up, and it always has for me, you will be relaying an email rather than pointing at a URL.

What happens to the feature when the license lapses?

Banuba is the only one of the five that publishes what happens after the license runs out. Its token management documentation describes the token as an automatically generated set of characters in .txt format, unique to each client, that activates the licensed SDK functionality in the client app.

The expiry ladder on that page has three rungs. A credential that has not expired means the SDK works as expected. One expired by a month or less still works, with a watermark displayed. Past a month, the functionality will not work.

The same page recommends keeping the credential on your own server instead of in the app bundle, and the reasoning is operational: ship it inside the binary and every renewal means a new build queued for App Store or Play Store approval.

What the page does not state is whether the SDK contacts Banuba at runtime to validate the credential. The expiry ladder is written down. Whether a network call sits behind it is left open, so a reviewer asking about outbound traffic needs that confirmed by the vendor.

Among the hosted four, only VideoBGRemover ties asset availability to account state in its published terms, saying that videos, masks, previews and outputs stay available while the account or job is active. For Remove.bg, Photoroom and fal, I found nothing published about what becomes of stored assets once an account ends.

What will the vendor put in writing?

Photoroom offers an attestation from an outside auditor plus an explicit carve-out for API traffic. VideoBGRemover offers a DPA built on the 2021 Standard Contractual Clauses for enterprise customers under GDPR. Remove.bg names a representative for EU and UK data subjects and enumerates the rights those people hold. fal offers something different in kind, two controls you operate yourself: the store header on the request and the ACL on the upload. Banuba issues its commercial license through an account manager after payment, so the terms conversation happens with a person and whatever you need has to end up in that thread.

Two of the five publish noticeably less than the rest. Remove.bg gives no duration whatsoever for the images sent to it, only that they go "shortly after", and shortly is not a number your reviewer can log. Banuba's material covers the license credential thoroughly and stops before the SDK's runtime network behavior, which leaves the most basic question about an on-device product unresolved on the public web.

Ask for the four answers in writing before you sign

Four items close most of the gaps above, and they fit in one email.

A number, in days, for how long processed media is held, and a second number for the backup tail after removal.
Whether traffic through the API is excluded from model training, stated separately from whatever applies to their consumer app.
A named list of who inside your own account can read request inputs and outputs.
What becomes of stored assets when the contract or license ends.

Each of the five publishes some of those, and none of the five had all four on the pages I went through on 8 September 2026. Whichever come back thin are the ones to raise before the integration lands in a sprint, since they get harder to ask once the feature ships.

5 Places a Real-Time Camera Filter Drops Frames

Dave Gordon — Thu, 03 Sep 2026 11:42:31 +0000

A real-time camera filter has about 33 milliseconds per frame at 30 FPS. When a filter that looked fine in a demo stutters on a real handset, the loss is almost always in one of five places: format conversion, frame backpressure, where inference actually executes, synchronous readback, or sustained thermal load. Here is how I find which one.

TL;DR

The budget is arithmetic you can do in your head: 1000 / 30 = 33.3 ms per frame, and 1000 / 60 = 16.7 ms.
Converting a camera frame to RGBA before anything else executes is the most common self-inflicted cost, and a byte count exposes it before you open a profiler.
If you never chose a backpressure strategy you already have the non-blocking one, which puts a leaked frame reference ahead of the strategy on the list of suspects.
A model that reports a plausible inference time can still be executing somewhere you did not pick, which makes the number describe a different pipeline from the one you are shipping.
Reading pixels back synchronously drains the queue you spent the whole frame filling.
Thermal decay does not show up in a 30 second run, which is why most of these pipelines pass their own tests.

No device numbers in this piece

I have not benchmarked anyone's SDK here and there are no handset figures below. A number without a named device, an OS build and a thermal state cannot be reused by anyone reading it. What follows is the order I work through when a camera pipeline misses its budget, plus the arithmetic that tells you whether a given stage can possibly fit before you go looking for it in a trace.

What I did do is verify every mechanism against primary documentation and link it: the image analysis contract and frame format from Android's own docs, the delegate behaviour from the TensorFlow Lite repository and its issue tracker, the profiling tools from Google and Apple, and the readback stall from a handset vendor's own optimisation guidance. The arithmetic is reproducible on paper, two divisions and a multiplication, so if any of it is wrong you can check it in about a minute.

The five places below are ordered by how often they turn out to be the answer in my own work. Ranking them by theoretical cost would give a different order.

The budget, before you profile anything

Two divisions set the ceiling for everything else.

At 30 FPS you have 1000 / 30 = 33.3 ms to receive a frame, convert it, run whatever model you are running, render, and composite. At 60 FPS that halves to 1000 / 60 = 16.7 ms. Each stage you add spends part of that, and the preview is only as smooth as its slowest frame.

Resolution is the other multiplier, and it is worth doing on paper first. A 1080p frame is 1920 x 1080 = 2,073,600 pixels. A 720p frame is 1280 x 720 = 921,600 pixels. That is a factor of 2.25, so any per-pixel stage costs 2.25 times more at 1080p than at 720p before you change a line of code.

I start here because the arithmetic frequently ends the investigation on its own. If a stage has to touch every pixel of a 1080p frame and you have four such stages, the budget was gone before the model loaded.

Do I need 60 FPS, or is 30 enough?

For a camera filter that a person is looking at while moving their own face, 30 is usually enough and 60 is usually not worth what it costs, because halving the budget to 16.7 ms tends to force a resolution cut that is more visible than the smoothness gain.

The exception is anything the user physically tracks with their hand or head, where the extra samples do register. I would decide this by shipping 30 and instrumenting it before assuming the answer, since the 95th percentile at 30 FPS is a better predictor of perceived quality than the target rate.

1. The format conversion you did not know you were paying for

Camera hardware on Android hands you YUV_420_888, a planar YCbCr format with 8 bits per sample and chroma subsampled 4:2:0. Most image processing code, and most tutorial code, wants RGBA instead. So a conversion gets inserted, often before anyone has decided whether it is needed.

The cost is a byte count you can compute without a profiler.

At 4:2:0, a pixel averages 1.5 bytes. A 1080p YUV frame is 2,073,600 x 1.5 = 3,110,400 bytes, roughly 3.11 MB. The same frame as RGBA_8888 at 4 bytes per pixel is 2,073,600 x 4 = 8,294,400 bytes, roughly 8.29 MB. Running that conversion at 30 FPS means writing 8.29 x 30 = 248.7 MB per second, versus 93.3 MB/s if you had left the frame in its native layout.

That traffic is not free on a mid-range memory bus, and it happens before your actual effect has done anything.

What I look at: whether the conversion is needed at all. A shader can sample the Y and UV planes directly and do the color transform on the GPU as part of a pass you are already running. CameraX will also hand ImageAnalysis an RGBA output natively if you genuinely need RGBA, which at least moves the conversion out of your own loop.

2. Why does my camera preview freeze after a few seconds on Android?

The cause is usually more boring than a slow filter.

If you are using CameraX ImageAnalysis, the analyzer contract puts the responsibility on you to close each image. Google's documentation is explicit that if images are not closed, they may block further images from being produced, which stalls the preview, or get dropped according to whichever backpressure strategy is configured. Earlier versions closed the image for you; current ones do not, so that multi-frame analysis stays possible. Close the ImageProxy, not the Media.Image it wraps, because closing the wrapped image directly breaks the image sharing mechanism inside CameraX.

So a freeze that arrives a second or two in, instead of immediately, usually means a leaked frame reference. Filter speed is a red herring in that case.

The second half of this is the strategy itself, and it is worth knowing what you already have. Non-blocking is the default: STRATEGY_KEEP_ONLY_LATEST caches only the newest frame in a buffer one deep and overwrites it while you are still working on the previous one. That is almost always what a live filter wants, because a queued frame is a stale frame by the time it renders. So if you never set a strategy, you are already on the non-blocking one, which means a freeze is far more likely to be a frame you did not close than a strategy you did not pick.

The blocking variety has to be chosen deliberately. STRATEGY_BLOCK_PRODUCER queues images and starts dropping only once the queue is full, and Android's documentation notes that the blocking occurs across the entire camera device scope: if several use cases are bound to that camera, all of them stall while CameraX works through the queue. That is the mechanism that turns a slow analyzer into a frozen preview instead of a dropped frame or two.

// androidx.camera:camera-core:1.3.4
val analysis = ImageAnalysis.Builder()
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .build()

analysis.setAnalyzer(executor) { image: ImageProxy ->
    try {
        frameClock.mark()      // see the timing helper below
        renderEffect(image)    // your pipeline
    } finally {
        image.close()          // ImageProxy.close(), required including on the failure path
    }
}

// When the screen goes away, including when the user backgrounds the app
// mid-session rather than pressing a stop button:
analysis.clearAnalyzer()
cameraProvider.unbind(analysis)

The finally block matters more than it looks. One exception on one frame, with close() sitting on the happy path only, produces the symptom above and nothing in the log that points at it.

3. The model may not be running where you think

A GPU delegate that reports a believable inference time is not proof that inference ran on the GPU. TensorFlow Lite's GPU delegate documentation covers which operations the backend supports, and unsupported ops may fall back to CPU depending on version and configuration. Reported behavior is not uniform: developers have filed cases where the expected fallback did not happen and produced a runtime error instead, and cases where it silently did.

Either way, a plausible millisecond figure comes back. It may be describing an execution path other than the one you intend to ship.

That picture comes from the documentation and the issue tracker, so verify it against the version you actually build with before you rely on it.

What I look at: the execution path in a trace, instead of trusting the timing. On Android that means Perfetto, which has shipped in Android system images since Android 9 and has been enabled by default on most devices since Android 11. On iOS the Metal debugger and Metal System Trace in Xcode show the parallel CPU and GPU timeline, which is where a stage that claimed the GPU but ran elsewhere becomes obvious.

I have not tried to name the exact log line for a fallback here. The strings vary across versions and I could not verify a current one while writing this, so I have left it out. The symptom is the reliable signal: GPU utilization that stays flat while inference time stays constant.

4. Why does my frame rate drop when I read pixels back from the GPU?

If any stage pulls pixels back from the GPU with a blocking read, the pipeline empties.

The mechanism is well understood and worth stating precisely, because it explains why the cost never shows up where you look for it. A blocking readback cannot return until every queued draw command ahead of it has completed, so the CPU waits for the GPU to drain. The call itself is cheap. The stall is what costs you, and it lands on whichever line touches the data.

Samsung's own OpenGL ES guidance, which is a handset maker writing about its own hardware, is one of the clearer public writeups of the pattern. The standard remedy is a pixel buffer object: bind the buffer, issue the read, and it returns immediately while the copy happens asynchronously, then use a fence to check for completion a frame or two later.

The question I ask first: is the readback needed on every frame at all? Histogram, autoexposure and face-region logic often tolerate a fraction of the preview rate, and dropping one of those to every fifth frame is usually a larger win than optimizing the read itself.

5. The run was too short to fail

A pipeline that holds its budget for 30 seconds tells you very little, because a phone under sustained camera load heats up and the governor responds by lowering clocks. The camera sensor, the GPU and the neural accelerator all work at once, which is close to a worst case for sustained power draw.

What I look at: I run one scene for several minutes, log per-frame intervals throughout, and compare the last minute with the first. If the tail is materially worse than the head, the problem is thermal and no amount of per-stage optimization will move it. Screen brightness, whether the device is charging, and case material all change the result, which is why this one has to run on real hardware in a realistic state, never on a bench.

It is also the step teams skip most often, since it is slow and it needs a person holding a phone.

Here is the timing helper referenced in the snippet above. It keeps the full interval history, since averaging smooths away the exact spikes you are hunting.

class FrameClock(private val capacity: Int = 600) {
    private val intervalsMs = ArrayDeque<Long>(capacity)
    private var lastNs = 0L

    fun mark() {
        val nowNs = System.nanoTime()
        if (lastNs != 0L) {
            if (intervalsMs.size == capacity) intervalsMs.removeFirst()
            intervalsMs.addLast((nowNs - lastNs) / 1_000_000)
        }
        lastNs = nowNs
    }

    /** p is 0.0 to 1.0. Returns null until at least one interval is recorded. */
    fun percentileMs(p: Double): Long? {
        if (intervalsMs.isEmpty()) return null
        val sorted = intervalsMs.sorted()
        val index = Math.round(p * (sorted.size - 1)).toInt()
        return sorted[index]
    }
}

Read percentileMs(0.95). A pipeline averaging 28 ms with a 95th percentile of 70 ms is visibly janky, and the average on its own will never say so.

The budget sheet I actually fill in

I keep this per device, per resolution. It is the artifact I hand to whoever asks why the filter is slow.

The budget sheet I actually fill in

Stage	The budget I start from	How I confirm it	Cheapest fix
Frame delivery and format	2 to 4 ms	Byte count on paper, then a trace	Sample YUV planes in the shader, skip the conversion
Backpressure and queueing	0 ms if correct	Frame interval history at p95	STRATEGY_KEEP_ONLY_LATEST plus close() in finally
Inference	Largest single share	Trace the execution path itself	Confirm the backend, then lower input resolution
Render and composite	4 to 8 ms	GPU timeline in Perfetto or Metal System Trace	Merge passes, avoid full-frame intermediates
Readback	0 ms if asynchronous	Look for a CPU wait beside a drained GPU queue	Pixel buffer object plus a fence, or run it less often
Thermal headroom	Whatever is left	Multi-minute run, last minute versus first	Lower resolution or frame rate under sustained load

Those figures are my own starting budgets. I have yet to see a device leave all six of them intact, which is the point of filling in the measured column beside them. The gap between the two columns is the article you should be writing about your own app.

Where this usually lands

In most stuttering camera pipelines I have looked at, the effect itself was cheap. The time went to a frame copied into a format nothing needed, or queued when it should have been dropped, or read back synchronously, or executed somewhere nobody checked, and none of it surfaced in a 30 second run on a cool phone.

Do the two divisions and the byte count before you open a profiler. They cost a minute and they routinely tell you the answer, or at least tell you which of the five places to look in first.

6 Background Removal API Tests I Run Before Shipping Video

Dave Gordon — Fri, 21 Aug 2026 13:33:50 +0000

TL;DR:

Real-time background removal is a frame budget problem before it is a quality problem.

Six tests, in this order:

- does it fit the budget on your worst supported phone,
- did the run actually use the accelerator you think it did,
- what happens at the hair line,
- does the mask flicker between frames,
- what breaks when the lighting is real,
- and what does it cost you once you count platforms instead of license fees.

Most shortlists lose half their candidates on tests 1 and 5.

I have swapped segmentation backends three times across two camera apps, and every swap started the same way: the library was fine in the sample project and came apart on a real device with a real user in front of it. So I stopped evaluating these things by looking at output stills.

This is the harness, not the results table. I am not publishing my numbers here, because my numbers are about my pipeline and my support matrix, and neither of those is yours. A frame budget is derived from your capture rate and your effects stack, and a mid range phone in one support range is a flagship in another.

What transfers is the setup and the order the tests run in. Copy the rig, run your own candidates through it, and the numbers you get will actually apply to the thing you are shipping.

The rig

Before the first test, fix these so the results mean something across candidates.

Three device tiers, named, not averaged

A current or previous cycle flagship, a mid range phone around three years old, and the cheapest device with real share in your own analytics. Record the model, OS version and chipset for each. Score every test per device. An average across the matrix hides the phone your median user is holding, which is the only phone the decision actually turns on.

One app state, held constant

Camera open, your effects stack running, screen at full brightness, no other app in the foreground. Candidates measured in different app states are not comparable, and the tempting shortcut of benchmarking inference in isolation produces the number that looks best and matters least.

A warm device, and a stated warm-up

Decide how long the app records before you take the sustained reading, write that duration down, and use the same one every time. Thermal state is the variable that quietly decides half of these comparisons.

One fixed clip set

The same recordings, reused for every candidate: a plain wall clip, a still subject clip, and the lighting cases from Test 5. New footage per candidate means you are scoring the footage.

Pinned versions

SDK or model version for every candidate, written next to every result. Vendors ship model updates that move these numbers, and a result without a version cannot be reproduced or re-checked later.

Why does the frame budget decide this before quality does?

Most background removal comparisons open with mask quality. That is the wrong end. If inference does not fit inside a frame, quality is irrelevant, because you are already dropping frames and the preview stutters.

Work out your own budget before you look at a single demo:

Take your target capture rate. 30 fps for most social and editor video, 60 fps if the feature has to feel like the native camera.
Divide 1,000 ms by that rate. 30 fps gives you 33.3 ms per frame. 60 fps gives you 16.7 ms.
Subtract the rest of the pipeline. Camera capture, color space conversion, your own filters or effects, the render pass, and encoding all take a slice.
What is left is your segmentation budget. Run that subtraction rather than assuming you have the whole frame, because in the pipelines I have worked in, the remainder has never been more than half.
Compare that number against measured inference time on your slowest supported device, not your average one and not the phone on your desk.

Write that number down. Every test below is scored against it.

Test 1: Does it hold the budget on your worst target device?

Run the candidate on the cheapest and oldest tier, in the app state fixed above.

Three things to capture.

Cold inference time

The first frames after model load are always slower. If your feature is triggered mid session, users feel this directly.

Sustained inference time after thermal throttling

This is the number that matters and the one almost nobody publishes. Measure it rather than extrapolating from the cold figure, and expect the two to be far enough apart that the cold figure is useless for capacity planning. If a vendor publishes one inference time without saying whether the device was warm, assume it is the cold one.

Memory ceiling

Segmentation model plus video buffers plus encoder is where low RAM Android devices start getting killed in the background. Watch dumpsys meminfo for your process across a long recording rather than reading a peak off a profiler graph once.

Test 2: Did the run you just measured actually use the accelerator you think it did?

This test exists because Test 1 produces a plausible looking number whether or not it measured the pipeline you intended.

The GPU delegate fails to initialise, the library falls back to CPU without raising anything to your code, the run completes, and you write down a figure that describes a completely different execution path. Nothing in the output tells you. The mask still looks correct, the app does not crash, and the number is wrong in a direction that will not reproduce on the next build.

ERROR: Failed to apply GPU delegate.

If that line is in your logs, every timing you collected in that session is CPU timing. Two habits fix this permanently:

Assert the execution path per run. Log which delegate or backend was actually created, and discard any run where the assertion is missing rather than any run that looks slow. Slow runs are data. Runs of unknown provenance are not.
Fail closed. A benchmark harness that silently completes when initialisation failed is worse than one that crashes, because it produces numbers you will act on.

The same applies to any library that quietly reduces input resolution or frame stride under load. If the candidate has an adaptive mode, either turn it off for the measurement or record what it chose, because otherwise you are comparing two vendors that each decided independently what to sacrifice.

Test 3: What happens at the hair line?

Every segmentation model produces a clean silhouette. The difference between them lives in about forty pixels around the edge.

Use the plain wall clip and score four cases:

Loose or curly hair. Naive segmentation cuts a helmet shape and drops the strands entirely.
Glasses frames, especially thin metal ones, which models frequently classify as background and slice through.
An object held in hand, like a mug or a phone. Person only models will amputate it. Whether that counts as a bug depends on your use case, so decide deliberately rather than finding out in review.
The gap between arm and torso when someone puts a hand on their hip. Soft mask models tend to fill it in.

Score these against a hard edged replacement background, not a blur. Blur hides everything, which is exactly why so many demos use it.

Test 4: Does the mask flicker between frames?

This is the failure mode that gets caught in QA rather than in evaluation, and it is the most expensive one to fix late.

A per frame image segmentation model has no memory of the previous frame. Feed it video and the mask boundary jitters frame to frame even when the subject is completely still. The stills look perfect. The clip looks cheap.

Two ways a library can address it.

Temporal architecture

Robust Video Matting is the reference example, in that it carries state between frames with a recurrent design instead of treating each one independently. Check the license before you get attached to it: the code was re released under GPL-3.0 in September 2021, which for most proprietary mobile apps means you are studying the architecture rather than shipping the repo.

Your own smoothing

You can apply temporal filtering to a per frame mask yourself, but you are trading latency for stability, and that trade comes straight out of the budget you calculated above.

The test is simple. Take the still subject clip, thirty seconds of someone barely moving, then step through the export frame by frame and watch the boundary. If it breathes, you either fix it or you ship it.

Test 5: What breaks when the room does?

Evaluation footage is always shot in good light. Your users are on a couch at 9pm with a window behind them.

Run each candidate through:

Backlight, with a bright window behind the subject. Classic failure: the person is segmented as background.
Low light, where sensor noise destabilises the edge and Test 4 gets much worse.
A busy background with furniture, a doorway, or a poster of a person on the wall.
Two people in frame, where some models pick one, some merge both, and some alternate between the two across frames.
Movement toward and away from the camera, which changes subject scale and is where fixed input resolutions show their limits.

If you automate one part of this harness, automate this one. It is the section that regresses silently when a vendor ships a model update.

Test 6: What does it actually cost you per platform?

The license fee is the visible number and usually not the deciding one. Count the real cost like this:

List every platform you ship on. iOS, Android, web, desktop, and any framework layer such as React Native or Flutter.
Check what the candidate covers natively. Apple's Vision framework offers person segmentation with selectable quality levels, and it is iOS and macOS only, so an Android build needs a second solution and a second set of edge case behaviour to test.
Multiply the platforms you have to solve separately by the integration and QA time for one platform. That multiplier, not the sticker price, is where the budget goes.
Add the ongoing cost. Two segmentation backends means two model update cycles, two sets of device regressions, and visible quality differences between your iOS and Android users.
Check the deployment model, which is where the arithmetic gets decisive.

Can you use a cloud background removal API for live video?

Not realistically. Cloud background removal is priced per image and needs an upload per frame, so a 30 fps preview is 30 requests per second per user with a network round trip on every one. That is the right tool for batch photo processing and the wrong one for a live camera preview, and it fails on arithmetic before you get to quality.

For the free, on device end of the shortlist, MediaPipe Image Segmenter is Apache 2.0, and ML Kit selfie segmentation is a closed source SDK that is free to use. Both run on device across iOS and Android, which makes them the honest baseline. Anything commercial has to beat them on the tests above by enough to justify the line item, and often it does, because Tests 1, 4 and 5 are where a general purpose model gives ground to one tuned for video.

The scorecard

Test	What it measures	Pass condition
1. Frame budget on worst device	Cold and sustained inference time, memory ceiling	Sustained time fits the segmentation budget on the slowest supported device, not the average one
2. Execution path	Whether the accelerator you configured is the one that ran	Delegate asserted in the log on every run, runs without the assertion discarded
3. Hair line	Edge quality on hair, thin frames, held objects, arm to torso gaps	Scored against a hard edged background, not a blur
4. Temporal stability	Mask jitter across frames on a still subject	Boundary holds frame to frame, or the smoothing cost is inside the budget
5. Real conditions	Backlight, low light, busy background, two people, scale change	No condition drops below your quality floor, and the suite is automated
6. Cost per platform	Platforms solved separately, integration and QA multiplier, deployment model	One integration covers your matrix, or the second one is budgeted

Pick your slowest supported device and your worst lighting scenario first, then evaluate. Run these six in order and you will usually eliminate half the shortlist before pricing comes up, and you will eliminate it for reasons you can put in front of a product manager.

If you run this harness on something I have not tried, I would like to hear what broke.