We build a virtual mirror for hair salons: a tablet
camera feed where the customer's hair changes colour in real time while they move their head.
Two constraints shaped every decision.
The first is privacy: no frame ever leaves the device. Everything runs locally — the model,
the fonts, the runtime — with no CDN dependency, because a CDN request would both break
offline use and leak the salon's IP to a third party.
The second is hardware: salons don't buy flagship tablets. We had to work on whatever
mid-range Android is on the counter.
This is what we learned making it fast enough. All numbers were measured on real devices with
diagnostics built into the engine — none of them are estimates.
The bug that looked like slowness
Our starting point was bad in a confusing way. A Pixel 8 Pro ran at 9 fps — poor, but working.
A Samsung A56 (Exynos 1580, Mali GPU) showed no colour at all.
The cause took a while to find, and it's the most useful thing in this article:
MediaPipe's ImageSegmenter with the GPU delegate was created without throwing, ran in 5 ms,
and returned zero hair pixels. Both the confidence mask and the category mask came back
empty. Every time.
Our fallback to CPU only triggered on exceptions. There were no exceptions. So it never
triggered, and the app simply looked slow instead of broken.
A clean comparison, same scene minutes apart:
| Delegate | Mask coverage | Time per frame |
|---|---|---|
| CPU | 1.5–2.1 % | 483 ms |
| GPU | 0.0 % | 5 ms |
Note the trap in that table. The GPU path is 96× faster precisely because it isn't doing
anything. If you benchmark by timing alone, the broken path wins.
The lesson: don't assume that a delegate which constructs successfully actually works.
Check the output, not the exit code. We now measure mask coverage and surface it in the
diagnostics.
Detecting it at runtime
The fix is a fallback that rebuilds the engine on CPU. The rule matters more than it looks:
After five consecutive empty masks, having never seen a good one in the whole session,
rebuild on CPU.
The condition is "never seen a mask", not "empty right now". An empty mask right now is
completely normal — it happens whenever nobody is in front of the lens. We had to encode the
difference between broken and nobody's there.
On the A56 this took us from 2 fps to 14 fps with colour applied.
A diagnostic that costs more than what it measures
While chasing this, we left outputCategoryMask enabled from an earlier experiment. The
diagnostic itself became a significant part of the frame budget. Worth remembering when your
measurements start shaping the thing you're measuring.
It was never the model
With the GPU path honest, we profiled the Pixel 8 Pro's 9 fps. The model inference cost 6 ms.
The other 126 ms were ours:
- 66 ms in a JavaScript loop recolouring pixels one at a time
- two GPU→CPU readbacks per frame, one of them costing 123 ms
That's the whole story of most "the ML model is too slow" complaints we've had since: the
model was never the problem. The problem was everything we did around it.
Fix 1 — recolour in a fragment shader (66 ms → 0)
The per-pixel JavaScript loop moved into a fragment shader. This is the obvious one, and it's
free: the pixels are already on the GPU.
Fix 2 — decouple the mask from the draw
Hair does not move at 60 Hz. We were segmenting every frame for no reason.
segmentForVideo is synchronous, and on an Exynos it costs 437 ms of blocked main thread.
Now we segment one frame in every PASSO_SEG, and draw the video on all of them using the
most recent mask. The step adapts to the device.
This is the change that fixed the A56, and it's worth stressing: we expected to need a Web
Worker and we didn't. Decoupling cadence was enough.
Fix 3 — share the WebGL context with MediaPipe (123 ms → 0)
This was the big one. We were copying the mask out of MediaPipe and back into our own
pipeline, round-tripping through the CPU.
MediaPipe accepts a canvas in its options. Pass it the same canvas you're drawing on, and
you can take the mask with getAsWebGLTexture() — it never leaves the GPU.
One catch, learned the hard way: MediaPipe recycles that texture under your feet. Binding
it directly made the colour flicker. We copy it into our own texture with a shader, which is
still enormously cheaper than a CPU round trip.
Fix 4 — drop preserveDrawingBuffer (5 ms per frame)
preserveDrawingBuffer: true costs real time on every single frame. We only needed it to
capture a still of the current look, which is a rare user action.
We defer the capture to the next draw call, within the same task, and read it there. Verified
on device.
Two more things worth knowing
MediaPipe will not start inside a type: "module" worker. It dies with
ModuleFactory not set, because its WASM loader registers the factory via importScripts(),
which doesn't exist in module workers. If you want MediaPipe in a worker, you need a classic
worker — or you don't get one.
Freeze the frame in a texture before you segment. If you segment first and draw after,
the video has already advanced, and the hair lags behind the head permanently. Not
occasionally. Always.
Pick the specialised model. We were running selfie_multiclass, which segments six classes
when we needed one. Switching to the dedicated hair_segmenter: 780 KB instead of 16 MB, and
the official Pixel 6 figures are 58 ms vs 217 ms on CPU, 52 ms vs 71 ms on GPU. Our consumer
APK dropped from 40.7 MB to 25.8 MB.
Where we landed
132 ms → 17 ms per frame. 9 fps → 15 fps on the Pixel 8 Pro, and 2 → 14 on the A56.
You'll notice 17 ms per frame should mean roughly 58 fps, and we get 15. That's honest and
worth explaining: the remaining ceiling is no longer in our code. It's in how the WebView is
composited inside Flutter, and we haven't opened that front yet.
We also kept the old CPU 2D path as a fallback for devices without WebGL2. That's not
nostalgia — a canvas has exactly one context. If you request WebGL and it fails, you can't
fall back to 2D on the same canvas. You have to decide before you commit.
The takeaway
Four of the five wins here came from removing work we had added ourselves, not from making the
model faster. And the single largest discovery — the silently empty GPU delegate — wasn't a
performance problem at all. It was a correctness bug wearing a performance costume, and it was
invisible until we measured the output instead of the duration.
If you're building anything similar: instrument mask coverage, not just milliseconds. The
frame that takes 5 ms might be the one doing nothing.
The engine described here ships in Miraviso, a consultation tool for
hair salons: the customer sees the colour and the haircut before the stylist starts. Happy to
answer questions about the WebGL side in the comments.
Top comments (0)