Wiring Android's CameraX to a Quantized Face Mesh Model for Real-Time AR Makeup

---
title: "Real-Time AR Makeup on Android: CameraX + MediaPipe Face Landmarker Under 40ms"
published: true
description: "Wire CameraX ImageAnalysis to MediaPipe Face Landmarker v2 for real-time AR makeup on Android. GPU delegate, 478-landmark mesh, blend shapes, and frame pacing under 40ms on Snapdragon 7-series."
tags: kotlin, android, mobile, architecture
canonical_url: https://mvpfactory.co/blog/camerax-mediapipe-ar-makeup-android
---

What We Are Building

By the end of this workshop you will have a production-grade AR makeup pipeline on Android that stays under 40ms end-to-end on mid-range Snapdragon 7-series hardware. That means CameraX feeding frames into MediaPipe Face Landmarker v2, blend shape coefficients driving makeup deformation, and compositing kept entirely on-GPU.

Get any one piece wrong and you either drop frames or melt the battery. Let me show you the architecture that holds together in production.

Prerequisites

• Android project targeting API 26+
• MediaPipe Tasks Vision dependency added to build.gradle
• Basic familiarity with CameraX and OpenGL ES 3.0
• A physical device with Adreno or Mali GPU for meaningful profiling

The Frame Budget First

Before touching code, understand why 40ms. At 30fps you have 33ms per frame. At 60fps, 16.6ms. The 40ms ceiling is the pragmatic limit for Snapdragon 7-series — it covers inference, compositing, and display pipeline overhead without triggering thermal throttling in a typical five-minute session.

MediaPipe Face Landmarker v2 with GPU delegate runs at roughly 8–12ms on that hardware tier. That leaves ~28ms for frame acquisition, format conversion, makeup compositing, and display submission. Tight, but achievable if you are disciplined about where the time actually goes.

Step 1: Wire CameraX With the Right Backpressure Strategy

Use ImageAnalysis, not Preview, for inference. Preview gives you no backpressure control.

val imageAnalysis = ImageAnalysis.Builder()
    .setTargetResolution(Size(640, 480))
    .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
    .setOutputImageFormat(ImageAnalysis.OUTPUT_IMAGE_FORMAT_RGBA_8888)
    .build()

imageAnalysis.setAnalyzer(inferenceExecutor) { imageProxy ->
    val bitmap = imageProxy.toBitmap()
    landmarker.detectAsync(
        MPImage.fromBitmap(bitmap),
        imageProxy.imageInfo.timestamp
    )
    imageProxy.close()
}

STRATEGY_KEEP_ONLY_LATEST is non-negotiable. Under inference backpressure you drop frames rather than queue them. Queuing is how you accumulate 300ms of lag that users perceive as the makeup chasing their face.

Step 2: Extract Blend Shape Coefficients From the 478-Point Mesh

Face Landmarker v2 produces a canonical 478-point mesh. For makeup, you operate on three sub-regions:

• Lips: landmarks 61–291 (outer contour + inner vermilion boundary)
• Eyes/lids: landmarks 33–263 (with dedicated lid-crease indices)
• Cheeks: landmarks 50, 280, 330, 100 as anchor quads for blush polygon fill

The 52 blend shape coefficients are ARKit-compatible. For expression-driven makeup the ones worth reacting to are eyeBlinkLeft, eyeBlinkRight, jawOpen, and mouthSmile*.

val result: FaceLandmarkerResult = // from callback
val blendShapes = result.faceBlendshapes().get()[0]
val smileCoeff = blendShapes
    .find { it.categoryName() == "mouthSmileLeft" }
    ?.score() ?: 0f

// Deform lip mesh UVs proportionally
lipMeshUVs = deformLipUVs(baseLipUVs, smileCoeff)

Step 3: Configure the GPU Delegate Correctly

Here is the gotcha that will save you hours: do not let TFLite allocate its own textures and then copy results back to CPU for rendering. That round-trip kills your frame budget entirely.

val gpuOptions = GpuDelegateFactory.Options().apply {
    isPrecisionLossAllowed = true   // INT8 activations, ~2x throughput
    inferencePreference = GpuDelegateFactory.Options
        .INFERENCE_PREFERENCE_SUSTAINED_SPEED
}

For compositing, use a single 512×512 tiled texture atlas covering lip colors, blush gradients, and eyeshadow variants. One glBindTexture plus UV remap per frame beats multiple draw calls by a wide margin on mobile tile-based deferred renderers like Adreno and Mali.

Step 4: Pace Frames Against Vsync

Do not use a raw executor loop. Synchronize inference submission with vsync using Choreographer:

Choreographer.getInstance().postFrameCallback { frameTimeNanos ->
    if (latestFrame != null) submitInference(latestFrame!!)
    Choreographer.getInstance().postFrameCallback(this)
}

Combine this with an EGL sync fence after the render pass — eglCreateSyncKHR — so you never submit a new frame while the GPU is still compositing the previous one. This eliminates tearing artifacts when lipstick renders half-updated during a blink.

Gotchas

Frame queuing will destroy you. The docs do not emphasize this strongly enough, but STRATEGY_KEEP_ONLY_LATEST is the single most important line in this entire setup. Buffer frames under load even once and users will notice.

CPU readbacks are silent budget killers. Every hop from GPU→CPU→GPU costs you more than the inference itself on some devices. Keep compositing on-GPU end-to-end.

Profile blend shapes before shipping. Not all 52 coefficients are equally cheap to react to. Profile mouthSmile* and jawOpen deformations on your actual target Snapdragon tier before adding expression-driven layers. Blend shape processing on CPU is where frame budgets quietly die.

Iris landmarks are in the mesh. Landmarks 468–477 are iris contours. Useful to know both for eye effects and for avoiding accidental overlap with lid geometry.

Wrapping Up

The architecture here — ImageAnalysis with latest-only backpressure, GPU delegate with sustained-speed preference, a texture atlas, and Choreographer-paced submission — is the minimal setup to get this working reliably in production. Each piece earns its place. Remove any one of them and you will feel it in the profiler within minutes.

Relevant references: MediaPipe Face Landmarker docs, CameraX ImageAnalysis guide, TFLite GPU Delegate.