We're Firefly Team, a small app studio in Vietnam. This is how we built a backlight mode for our Android camera: two frames at different
exposures, aligning them, fusing them without halos — and what still doesn't work on a real phone. Everything runs on-device, on the CPU,
in Kotlin.
The problem
Someone stands in front of a window or a sunset. The phone sensor can't hold both: either the face is dark, or the window is a white blob.
The obvious answer is the manufacturer's HDR. But it isn't everywhere: on our test OPPO CPH2203 there's no CameraX Extensions library at all
(which is how you'd reach the vendor's HDR). So we shoot the series ourselves:
- Take two frames: one exposed for the person, one for the background.
- Align them: the phone is handheld and there's a noticeable gap between frames.
- Fuse them so the person comes from the bright frame and the background from the dark one, with no halo at the boundary.
Bracketing: Camera2 from inside CameraX
We capture with ImageCapture at full resolution (not video-stream frames) and set exposure manually through Camera2CameraControl:
auto-exposure off, then our own exposure time, ISO and frame duration.
// Simplified. Requires @ExperimentalCamera2Interop.
val options = CaptureRequestOptions.Builder()
.setCaptureRequestOption(CaptureRequest.CONTROL_AE_MODE, CaptureRequest.CONTROL_AE_MODE_OFF)
.setCaptureRequestOption(CaptureRequest.SENSOR_EXPOSURE_TIME, exposureNs)
.setCaptureRequestOption(CaptureRequest.SENSOR_SENSITIVITY, iso)
// Set the frame duration ourselves: no shorter than 1/30 s
.setCaptureRequestOption(CaptureRequest.SENSOR_FRAME_DURATION, maxOf(exposureNs, 33_333_333L))
.build()
Camera2CameraControl.from(camera.cameraControl).setCaptureRequestOptions(options)
The exposure ceiling for the bright frame is min(SENSOR_INFO_EXPOSURE_TIME_RANGE.upper, 1/30 s). If the camera lacks the
MANUAL_SENSOR capability, we fall back to exposure compensation (EV).
Which stops to shoot
We started with symmetric ±1.7 EV, like our iOS version. On the front camera in a room with a window that broke: the +1.7 EV frame came out
at ISO 2383–2681 with mean brightness 0.91–0.93, and the face clipped. The fused result looked worse than the preview.
The reason is simple: the phone's metering already prioritizes the face. So the subject frame is 0 EV (exactly what the preview shows)
and the background frame is −3.4 EV. Same total spread as iOS (3.4 EV), just shifted down.
Camera shake
The test phone has no optical stabilization, so we cap the subject frame at 1/60 s and make up the rest with ISO.
The two frames are 720–820 ms apart, about 340 ms of which is waiting for the first exposure to settle. The hand moves in that time.
To know how much, we integrate the gyroscope exactly between the two frames' SENSOR_TIMESTAMPs and convert the angles into an expected
pixel shift. Real handheld measurements: 2.0 / 5.0 / 6.4 px of shift and 1–5 px of rotation at the frame corners.
Gotcha: the first build crashed on the shutter press. We asked the gyroscope for SENSOR_DELAY_FASTEST, which on recent Android versions
requires the HIGH_SAMPLING_RATE_SENSORS permission — otherwise SecurityException. Now we read at 200 Hz and guard the subscription.
Alignment: why "sum of gradients" lies
Our first alignment maximized the sum of products of the two frames' "edge energies", Σ E_ref · E_mov · w, over candidate shifts.
Without subtracting the mean, this favors whichever shift drags an edge-rich area — a window frame, blinds — into the comparison window.
On a tripod with a still person (true shift 0–2 px) the engine "found" 6.54 px, and sometimes refused to fuse at all with
"frames are too far apart".
The fix is masked normalized cross-correlation (masked NCC): subtract the mean, normalize, weight each pixel by w_ref(x) · w_mov(x − d).
At the coarse level (160 px on the long side) we exhaustively search the whole ±12% grid — cheaper than finding a seed.
Two heuristics on top:
- Tripod pairs (shot back-to-back by our own camera): search ±4% and apply the shift only if it recovers ≥ 15% of the gap between the peak and the unaligned score. Measured: still — 0.024 and 0.039; with 24 px of shake — 0.43.
-
Handheld: if the gyroscope predicts ≥ 3 px, widen the search, but trust the result only if it's at most
max(8 px, 2 × prediction). This rejects false vertical peaks.
A separate trap was the confidence score. We kept the old formula (peak − unaligned) / peak, but with mean-subtracted NCC the unaligned
score is almost always near zero — so confidence read 0.84–1.00 even for two completely different scenes. A unit test ("two different
scenes → low confidence") caught it. Confidence is now the NCC peak itself, clamped to 0…1: the same synthetic scene gives 0.995, different
scenes 0.06–0.11. The thresholds are provisional until we calibrate them on real pairs, where the peak is noticeably lower.
Fusion: why not cut and paste
The most direct approach is a person mask: take the person from the bright frame and everything else from the dark one. We tried it and got
a white halo around the person. The segmentation mask is coarse, its soft edge runs up to ~200 px, and the bright frame's clipped background
leaks into that band.
We compared three approaches on real pairs and settled on multi-scale Mertens exposure fusion with the mask as a prior. The mask doesn't
cut anything — it only tilts the weights:
| Region | Bright-frame weight | Dark-frame weight |
|---|---|---|
| Person | ×2.0 | ×0.2 |
| Background where the bright frame is clipped | ×0.2 | ×1.6 |
| Background where the bright frame still has detail | ×1.6 | ×0.4 |
The mask is blurred heavily first (σ = 3% of the frame size), so transitions are smooth and there's no halo. The "subject brightness"
slider in the UI is a gamma on the already-fused image, blended through the same blurred mask.
The mask comes from ML Kit: Selfie Segmentation in single-image mode, falling back to Subject Segmentation (any foreground object) when
there's no person.
That table isn't the first version. Initially the background weights were 0.2 : 1.6 everywhere, and backgrounds that still had detail came
out darker than they should. Once we factored in the clipping map, those areas got back about half of their brightness
(measured 0.34 → 0.44, against an original 0.49).
Ghosts
If the two frames were taken with a pause — say, the user picked two gallery photos taken 6 seconds apart — the person moves, and a sleeve
lands semi-transparently on the sky.
Before fusing, we bring the bright frame down to the dark frame's brightness and look for disagreement. Crucially, we use a curve, not a
single factor: the median over 24 brightness bins on the background, forced monotonic. The two photos went through an automatic ISP with
different tone curves, so one multiplier doesn't work. Where they disagree by more than ~0.6 stop, we use the bright frame only.
The ghost map is built on a ≤ 768 px copy and upscaled. At full resolution this step added 2.7 s to saving; now it's about 0.6 s.
Median of ratios, not ratio of means
One more stumble. For patches (where the dark frame has nothing usable) we need a brightness factor between the frames. Computed as a ratio
of means, clipped areas — pinned at 1.0 — drag the estimate up: one pair gave ×0.56 instead of the real ×0.31. A dark curtain then got
classified as a "hole" and patched with a white blob. The median of per-pixel ratios over unclipped, non-flat background gets it right.
What still doesn't work
Honestly, the main open problem: on the front camera the −3.4 EV dark frame has almost no clipping — 0.2–0.7% of the background versus
7–21% before. Yet in the result the window stays white over 42–61% of the area that was clipped. In those pixels the dark frame reads
0.87–0.97 in its brightest channel.
The front camera's ISP seems to compress highlights very hard: the midtone difference measured on the JPEGs was about 6.4 EV, although we
asked for 3.4. We tried five fusion-weight variants; the share of recovered highlights only moved from 0.35 to 0.42. So it isn't the
weights — the information simply isn't in the dark frame any more.
Next we'll try:
- dropping the background frame to around −5 EV;
- locking the ISP tone curve for the bracket pair (
TONEMAP_MODE_CONTRAST_CURVE; the test phone hasMANUAL_POST_PROCESSING).
We haven't tested the main camera thoroughly on this scenario yet. If you've fought front-camera ISPs on the same problem, I'd love to
hear what worked.
Why the CPU, not shaders
We planned GLSL but stayed with plain Kotlin:
- the input is capped at 2048 px, all measurements run on 160…1024 px copies, and full resolution only needs a few per-pixel passes plus an O(n) blur;
- most importantly, it's verifiable with numbers in plain JVM tests: a mask covering 18% of the frame must read as 0.18.
On the OPPO CPH2203, fusion at preview size (1200 px) takes ≈ 0.8 s and is cached per size, so moving the slider doesn't re-fuse. Full
resolution on save takes ≈ 2.2 s. The Kotlin implementation matches our Python prototype within a mean difference of 2/255.
For debugging, the debug build drops frame pairs with metadata (exposure, ISO, gyroscope data) into the app cache, and an instrumented test
replays the engine on them. That lets us change the algorithm and compare results on the same real shots without going outside every time.
Takeaways
- If the vendor HDR isn't available, bracketing from CameraX via
Camera2CameraControlworks — but build the stops from the phone's own metering (0 EV for the person), not symmetrically. - For masked alignment, use mean-subtracted NCC, not "sum of edges"; and derive confidence from the peak, not from a difference.
- A segmentation mask is a bad knife but a good weight for Mertens fusion.
- Highlights the ISP already "ate" in the dark frame can't be brought back by fusion.
The mode ships in our camera app Postcam for Android and iOS (which also puts postage-stamp and postcard frames right in the viewfinder):
Google Play.
Questions and criticism welcome in the comments.
Top comments (0)