<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Firefly Team</title>
    <description>The latest articles on DEV Community by Firefly Team (@fireflyteamapps).</description>
    <link>https://dev.to/fireflyteamapps</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4223937%2F8a7310f6-2db2-4225-a13b-354ac3a04248.png</url>
      <title>DEV Community: Firefly Team</title>
      <link>https://dev.to/fireflyteamapps</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/fireflyteamapps"/>
    <language>en</language>
    <item>
      <title>Shooting against the light on Android: Camera2 bracketing and exposure fusion on the CPU</title>
      <dc:creator>Firefly Team</dc:creator>
      <pubDate>Mon, 16 Nov 2026 09:40:31 +0000</pubDate>
      <link>https://dev.to/fireflyteamapps/shooting-against-the-light-on-android-camera2-bracketing-and-exposure-fusion-on-the-cpu-2anc</link>
      <guid>https://dev.to/fireflyteamapps/shooting-against-the-light-on-android-camera2-bracketing-and-exposure-fusion-on-the-cpu-2anc</guid>
      <description>&lt;p&gt;We're Firefly Team, a small app studio in Vietnam. This is how we built a backlight mode for our Android camera: two frames at different&lt;br&gt;
exposures, aligning them, fusing them without halos — and what still doesn't work on a real phone. Everything runs on-device, on the CPU,&lt;br&gt;
in Kotlin.&lt;/p&gt;
&lt;h2&gt;
  
  
  The problem
&lt;/h2&gt;

&lt;p&gt;Someone stands in front of a window or a sunset. The phone sensor can't hold both: either the face is dark, or the window is a white blob.&lt;/p&gt;

&lt;p&gt;The obvious answer is the manufacturer's HDR. But it isn't everywhere: on our test OPPO CPH2203 there's no CameraX Extensions library at all&lt;br&gt;
(which is how you'd reach the vendor's HDR). So we shoot the series ourselves:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Take two frames: one exposed for the person, one for the background.&lt;/li&gt;
&lt;li&gt;Align them: the phone is handheld and there's a noticeable gap between frames.&lt;/li&gt;
&lt;li&gt;Fuse them so the person comes from the bright frame and the background from the dark one, with no halo at the boundary.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2&gt;
  
  
  Bracketing: Camera2 from inside CameraX
&lt;/h2&gt;

&lt;p&gt;We capture with &lt;code&gt;ImageCapture&lt;/code&gt; at full resolution (not video-stream frames) and set exposure manually through &lt;code&gt;Camera2CameraControl&lt;/code&gt;:&lt;br&gt;
auto-exposure off, then our own exposure time, ISO and frame duration.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight kotlin"&gt;&lt;code&gt;&lt;span class="c1"&gt;// Simplified. Requires @ExperimentalCamera2Interop.&lt;/span&gt;
&lt;span class="kd"&gt;val&lt;/span&gt; &lt;span class="n"&gt;options&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;CaptureRequestOptions&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;Builder&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setCaptureRequestOption&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;CaptureRequest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;CONTROL_AE_MODE&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nc"&gt;CaptureRequest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;CONTROL_AE_MODE_OFF&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setCaptureRequestOption&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;CaptureRequest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SENSOR_EXPOSURE_TIME&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;exposureNs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setCaptureRequestOption&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;CaptureRequest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SENSOR_SENSITIVITY&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;iso&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;// Set the frame duration ourselves: no shorter than 1/30 s&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setCaptureRequestOption&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;CaptureRequest&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="no"&gt;SENSOR_FRAME_DURATION&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;maxOf&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;exposureNs&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;33_333_333L&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nc"&gt;Camera2CameraControl&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;from&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;camera&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cameraControl&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;setCaptureRequestOptions&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;options&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The exposure ceiling for the bright frame is &lt;code&gt;min(SENSOR_INFO_EXPOSURE_TIME_RANGE.upper, 1/30 s)&lt;/code&gt;. If the camera lacks the&lt;br&gt;
&lt;code&gt;MANUAL_SENSOR&lt;/code&gt; capability, we fall back to exposure compensation (EV).&lt;/p&gt;

&lt;h3&gt;
  
  
  Which stops to shoot
&lt;/h3&gt;

&lt;p&gt;We started with symmetric ±1.7 EV, like our iOS version. On the front camera in a room with a window that broke: the +1.7 EV frame came out&lt;br&gt;
at ISO 2383–2681 with mean brightness 0.91–0.93, and the face clipped. The fused result looked worse than the preview.&lt;/p&gt;

&lt;p&gt;The reason is simple: the phone's metering already prioritizes the face. So the subject frame is &lt;strong&gt;0 EV&lt;/strong&gt; (exactly what the preview shows)&lt;br&gt;
and the background frame is &lt;strong&gt;−3.4 EV&lt;/strong&gt;. Same total spread as iOS (3.4 EV), just shifted down.&lt;/p&gt;

&lt;h3&gt;
  
  
  Camera shake
&lt;/h3&gt;

&lt;p&gt;The test phone has no optical stabilization, so we cap the subject frame at 1/60 s and make up the rest with ISO.&lt;/p&gt;

&lt;p&gt;The two frames are &lt;strong&gt;720–820 ms&lt;/strong&gt; apart, about 340 ms of which is waiting for the first exposure to settle. The hand moves in that time.&lt;br&gt;
To know how much, we integrate the gyroscope exactly between the two frames' &lt;code&gt;SENSOR_TIMESTAMP&lt;/code&gt;s and convert the angles into an expected&lt;br&gt;
pixel shift. Real handheld measurements: 2.0 / 5.0 / 6.4 px of shift and 1–5 px of rotation at the frame corners.&lt;/p&gt;

&lt;p&gt;Gotcha: the first build crashed on the shutter press. We asked the gyroscope for &lt;code&gt;SENSOR_DELAY_FASTEST&lt;/code&gt;, which on recent Android versions&lt;br&gt;
requires the &lt;code&gt;HIGH_SAMPLING_RATE_SENSORS&lt;/code&gt; permission — otherwise &lt;code&gt;SecurityException&lt;/code&gt;. Now we read at 200 Hz and guard the subscription.&lt;/p&gt;

&lt;h2&gt;
  
  
  Alignment: why "sum of gradients" lies
&lt;/h2&gt;

&lt;p&gt;Our first alignment maximized the sum of products of the two frames' "edge energies", &lt;code&gt;Σ E_ref · E_mov · w&lt;/code&gt;, over candidate shifts.&lt;br&gt;
Without subtracting the mean, this favors whichever shift drags an edge-rich area — a window frame, blinds — into the comparison window.&lt;br&gt;
On a tripod with a still person (true shift 0–2 px) the engine "found" 6.54 px, and sometimes refused to fuse at all with&lt;br&gt;
"frames are too far apart".&lt;/p&gt;

&lt;p&gt;The fix is &lt;strong&gt;masked normalized cross-correlation&lt;/strong&gt; (masked NCC): subtract the mean, normalize, weight each pixel by &lt;code&gt;w_ref(x) · w_mov(x − d)&lt;/code&gt;.&lt;br&gt;
At the coarse level (160 px on the long side) we exhaustively search the whole ±12% grid — cheaper than finding a seed.&lt;/p&gt;

&lt;p&gt;Two heuristics on top:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Tripod pairs&lt;/strong&gt; (shot back-to-back by our own camera): search ±4% and apply the shift only if it recovers ≥ 15% of the gap between the
peak and the unaligned score. Measured: still — 0.024 and 0.039; with 24 px of shake — 0.43.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Handheld&lt;/strong&gt;: if the gyroscope predicts ≥ 3 px, widen the search, but trust the result only if it's at most &lt;code&gt;max(8 px, 2 × prediction)&lt;/code&gt;.
This rejects false vertical peaks.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A separate trap was the confidence score. We kept the old formula &lt;code&gt;(peak − unaligned) / peak&lt;/code&gt;, but with mean-subtracted NCC the unaligned&lt;br&gt;
score is almost always near zero — so confidence read 0.84–1.00 even for two completely different scenes. A unit test ("two different&lt;br&gt;
scenes → low confidence") caught it. Confidence is now the NCC peak itself, clamped to 0…1: the same synthetic scene gives 0.995, different&lt;br&gt;
scenes 0.06–0.11. The thresholds are provisional until we calibrate them on real pairs, where the peak is noticeably lower.&lt;/p&gt;

&lt;h2&gt;
  
  
  Fusion: why not cut and paste
&lt;/h2&gt;

&lt;p&gt;The most direct approach is a person mask: take the person from the bright frame and everything else from the dark one. We tried it and got&lt;br&gt;
a white halo around the person. The segmentation mask is coarse, its soft edge runs up to ~200 px, and the bright frame's clipped background&lt;br&gt;
leaks into that band.&lt;/p&gt;

&lt;p&gt;We compared three approaches on real pairs and settled on &lt;strong&gt;multi-scale Mertens exposure fusion&lt;/strong&gt; with the mask as a prior. The mask doesn't&lt;br&gt;
cut anything — it only tilts the weights:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Region&lt;/th&gt;
&lt;th&gt;Bright-frame weight&lt;/th&gt;
&lt;th&gt;Dark-frame weight&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Person&lt;/td&gt;
&lt;td&gt;×2.0&lt;/td&gt;
&lt;td&gt;×0.2&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Background where the bright frame is clipped&lt;/td&gt;
&lt;td&gt;×0.2&lt;/td&gt;
&lt;td&gt;×1.6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Background where the bright frame still has detail&lt;/td&gt;
&lt;td&gt;×1.6&lt;/td&gt;
&lt;td&gt;×0.4&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The mask is blurred heavily first (σ = 3% of the frame size), so transitions are smooth and there's no halo. The "subject brightness"&lt;br&gt;
slider in the UI is a gamma on the already-fused image, blended through the same blurred mask.&lt;/p&gt;

&lt;p&gt;The mask comes from ML Kit: Selfie Segmentation in single-image mode, falling back to Subject Segmentation (any foreground object) when&lt;br&gt;
there's no person.&lt;/p&gt;

&lt;p&gt;That table isn't the first version. Initially the background weights were 0.2 : 1.6 everywhere, and backgrounds that still had detail came&lt;br&gt;
out darker than they should. Once we factored in the clipping map, those areas got back about half of their brightness&lt;br&gt;
(measured 0.34 → 0.44, against an original 0.49).&lt;/p&gt;

&lt;h3&gt;
  
  
  Ghosts
&lt;/h3&gt;

&lt;p&gt;If the two frames were taken with a pause — say, the user picked two gallery photos taken 6 seconds apart — the person moves, and a sleeve&lt;br&gt;
lands semi-transparently on the sky.&lt;/p&gt;

&lt;p&gt;Before fusing, we bring the bright frame down to the dark frame's brightness and look for disagreement. Crucially, we use a &lt;strong&gt;curve&lt;/strong&gt;, not a&lt;br&gt;
single factor: the median over 24 brightness bins on the background, forced monotonic. The two photos went through an automatic ISP with&lt;br&gt;
different tone curves, so one multiplier doesn't work. Where they disagree by more than ~0.6 stop, we use the bright frame only.&lt;/p&gt;

&lt;p&gt;The ghost map is built on a ≤ 768 px copy and upscaled. At full resolution this step added 2.7 s to saving; now it's about 0.6 s.&lt;/p&gt;

&lt;h3&gt;
  
  
  Median of ratios, not ratio of means
&lt;/h3&gt;

&lt;p&gt;One more stumble. For patches (where the dark frame has nothing usable) we need a brightness factor between the frames. Computed as a ratio&lt;br&gt;
of means, clipped areas — pinned at 1.0 — drag the estimate up: one pair gave ×0.56 instead of the real ×0.31. A dark curtain then got&lt;br&gt;
classified as a "hole" and patched with a white blob. The median of per-pixel ratios over unclipped, non-flat background gets it right.&lt;/p&gt;

&lt;h2&gt;
  
  
  What still doesn't work
&lt;/h2&gt;

&lt;p&gt;Honestly, the main open problem: on the front camera the −3.4 EV dark frame has almost no clipping — 0.2–0.7% of the background versus&lt;br&gt;
7–21% before. Yet in the result the window stays white over 42–61% of the area that was clipped. In those pixels the dark frame reads&lt;br&gt;
0.87–0.97 in its brightest channel.&lt;/p&gt;

&lt;p&gt;The front camera's ISP seems to compress highlights very hard: the midtone difference measured on the JPEGs was about 6.4 EV, although we&lt;br&gt;
asked for 3.4. We tried five fusion-weight variants; the share of recovered highlights only moved from 0.35 to 0.42. So it isn't the&lt;br&gt;
weights — the information simply isn't in the dark frame any more.&lt;/p&gt;

&lt;p&gt;Next we'll try:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;dropping the background frame to around −5 EV;&lt;/li&gt;
&lt;li&gt;locking the ISP tone curve for the bracket pair (&lt;code&gt;TONEMAP_MODE_CONTRAST_CURVE&lt;/code&gt;; the test phone has &lt;code&gt;MANUAL_POST_PROCESSING&lt;/code&gt;).&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;We haven't tested the main camera thoroughly on this scenario yet. If you've fought front-camera ISPs on the same problem, I'd love to&lt;br&gt;
hear what worked.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the CPU, not shaders
&lt;/h2&gt;

&lt;p&gt;We planned GLSL but stayed with plain Kotlin:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;the input is capped at 2048 px, all measurements run on 160…1024 px copies, and full resolution only needs a few per-pixel passes plus
an O(n) blur;&lt;/li&gt;
&lt;li&gt;most importantly, &lt;strong&gt;it's verifiable with numbers in plain JVM tests&lt;/strong&gt;: a mask covering 18% of the frame must read as 0.18.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;On the OPPO CPH2203, fusion at preview size (1200 px) takes ≈ 0.8 s and is cached per size, so moving the slider doesn't re-fuse. Full&lt;br&gt;
resolution on save takes ≈ 2.2 s. The Kotlin implementation matches our Python prototype within a mean difference of 2/255.&lt;/p&gt;

&lt;p&gt;For debugging, the debug build drops frame pairs with metadata (exposure, ISO, gyroscope data) into the app cache, and an instrumented test&lt;br&gt;
replays the engine on them. That lets us change the algorithm and compare results on the same real shots without going outside every time.&lt;/p&gt;

&lt;h2&gt;
  
  
  Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;If the vendor HDR isn't available, bracketing from CameraX via &lt;code&gt;Camera2CameraControl&lt;/code&gt; works — but build the stops from the phone's own
metering (0 EV for the person), not symmetrically.&lt;/li&gt;
&lt;li&gt;For masked alignment, use mean-subtracted NCC, not "sum of edges"; and derive confidence from the peak, not from a difference.&lt;/li&gt;
&lt;li&gt;A segmentation mask is a bad knife but a good weight for Mertens fusion.&lt;/li&gt;
&lt;li&gt;Highlights the ISP already "ate" in the dark frame can't be brought back by fusion.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The mode ships in our camera app Postcam for Android and iOS (which also puts postage-stamp and postcard frames right in the viewfinder):&lt;br&gt;
&lt;a href="https://play.google.com/store/apps/details?id=com.postcam.frame.photo.editor.retro.camera&amp;amp;referrer=utm_source%3Ddevto%26utm_medium%3Dforum%26utm_campaign%3Dpostcam_launch" rel="noopener noreferrer"&gt;Google Play&lt;/a&gt;.&lt;br&gt;
Questions and criticism welcome in the comments.&lt;/p&gt;

</description>
      <category>android</category>
      <category>kotlin</category>
      <category>mobile</category>
      <category>computervision</category>
    </item>
  </channel>
</rss>
