I build an Android camera that tells you which way to move the phone. You tap a detail, the app follows it, and a ring turns green when that detail sits on a composition guide and the phone is ready. This post covers the parts that were harder than they looked: following a tapped patch of pixels, bridging fast turns with the gyroscope, and refusing to let any estimate produce the green state.
Spiralens screenshot by the author, taken on the Android emulator's virtual scene (a rendered room, not a real photo).
Why not object detection
The user decides what the subject is. A mug, a dog's eye, a doorway: a detector would need a class for each. So the tracker follows whatever patch was tapped. The only model in the app is on-device face landmarks, used in people scenes to pick the eye nearest the camera as a starting point. The user can override or clear it.
The patch tracker
The tracker lives in the shared Kotlin module, with no platform imports, so it runs in plain JVM tests. It works on small grey frames, matches a template of the tapped patch by normalised cross-correlation, and searches near the last position.
A few rules came from failure:
- Reject plain patches. If the tapped spot has too little variance, the tracker tries up to six textured, distinct spots within 10% of the shorter frame side, nearest first. The ring keeps the original tap offset, so it does not visibly jump to the new spot.
- Ambiguity is a failure. Windows, stripes and hair match themselves. A match counts only if its best peak clearly beats its rivals.
- Recover, do not drift. After a miss, the search widens, and recovery needs two consecutive strong, unambiguous matches against the unchanged template.
- Bounded work. Analysis starts at most one frame every 80 ms on a serial executor that keeps only the latest buffer, and every buffer is closed, skipped or not.
Gyro bridging that never grants green
Fast turns blur the image, so matching fails exactly when the user moves most. The fix is to predict where the patch went from the phone's rotation.
RotationHistory integrates timestamped gyroscope samples into quaternions in a ring buffer. A gap longer than 0.2 s clears it, because a rotation across a gap is a guess. between(fromNs, toNs) interpolates both ends and returns the relative rotation, which is remapped to display axes and projected through a pinhole model: the tapped ray is rotated, then projected back to the image.
The tracker uses it in three ways:
- With rotation data it starts the search where the turn moved the patch.
- A frame turning faster than 1.2 rad/s is not matched at all. The ring follows the estimate, and the result is marked as moving, not tracked.
- When the image cannot confirm the patch, the gyro estimate is kept for up to about 12 to 15 seconds of frames if rotation data is unbroken, and about 1.5 to 2 seconds without it.
The rule that matters: an estimate is shown as an estimate, with direction guidance, and it never counts as a measurement. Green, auto capture and feedback read only the measured point. Offscreen estimates are hidden, not clamped to an edge, and display estimates expire after 400 ms.
A gyroscope cannot see translation, so the tracker learns a small per-frame drift from confirmed matches, capped at 8 px, and drops it when rotation data has a gap.
Freshness rules for camera, focus and sensor data
Most of the bugs I fixed were about stale data looking current. These are the rules in code:
| Data | Rule |
|---|---|
| Tracked frame | Must be at most 599 ms old to count as fresh |
| Level sensor sample | Dropped if older than 749 ms, or stamped more than 250 ms ahead of the clock |
| Camera timestamp | Trusted only if on the realtime clock and not more than 50 ms ahead; otherwise freshness uses arrival time, and gyro alignment uses arrival time minus an estimated 40 ms delivery delay |
| Tap snapshot | Template taken from a frame within 200 ms of the tap, or up to 1 s if the gyro shows the phone was still |
| Autofocus | Green needs focused or unsupported; a failed focus is retried once after 300 ms |
| Capture gate | One second of continuous readiness; a gap over 600 ms resets it |
The green condition, as written:
val fresh = tracking && frameTime > 0 && now - frameTime in 0..599
...
val captureReady: Boolean get() = guidance?.ready == true && cameraReady && !busy && !cameraAdjusting &&
focusState in setOf(FocusState.FOCUSED, FocusState.UNSUPPORTED)
A processing delay must not make an old image fresh, so the acquisition timestamp survives the whole pipeline. And the shutter is never blocked. Green is a hint, not a gate.
One coordinate system
Preview, analysis, photo and overlay share one normalised space, x right and y down in 0..1, and one CameraX viewport at 9:16. The spiral that is drawn is the same geometry object the engine reads its targets from, so the picture cannot disagree with the judgement.
Details that cost time:
- Position is judged in frame-width units, so the target zone is round on screen: enter within 0.07, leave at 0.10. Roll uses 2.5 degrees in and 3.5 out. The hysteresis stops hand tremor flickering the ring.
- Gravity and gyro vectors map to display axes by Surface rotation. At 90 degrees, (x, y, z) becomes (-y, x, z). A test compares that mapping with Android's own
SensorManager.remapCoordinateSystem, not with my own expectation. - The arrow shows the phone's move, which is opposite to the move the subject needs in the image.
Performance, with the number I do not like
My performance notes are dated 19 September, on an emulator (API 36, two virtual CPUs, 1536 MB RAM) running a shrunk QA build. Emulator numbers are not hardware benchmarks, and I say so in the notes.
- Cold activity start: 373 ms. That is the activity launch, not the first usable preview.
- Camera service output: 2,047 frames in 68.26 s, about 29.99 frames per second. That is a sensor throughput proxy, not displayed FPS.
- UI rendering over about 20 seconds: 265 frames, p50 17 ms, p95 27 ms, p99 32 ms.
- Memory: 41.8 MB PSS in one snapshot.
And the one that I have not explained: in the same UI sample, 229 of 265 frames missed the deadline by Android's deadline metric (86.42%). The older jank metric counted 22 of 265 (8.30%). After I cached the spiral path and targets, a later sample gave 241 of 256 (94.14%) deadline misses, with lower percentile times. A frame-timeline trace showed mostly early-present prediction errors, which cannot explain the count. I have kept it as an open investigation, not a pass. I did not claim a speed-up from reading the source either.
What I can say: in that trace the main thread ran about 397 ms and the render thread about 454 ms over roughly 20 seconds, and no single CPU or GPU cause is proven. What I cannot say is how this behaves on a real phone under heat for 15 minutes. That acceptance run is still open.
Testing
- Shared JVM tests cover geometry, the capture gate, rotation history and tracking stress.
- A scene replay test runs recorded tracking scenes (about 160 px grey frames, measured rotation, the tapped point) through the current tracker and fails if it tracks less than the phone did.
- A device run on 9 October, on an emulator with a virtual camera scene, finished 177 tests: 172 passed, 0 failed, 5 skipped.
If you build something like this, the general lesson I would offer is narrow: decide which data is allowed to say "yes", and let everything else only say "maybe".
Disclosure: I'm the developer of Spiralens. An AI assistant wrote this post from the project's code and notes.
Top comments (0)