DEV Community: Mehmet Apaydın

Tracking a Tapped Pixel Patch with a Gyroscope That Is Never Allowed to Say "Green"

Mehmet Apaydın — Tue, 13 Oct 2026 08:23:31 +0000

I build an Android camera that tells you which way to move the phone. You tap a detail, the app follows it, and a ring turns green when that detail sits on a composition guide and the phone is ready. This post covers the parts that were harder than they looked: following a tapped patch of pixels, bridging fast turns with the gyroscope, and refusing to let any estimate produce the green state.

Spiralens screenshot by the author, taken on the Android emulator's virtual scene (a rendered room, not a real photo).

Why not object detection

The user decides what the subject is. A mug, a dog's eye, a doorway: a detector would need a class for each. So the tracker follows whatever patch was tapped. The only model in the app is on-device face landmarks, used in people scenes to pick the eye nearest the camera as a starting point. The user can override or clear it.

The patch tracker

The tracker lives in the shared Kotlin module, with no platform imports, so it runs in plain JVM tests. It works on small grey frames, matches a template of the tapped patch by normalised cross-correlation, and searches near the last position.

A few rules came from failure:

Reject plain patches. If the tapped spot has too little variance, the tracker tries up to six textured, distinct spots within 10% of the shorter frame side, nearest first. The ring keeps the original tap offset, so it does not visibly jump to the new spot.
Ambiguity is a failure. Windows, stripes and hair match themselves. A match counts only if its best peak clearly beats its rivals.
Recover, do not drift. After a miss, the search widens, and recovery needs two consecutive strong, unambiguous matches against the unchanged template.
Bounded work. Analysis starts at most one frame every 80 ms on a serial executor that keeps only the latest buffer, and every buffer is closed, skipped or not.

Gyro bridging that never grants green

Fast turns blur the image, so matching fails exactly when the user moves most. The fix is to predict where the patch went from the phone's rotation.

RotationHistory integrates timestamped gyroscope samples into quaternions in a ring buffer. A gap longer than 0.2 s clears it, because a rotation across a gap is a guess. between(fromNs, toNs) interpolates both ends and returns the relative rotation, which is remapped to display axes and projected through a pinhole model: the tapped ray is rotated, then projected back to the image.

The tracker uses it in three ways:

With rotation data it starts the search where the turn moved the patch.
A frame turning faster than 1.2 rad/s is not matched at all. The ring follows the estimate, and the result is marked as moving, not tracked.
When the image cannot confirm the patch, the gyro estimate is kept for up to about 12 to 15 seconds of frames if rotation data is unbroken, and about 1.5 to 2 seconds without it.

The rule that matters: an estimate is shown as an estimate, with direction guidance, and it never counts as a measurement. Green, auto capture and feedback read only the measured point. Offscreen estimates are hidden, not clamped to an edge, and display estimates expire after 400 ms.

A gyroscope cannot see translation, so the tracker learns a small per-frame drift from confirmed matches, capped at 8 px, and drops it when rotation data has a gap.

Freshness rules for camera, focus and sensor data

Most of the bugs I fixed were about stale data looking current. These are the rules in code:

Data	Rule
Tracked frame	Must be at most 599 ms old to count as fresh
Level sensor sample	Dropped if older than 749 ms, or stamped more than 250 ms ahead of the clock
Camera timestamp	Trusted only if on the realtime clock and not more than 50 ms ahead; otherwise freshness uses arrival time, and gyro alignment uses arrival time minus an estimated 40 ms delivery delay
Tap snapshot	Template taken from a frame within 200 ms of the tap, or up to 1 s if the gyro shows the phone was still
Autofocus	Green needs focused or unsupported; a failed focus is retried once after 300 ms
Capture gate	One second of continuous readiness; a gap over 600 ms resets it

The green condition, as written:

val fresh = tracking && frameTime > 0 && now - frameTime in 0..599
...
val captureReady: Boolean get() = guidance?.ready == true && cameraReady && !busy && !cameraAdjusting &&
    focusState in setOf(FocusState.FOCUSED, FocusState.UNSUPPORTED)

A processing delay must not make an old image fresh, so the acquisition timestamp survives the whole pipeline. And the shutter is never blocked. Green is a hint, not a gate.

One coordinate system

Preview, analysis, photo and overlay share one normalised space, x right and y down in 0..1, and one CameraX viewport at 9:16. The spiral that is drawn is the same geometry object the engine reads its targets from, so the picture cannot disagree with the judgement.

Details that cost time:

Position is judged in frame-width units, so the target zone is round on screen: enter within 0.07, leave at 0.10. Roll uses 2.5 degrees in and 3.5 out. The hysteresis stops hand tremor flickering the ring.
Gravity and gyro vectors map to display axes by Surface rotation. At 90 degrees, (x, y, z) becomes (-y, x, z). A test compares that mapping with Android's own SensorManager.remapCoordinateSystem, not with my own expectation.
The arrow shows the phone's move, which is opposite to the move the subject needs in the image.

Performance, with the number I do not like

My performance notes are dated 19 September, on an emulator (API 36, two virtual CPUs, 1536 MB RAM) running a shrunk QA build. Emulator numbers are not hardware benchmarks, and I say so in the notes.

Cold activity start: 373 ms. That is the activity launch, not the first usable preview.
Camera service output: 2,047 frames in 68.26 s, about 29.99 frames per second. That is a sensor throughput proxy, not displayed FPS.
UI rendering over about 20 seconds: 265 frames, p50 17 ms, p95 27 ms, p99 32 ms.
Memory: 41.8 MB PSS in one snapshot.

And the one that I have not explained: in the same UI sample, 229 of 265 frames missed the deadline by Android's deadline metric (86.42%). The older jank metric counted 22 of 265 (8.30%). After I cached the spiral path and targets, a later sample gave 241 of 256 (94.14%) deadline misses, with lower percentile times. A frame-timeline trace showed mostly early-present prediction errors, which cannot explain the count. I have kept it as an open investigation, not a pass. I did not claim a speed-up from reading the source either.

What I can say: in that trace the main thread ran about 397 ms and the render thread about 454 ms over roughly 20 seconds, and no single CPU or GPU cause is proven. What I cannot say is how this behaves on a real phone under heat for 15 minutes. That acceptance run is still open.

Testing

Shared JVM tests cover geometry, the capture gate, rotation history and tracking stress.
A scene replay test runs recorded tracking scenes (about 160 px grey frames, measured rotation, the tapped point) through the current tracker and fails if it tracks less than the phone did.
A device run on 9 October, on an emulator with a virtual camera scene, finished 177 tests: 172 passed, 0 failed, 5 skipped.

If you build something like this, the general lesson I would offer is narrow: decide which data is allowed to say "yes", and let everything else only say "maybe".

Disclosure: I'm the developer of Spiralens. An AI assistant wrote this post from the project's code and notes.

What broke when I vectorized real-world images in the browser (and how I fixed it)

Mehmet Apaydın — Mon, 12 Oct 2026 16:12:08 +0000

I build Sticraft, a set of free browser tools. The one I spend most time on is an image vectorizer that turns a JPG or PNG into SVG, DXF or G-code for laser cutters, Cricut machines and pen plotters. Everything runs on the user's device. Nothing is uploaded.

Clean test logos traced fine from day one. Real files from real people did not. This post is a list of the failures I hit and what fixed each one, in case you are building something similar.

The engine

Colour and silhouette tracing use vtracer (visioncortex, 0.6.5) compiled to WebAssembly. The binary is about 143 KB. I self-host it and fetch it on first use, so the page does not pay for it until someone actually traces something. vtracer does proper curve fitting with corner detection, which is much better than midpoint corner cutting.

vtracer has no skeletonization, so the centerline mode (single-line paths for engraving and plotting, not outlines) runs on a separate tracer.

The tracing runs in a Web Worker so the page stays responsive on big images.

Failure 1: notches on every small circle

A 256 px logo traced into a circle with eight small notches, one per octant. They showed up in SVG, DXF and in the laser preview.

Cause: a hard per-pixel threshold turns a small anti-aliased circle into a staircase. The curve fitter reads the points where the stair pattern changes direction as corners.

Fix: do not threshold the native pixels. I build an "on-ness" field, blur it once with a 3x3 binomial kernel to kill JPEG ringing, upsample it (4x up to 400 px, 2x up to 800 px, native above that), threshold the upsampled field, trace at the larger size and scale the paths back. The edge now lands where the anti-aliased source actually crosses the threshold, so the staircase never exists. Cost stays under roughly 2.6 megapixels.

Failure 2: cut files nobody can weed

vtracer's speckle filter drops small black islands. A small white hole inside a shape is part of that shape's outline, though, so holes survive. One traced etching came out with around 1,600 holes under 100 square pixels. None of them can be cut or weeded on a vinyl cutter.

Fix: a connected-region pass on the mask before tracing. Black islands under minArea become background. Enclosed holes are filled only under a quarter of that, because the inside of a small "a" in 15 mm lettering is a hole too and has to stay open. Background that touches the image edge is the outside and is never touched.

On a wolf engraving this took the Cricut cut from 3,130 pieces to 366, and the letter test (the dot on an i, the counter of an a) still came through unchanged.

Failure 3: the background is not one colour

My first version assumed the background was whatever colour the corners were. Real inputs break that constantly:

a printed logo photographed on a desk, with the phone's shadow across half of it
old paper that yellows towards one corner
clip art downloaded with the grey and white "transparency" checkerboard baked into the pixels

Each of these got traced as artwork. A Cricut file full of black blobs, or a laser burning a grid into wood.

Fix: a per-pixel background model. I split the image into a coarse grid of cell medians and walk inward from the border, following gradual colour change (paper tone, light falloff) but stopping at a sharp step, which is where ink starts. Cells the walk never reaches are the artwork; their background is interpolated from the surrounding paper. Then every pixel's local background is mapped to one flat colour, and everything downstream sees a clean card. Photos with no dominant border background are left alone.

How I measured it

I collected 20 deliberately bad inputs from CC0 sources: five phone photos of prints taken at an angle in shadow, five yellowed and stained scans, five images with a baked-in checkerboard, five tiny blurry 120 px files. Before these changes 8 of 20 produced a usable cut or engrave file. After, 17 of 20. The three that still fail are a decorative letter that runs to the paper edge on stained paper, and two 120 px images where the detail is simply gone.

I also re-ran 24 ordinary real images after every change to make sure photos were not being flattened by mistake.

What I would tell someone starting this

Test on ugly files from day one. Clean logos hide almost every real problem.
Fix things before the tracer, not after. Cleaning the mask is simple; cleaning curves is not.
Keep one coordinate space for every export (SVG, DXF, EPS, PDF, G-code). Supersampling only works because the paths are scaled back before anything else sees them.

If you have built a vectorizer and found inputs that break it, I'd like to hear about them.

Disclosure: an AI agent wrote this post for me from my project's code and test notes. The numbers come from the project's own measurements.