I spent a week fixing a stitching bug in an Android app, and the single most expensive thing was not the algorithm. It was that my offline test harness was quietly not exercising the code I was changing.
Here is the shape of it, because I doubt it is rare.
The setup
The app records a scroll and joins the frames into one tall image. To iterate on that without re-shooting captures by hand, debug builds write a journal of every frame, and a test replays the journal through the same engine offline. Same input, same code, repeatable.
That harness is the whole basis for every claim I make about the engine.
Three ways it lied
1. It never called the guard it was grading.
The engine has a pass that rewrites a video area from one frame so it does not come out as a collage of moments. The call existed in one test function and not in the one I was using. So every measurement of video tearing I had taken was of the engine with that guard switched off.
That is not a small discrepancy. It meant a change to that guard could not move the number in either direction, which is exactly what happened: I made a fix, measured no effect, and concluded the fix was wrong. It was not tested at all.
2. One missing callback made recovery impossible.
Production wires three callbacks into the engine. The replay wired two. The missing one reports how far down the page the canvas currently reaches - and a guard deep in the relock path reads it. With it null, that guard can never be true, so a chain that loses its place in the replay can never recover.
Same 908-frame journal:
before after real device
output rows 2464 13305 13720
writes 7 99 -
One line. The harness had been reproducing a crippled engine and I had been reading its output as production behaviour.
3. I pasted a call one nesting level too deep.
After fixing the above I added the guard call by line number and it landed inside if (repairs <= 40) { ... }, itself inside another conditional. It ran only when an unrelated repair had just succeeded - which never happens on the journal I was testing. The counter read zero, and I published the conclusion 'this guard never fires'. It fires four times.
Why none of these looked like failures
A broken measurement does not raise an error. It returns a number, and the number is usually plausible.
Worse, the failure mode is biased. An empty output reads as 'nothing found'. A zero counter reads as 'no effect'. A skipped task reads as 'no change since last time'. All three are indistinguishable from the most common real result, which is that your change did nothing.
In one week I hit six of these. A grep pattern that missed indented output. A build task skipped as up-to-date, so I compared against a run that never happened. A template calibrated on one recording and applied to another, matching 77% of the page and reporting no difference. A -D flag the build never forwarded to the test JVM. Launching the wrong package on the phone, because the debug build installs alongside the store build under a different application id. And the nesting mistake above.
Three of them produced a diagnosis I acted on and had to undo.
What I do now
Prove the instrument can move before you trust it. Feed it a case where it MUST report non-zero. If it cannot produce a positive, its negative is worthless.
This is what finally caught the harness problem. My tearing metric said all three recordings were clean. Instead of accepting it I checked whether it could report a tear at all - and the check failed, which is how I found the guard was never being called.
Check the wiring matches production, not just that the test passes. A green suite proves the code compiles and the assertions hold. It says nothing about whether the code path you care about was reached.
An empty output is a question, not an answer. Every time this bit me, the tell was output that was emptier than it should have been - and I read it as a result instead of asking whether the tool had run.
The other half
The same week taught me a related one: an optimisation measured only where the system already works tells you nothing.
I sampled every second pixel in a hot loop. On the reference recording the worst frame halved and detection was unaffected, so I shipped it. Re-run on the recording where detection was already weak, it fired twice as many repairs while landing fewer rows, and the worst frame nearly tripled. The gain was specific to one recording; the cost was general.
Reverted, with the numbers written into the source above the function so the next person to have that idea finds the measurement instead of repeating the week.
The app is LongShot - free, no ads, and no INTERNET permission, which is checkable rather than promised: it is stripped from the merged manifest, so the process cannot open a socket.
Known limitation, stated in the app itself: pause any playing video before capturing. Moving content is read a few rows at a time as you scroll past it, so it can come out stitched from several moments. That one is measured and located, and not yet fixed.
Top comments (0)