Why scrolling screenshots fall apart on video feeds

Codnet Software Company — Mon, 24 Aug 2026 16:20:57 +0000

I maintain a small Android app that stitches a scrolling screen into one long image. It worked well until people started pointing it at social feeds, and then users began reporting that captures "overlapped" around video posts.

Chasing that report taught me it was two separate defects wearing one description, and that my measurement could only see one of them.

The measurement comes first

Before changing anything I made the failure reproducible offline. Debug builds write every delivered frame's grayscale to a journal file, so a real session can be replayed against any candidate algorithm - the only honest way to compare two engines, since you can never reproduce a hand gesture twice.

Then I needed ground truth that did not come from the engine. Matching consecutive journal frames directly, with no chaining and no accumulated estimate, gives the real scroll between them, and therefore the real page position of every frame. Now every row the engine commits can be checked: canvas rows increase, so the page rows they carry must increase too. A step backwards means content already emitted was emitted again.

That audit is what let me stop guessing.

Defect one: the video captures the measurement

A playing video is not noise. Uncorrelated noise costs the same at every candidate offset, so it cannot move the argmin - I built a noise fixture first and it passed happily while the real bug sat untouched. Real video is structured and its scene moves coherently, which plants a second minimum in the offset landscape at the video's own motion.

When the video covers more of the textured pixels than the page does, every majority-of-pixels metric prefers that minimum. Rows land at the wrong page address. On a synthetic feed with a coherently panning video the error compounded at exactly the video's pan rate, every single frame.

The obvious fix is to exclude moving rows from the measurement. That fix has a trap in it.

The inversion

To know which rows are moving you compare the last two frames - but you have to compare them at some offset, and the only offset available is the last one you accepted. So the test is anchored to the answer it is supposed to be checking.

The moment that offset is wrong, the test inverts. Under the video's inflated offset the video's rows line up perfectly and are judged still, while the real page's rows do not and are judged to be the thing that moves. The exclusion mask ends up protecting the noise and discarding the evidence.

It cannot be repaired downstream either. I tried re-registering against the committed image - but that image was written by the same wrong chain, so it agrees with the error and confirms it. Measured: the trimmed cost was 73-89, flat, at every single offset. There was nothing left to appeal to.

Rigid slice consensus

What worked was removing the anchor.

Page content is a rigid body. Every horizontal slice of it moves by the same offset and reproduces the previous frame almost exactly there. Whatever refuses to fit that one offset is moving by itself: a video, a GIF, an advert, a live chart.

So: split the visible band into slices of about 128 px, score each against the offset the frame just produced, and read off which ones fit. If nothing fits at all, refuse the frame. If the winner explains almost nothing but a distant rival exists, score the rival too and let consensus - not average cost - pick between them.

This needs no previous estimate, which is the entire point. A consensus cannot invert. At worst it finds too little agreement, and that is a refusal, not a wrong answer.

Two details mattered more than the idea itself:

The consensus must never be fed back into verification. I tried it. An offset that is allowed to excuse whatever it fails to explain will certify itself, and my fuzz harness immediately started putting solid rows on the wrong page. Verification has to use evidence that is independent of the offset under test.

Excluding a slice and remembering it need different thresholds. Excluding a slice from one frame is cheap and reversible. Recording a page region as animated steers every later frame and survives freezes, so it needs roughly double the evidence - otherwise a one-off entrance animation brands that address for the rest of the session.

Results on the synthetic worst case - a coherently panning video over 76% of the viewport - went from 10, 30 and 2 misplaced rows across three seeds to 0, 1 and 11. On a real captured session, re-emitted rows went from 1056 to 0.

Defect two: the one placement checks cannot see

Meanwhile the user kept saying the video area still overlapped, and my audit kept reporting zero misplaced rows. We were both right.

A media area's rows arrive over many frames. The video moves on between them. So the area ends up assembled from different instants - the scene visibly cuts part-way down - while every row sits at exactly the page address it belongs to. Positional accuracy cannot fix it, because the addresses were never wrong. The moment was.

You need a second measure entirely: for each animated page range, how many distinct source frames contributed. One is coherent. More is a tear. Adding that number to the audit was the first time the defect became visible as a number instead of as a complaint.

The fix follows directly: stop stitching those rows. Once a media area is wholly on screen in a single frame, that frame holds a coherent picture of it, so write the whole area from that one frame, with static rows riding along on both sides so the canvas can prove the placement before anything lands.

What I would tell my past self

Build ground truth that does not run through the thing you are testing. My audit and my engine shared a measurement for a while, so they agreed with each other while both were wrong.

Fixtures can lie by being too kind. The noise-video fixture passed. It felt like evidence. It was the absence of evidence.

When two failures share one symptom, the user is usually reporting the one your instrument cannot see. Three rounds of "still broken" against a clean audit should have told me the instrument was incomplete far sooner than it did.

The app is LongShot - free, no ads, no accounts, and no INTERNET permission at all, which for a tool that reads your screen felt like the only defensible design. Text tools are Latin-script only and you do the scrolling yourself; there is deliberately no accessibility service.