This is a decision log, not a mechanism tutorial — part 4 of this series covers how our frame-skipping gate and temporal EMA work internally. Here we cover the other half of the story: why we replaced a shipped, working safety mechanism with a simpler one, how we validated the change without betting the product on it, where the first run regressed hard enough to trip our preset abort criterion, and the two small fixes that turned the redesign into a 1.5dB quality win over the old baseline. If you take one thing home, take the discipline: decide your stop line before you run the experiment, and make sure your test set includes the worst case you have ever shipped a bug on.
How to speed up video inpainting that already works
Some context. Our in-browser video watermark remover inpaints frames one by one, but most frames near a static watermark barely change, so a motion gate lets us skip inference and reuse an earlier result. The first version of that gate (v1) compared each frame to its immediate neighbor and reused the previous output. That design had already failed once, badly: on footage with a slow global gradient, per-frame differences stay tiny forever, the gate skips forever, and the repaired region freezes into a visible rectangle. That incident cost us 22dB on our end-to-end masked-region PSNR (38.7 down to 16.7) and earned itself a dedicated regression asset: a synthetic clip whose background drifts continuously, built precisely so no future change could reintroduce chain-freezing unnoticed.
The fix we shipped was a cumulative drift budget plus a brightness-delta compensation: accumulate the error of skipped frames, force re-inference when the budget is spent, and nudge reused pixels by the scene's measured exposure shift. It worked. The frozen-rectangle bug stayed dead, quality held at 38.44dB against a 38.74 no-skip reference, and the safety net went to production.
So the starting position of this story is not a broken system. It is a working one with a layer of indirection in it.
The anchor idea: keyframe reuse in video processing
In a design review, someone asked the question that started everything: why does each frame compare itself to its neighbor at all?
Think about what the skip error actually is. When we skip a frame, we reuse the output of the last frame that really ran inference — call it the anchor. The error we are committing is exactly "how different is the current frame from the anchor." The drift budget only ever approximated that quantity by summing per-step differences and hoping the steps add up. Comparing against the anchor directly measures the thing itself: the skip error is bounded by the threshold, by definition, with no accumulator, no proxy, no edge cases about whether slow drift can sneak past.
The theory also predicted we would lose nothing on speed. Summing noisy per-step differences can only overcount relative to a direct comparison (oscillation and noise accumulate in the sum but cancel in the direct diff), so the skip rate should stay flat or improve. Fewer lines of state, a stronger guarantee, equal-or-better performance. Elegant ideas like this are exactly the ones that blow up in your face, so we wrote the rules down first.
PSNR regression testing with a preset stop line
Our rule for pipeline changes is that they pass the end-to-end quality gates before they ship, full stop. For this experiment we made the gate explicit before running anything: the redesign would be aborted — not patched, not negotiated — if masked-region PSNR on any e2e asset dropped below 36dB. That number was chosen in advance, while we were still neutral about the outcome. Deciding the abort criterion after seeing the results is how regressions get rationalized into production.
The gates themselves deserve a mention, because they are the second time this story's real hero shows up. The e2e suite includes that synthetic gradient-drift clip born from the 16.7dB chain-freeze incident — deliberately the worst case for any reuse-based optimization, with a background that never holds still but changes too slowly to trip naive motion detection — plus a four-segment subtitle asset where the masked content changes constantly. One asset attacks stale reuse, the other attacks insufficient reuse. Whatever we changed, one of them would notice.
First run: pass, pass, pass, stop
The anchored rewrite went in, and the first full run was a split decision:
- Subtitle asset: all four segments improved by +3–4dB (37.3 / 32.3 / 28.7 / 30.1dB). The direct comparison was doing exactly what theory promised on fast-changing content.
- Gradient-drift asset: masked-region PSNR fell from 38.30 to 34.69dB — below the 36dB stop line.
This is the moment the discipline is for. The aggregate picture looked fine — better, even — and it would have been easy to average the results, declare victory on subtitles, and tune later. We stopped instead, exactly as agreed, and treated 34.69 as a bug report the experiment had written for us. No rollback either: the stop line meant "stop and explain," not "revert on sight."
Error accumulation in video processing, round two
The root cause analysis turned up something genuinely new — a failure mode the chained design had been hiding from us.
The gradient clip's background does not drift uniformly. Its exposure shift varies with vertical position (the asset synthesizes it roughly as sin(y/60 + t·4) and cos(y/80 + t·3) components), while our delta compensation is a single per-channel mean over the region. A mean cannot cancel a gradient. Under the old chained design, reused pixels were always exactly one frame old, so the uncompensated residual stayed small and roughly constant. Under anchoring, a reused output can be two or three frames away from its anchor, and the residual grows with anchor age. The redesign had not created the compensation error; it had amplified it by extending reuse lifetimes — the same mechanism that made everything else better.
We also found a second, dumber bug in the same territory. The temporal EMA blends the current output against the previous frame's output, so its history term needs compensation against the previous frame. In the rewrite we had fed it the to-anchor delta instead — a systematic overshoot on every blended pixel, because the compensation was answering a different question than the blend was asking.
Two small fixes, no rollback
Both fixes were small and stayed inside the new design:
- Threshold ceiling 8 → 5. The skip threshold is self-calibrating (median of the first five frames' differences, times three, clamped to [2, 5]/255). Lowering the ceiling forces re-anchoring more often, which directly bounds anchor age — attacking the residual-accumulation mechanism rather than its symptom.
- Split the deltas. Compositing reuses the anchor, so it uses the to-anchor delta. The EMA blends against the previous frame, so it gets the to-neighbor delta. One line of reasoning each, no shared state pretending two timebases are one.
The calibration logic itself did not change at all — only the clamp ceiling and the delta plumbing.
Results: beating the baseline
The final anchored version, same gates:
- Gradient-drift asset: 39.84dB masked-region PSNR — 1.5dB above the chained baseline, not merely recovered. With the delta semantics fixed, direct comparison and exact to-anchor compensation turned out to be strictly better than the approximation we had been defending.
- Subtitles: the +3–4dB gains held, with 67% of frames skipping inference.
- Speed: 58% of frames skipped on the watermark asset, flat versus the budget version as predicted, with the entire drift-budget code path deleted.
The safety net we removed was never missed. The one we kept — the preset stop line and the worst-case test asset — caught the one regression that mattered.
Takeaways
- Write the abort criterion before the experiment. "Masked PSNR below 36 means stop" was decided while we were neutral. After seeing +3–4dB wins, that line is much harder to draw honestly.
- Your test set should contain your worst historical bug, weaponized. The gradient-drift asset was built to catch chain-freezing two years of refactors ago; it has now caught a second, completely different failure. Worst-case assets appreciate in value.
- Aggregates hide regressions. Subtitles up 3–4dB and gradient down 3.6dB averages to "improved." We read per-asset results first, means second, and the stop line was per-asset too.
- Simpler mechanisms fail differently, not gently. Deleting the accumulator removed a whole class of proxy-error bugs and introduced none — but it amplified a compensation error the old design had been silently capping. Budget analysis time for the new failure modes of elegant ideas.
- Fix forward when the root cause is small. Both fixes were a few lines inside the new design. Reverting would have thrown away a real quality win to avoid understanding a small bug.
Try it
The anchored skip gate ships in our free video watermark remover, and the same pipeline drives our subtitle removal tool — both run entirely in your browser, no uploads. If you have footage that breaks our gates, we genuinely want to hear about it; our best test assets were born that way.
Part 7 of the ClearPix engineering series — how we build free, private, in-browser media tools at clearpix.org.
Top comments (0)