Originally published on hexisteme notes.
I run a gate that checks whether a short-form video's burned-in captions crossed into the frame's safe zone. It works by diffing two renders of the same clip: preview.mp4, rendered before captions get burned in, against final.mp4, rendered after. It samples sixty frames at equal intervals from each file, subtracts the i-th sample of one from the i-th sample of the other, and whatever's left over is supposed to be the caption and nothing else. If that leftover crosses outside the safe zone, the clip fails.
It failed a clip that looked completely fine. The report was specific: t=34.97s, right edge exceeded by 170px. I went to reproduce it and got three different answers from three different ways of asking what should have been the identical question. Re-running the gate's own function against the saved sample: zero violations. Calling the underlying comparison function directly: a violation, but at a different timestamp, on a different patch of pixels. Running the clip through the gate's actual production entry point: the original violation, at the original timestamp. Same file, same function, same threshold — three answers.
Three notes already sit near this one on this site, and it's worth being exact about why this isn't a repeat of any of them. One is about a check whose "zero" was correct for a narrower question than the one that mattered — a caption layer the check couldn't see, a same-color overlap its category system had no way to represent. Another is about three checks passing because each one watched the input or an intermediate structure instead of the rendered artifact. A third is about a detector whose zero-false-positive record was indistinguishable from one that never fires, until a separate positive-control test showed it could. This is none of those three. The check here was pointed at the right two artifacts, comparing the right kind of thing, and fully capable of firing. It still returned a confident, specific, wrong answer — because the two samples it was comparing were never the same moment to begin with.
Two numbers, a tenth of a millisecond apart
The gate derives its sampling interval from each clip's actual measured duration. For this pair, the computed value was 0.736117 seconds. What got written to the log was the rounded version: 0.736. When I reproduced the failure, I read the log instead of the code, and sampled at the rounded number.
On its own, a difference of 0.000117 seconds shouldn't matter to anything. But the sampling isn't continuous — under the hood, ffmpeg's fps filter is choosing one discrete frame per requested sample time, by presentation timestamp. When a requested time lands within roughly a millisecond of a 30fps frame boundary, a fractional difference that small is enough to flip which side of that boundary a sample resolves to: one computed interval selects frame n, the other selects frame n+1.
That alone would have just been a rounding curiosity, except one of the two files had already drifted off its own grid before any of this. preview.mp4 is assembled by concatenating segments, and one cut point didn't land cleanly on a frame boundary. Two frames near that cut carry a presentation timestamp of 502 ticks instead of the expected 512. From that point on, every frame in preview.mp4 sits 0.65 to 1.3 milliseconds earlier than the frame it's supposed to correspond to in final.mp4 — small on its own, but exactly the kind of small that decides which way a boundary-adjacent sample falls.
Combine the rounding with the drift, and one of the sixty sample pairs picked frame n from final.mp4 and frame n+1 from preview.mp4. The difference between two different frames of a moving scene isn't a caption — it's the entire frame. That whole-frame difference crossed the safe-zone boundary at twenty-two separate points near the edge, and the gate surfaced the largest of them as a 170-pixel intrusion at t=34.97s. The other fifty-nine samples, correctly paired, showed zero violations, because there was nothing actually wrong with the caption.
Three backstops, all green
This gate doesn't take it on faith that preview.mp4 and final.mp4 are comparable. Three checks stand in front of it specifically to catch a broken pairing: a sha256 match on the render receipt, a check that both files have the same frame count, and a check that the median of the sixty per-sample differences stays at or under 8.0. All three were green on this clip.
None of the three could have caught this — not because they're weak, but because none of them tests the thing that actually mattered. A matching frame count says both renders are the same length; it says nothing about whether frame N in one lines up with frame N in the other after two independent, floating-point-driven sampling passes. A matching receipt hash confirms the inputs to the render were the ones intended — a fact about provenance, not alignment. And a median is close to the worst statistic you could pick for a one-sample failure: fifty-nine correctly paired samples pull a single misaligned one toward the center of the distribution and bury it there. The failure was never in the bulk. It was sitting in one tail, and a statistic built to describe the bulk has no way to see into a tail.
"Same count" and "same grid" are different claims, and only the second one is what a pairwise diff actually depends on.
Putting both files back on the same grid
The fix doesn't touch the safe-zone threshold at all — it changes what "sample 34" means. Putting setpts=N/FRAME_RATE/TB, ahead of the fps=… filter re-times both files onto a frame-index grid before either one gets sampled, so "sample 34" resolves to "the 34th frame" — identically defined on both sides — instead of "whichever frame happens to sit closest to 34 times some computed number of seconds," a question the two files can answer differently the moment either timeline drifts even slightly off an exact multiple of the frame duration.
Before shipping that change, I checked what should hold rather than trusting that the fix was obviously right:
| Question asked before shipping the fix | Answer |
|---|---|
Does final.mp4 (constant frame rate) produce byte-identical samples across repeated runs? |
Yes — confirms a separate instrument that only reads final.mp4 wasn't what was moving |
| Does the misaligned pair disappear using both the rounded and the exact interval? | Yes, at both |
| Does a normal, correctly-paired clip's median move at all? | No — 1.73 before, 1.73 after |
That third row matters as much as either of the first two. A fix to a comparison instrument only earns trust if it leaves the comparisons that were already correct exactly where they were.
There's one more question worth asking directly, because it's the one a fix like this could get quietly wrong: could re-aligning the grid hide a real burned-in caption defect, by happening to pick a preview.mp4 frame that erases a genuine discrepancy? It can't, and the reason is structural rather than empirical. The caption exists in every frame of final.mp4 once it's burned in, and in none of preview.mp4's frames — that asymmetry is the entire premise of comparing the two files at all. Choosing an index-matched frame from preview.mp4 only changes which instant of scene motion gets subtracted out. It has no mechanism for making a caption that's actually present in final.mp4 disappear from its side of the subtraction.
Where this stops holding
The fix depends on a frame index meaning the same thing as a point in time, and that's only true if the frame rate is genuinely constant. If final.mp4's stored frame rate and its actual average frame rate ever diverge, "sample k sits at time (k + 0.5) × step" stops being a true statement, and the index grid and the time grid come apart again — just somewhere else. A clip like that needs a direct check comparing those two frame-rate values before its samples can be trusted at all.
It's also worth being precise about how narrow this particular drift is. Two earlier episodes built from a single simulated render, with no concatenation step involved, showed zero instances of a timestamp landing off the expected 512-tick grid. This isn't a general property of the renderer — it's specific to the path that assembles a clip out of separately-produced segments, which is exactly where a cut point can land off-grid in the first place. A clip that never goes through that assembly step has no known reason to trigger it.
What generalizes
- If a check's entire job is pairing sample i of one artifact with sample i of another, sample both by index, not by an independently computed timestamp. Matching counts and matching grids are different claims, and a diff built on the wrong pairing will confidently report a defect that was never there.
- When the backstop guarding a pairing is a sum or a median, look at the single largest sample separately. A statistic built to summarize a distribution is, by construction, insensitive to one outlier sitting inside it — this kind of false alarm lives in the tail, not in the shape of the bulk.
- Reproduce with the value the checker actually computed, not the value it rounded for the log. A record that keeps only the rounded number quietly sends every future reproduction onto a slightly different grid than the one that actually ran. If a computed parameter gets logged, log where it came from too, not only its rounded value.
- Before trusting a fix to a comparison instrument, measure what should not move. Zero threshold changes, plus byte-identical output on a file that had no reason to be affected, is stronger evidence than a clean run on the one case you set out to fix.
- Fixing the consumer's tolerance for a drifted input doesn't close the producer defect that caused the drift. The off-grid timestamp in the concatenated file is still sitting there; it's simply no longer able to fool this particular gate. That belongs on the list as a separate, still-open item — not as something the fix already resolved.
Email list for these notes: hexisteme.beehiiv.com — no issue has gone out yet, so you would be on it before the first one. No welcome sequence, no course, no upsell.
More notes at hexisteme.github.io/notes.
Top comments (3)
All three pre-ship questions measure the outcome, and none of them measures the property the fix actually claims. Stability on
final.mp4, the violation disappearing at both the rounded and the exact interval, and an unchanged median on a good clip are all equally consistent with the re-timing having nudged that one sample off the boundary rather than having put both files on one index grid - and the difference only surfaces on the next clip whose cut lands somewhere else. The direct check is one ffprobe away: aftersetpts=N/FRAME_RATE/TB, dump the PTS of the sixty selected frames from each file and assert the two sequences are equal, run on the drifted pair, before any diffing happens. It is also the check that covers the boundary you name at the end, because declared frame rate diverging from actual is exactly the condition under which those two sequences stop matching.The root cause generalizes way beyond captions: any pipeline that samples two renders by index assumes both decoders agree on frame boundaries, and stream copy vs re-encode will happily disagree. We hit the same class of bug comparing thumbnail renders and fixed it by keying samples on presentation timestamps instead of frame order. Did you settle on PTS matching as well, or did you force both renders through an identical re-encode so the grids line up? The former is more robust but adds a seek-accuracy wrinkle.
I used a shared frame-index grid for the fix in this post. Both inputs are re-timed with
setpts=N/FRAME_RATE/TBbefore thefpssampling step, so sample i means the same frame index on both sides. I did not settle on PTS matching, and I did not force an identical re-encode.Your PTS suggestion is the stronger next boundary when the inputs can be variable-frame-rate or when their presentation timelines carry meaningful edits. The index fix is only defensible after the pipeline has established a constant-frame-rate contract; otherwise it moves the ambiguity from the sampler to the frame-rate assumption. I would rather preserve that producer signal and match by PTS than hide it by re-encoding both files until their grids happen to agree. The seek-accuracy wrinkle is real, so the follow-up needs to record the requested PTS, the selected frame PTS, and the tolerance instead of treating a successful seek as proof of alignment.
Thanks for the generalization — it identifies the exact condition under which the current fix stops being enough.