Most "remove burned-in subtitles" tutorials end at a before/after. The after always looks fine as a thumbnail. The useful question is what "fine" means numerically, and almost nobody answers it.
I spent a while on this, so I measured it. Here is the method, the numbers, and the two clips where it goes wrong.
The setup
PSNR needs a reference. So use clips where you can produce both versions: same camera, same encoder settings, one with text burned in, one without. Then you compare your output against the clean plate instead of against your memory of it.
ffmpeg -i clean.mp4 -i output.mp4 -lavfi psnr -f null -
Two caveats before the numbers.
PSNR is per-pixel and per-frame. It cannot see temporal crawl, which is the artifact people actually notice on subtitle removal. A method can score well and still shimmer when you pause it. PSNR is a lower bound on quality, not a verdict.
The ceiling matters more than the score. Re-encode the clean clip and compare it to itself, and you get the best number reachable at those settings. Mine was 37.4 dB. A 28 dB result against a 37.4 dB ceiling is a much better outcome than 28 dB sounds on its own.
Results
| clip | PSNR | what's in frame |
|---|---|---|
| clean re-encode (ceiling) | 37.4 dB | reference |
| street, slow pan | 28.3 dB | text over a moving background |
| crowd, text crossing faces | 19.8 dB | people moving under the text |
| vertical, drifting text | 20.5 dB | 1080x1920, karaoke-style motion |
Same clips, naive baselines:
| method | PSNR |
|---|---|
ffmpeg delogo |
16.8 dB |
| box blur over the region | 18.0 dB |
| feeding the subtitled clip back in as "output" | 8.8 dB |
That last row is not a joke, it is the floor. It tells you how much of any score is just the text still being there.
Both naive methods land below the crowd clip that everyone would call a failure. That is the honest summary of covering text instead of rebuilding it: you are not removing anything, you are smearing it, and the metric says so.
Why the crowd clip collapses
Text over faces fails for a specific reason. The text occupies the same pixels across many frames, so there is no clean frame to borrow the background from. The information was never recorded. Whatever fills that hole is invented, not recovered.
Then the content compounds it. What has to be reconstructed is skin and hair with fine structure, and any residual smearing reads as wrong instantly, because we are tuned to faces. On the street clip the background is a wall. A slightly soft wall is still a wall. A slightly soft cheek is a mistake.
The vertical clip fails differently. The mask moves with the text, so the region that needs rebuilding changes every frame and nothing can be amortised across the clip. Static subtitles are the easy case, and most subtitles are not static.
What PSNR does not tell you
- Temporal consistency. The main reason a good-scoring result still looks bad.
- How it holds up at 2x on a phone screen, which is where most of this content is watched.
- Anything about audio, obviously.
And one bias worth knowing: PSNR rewards blur. A soft reconstruction of a textured region often scores higher than a sharp one with slightly wrong structure. So the metric leans toward exactly the failure mode people complain about. If you optimise against PSNR alone you will ship something that scores well and looks like vaseline.
Reproducing it
Disclosure: I build one of these tools, so treat the numbers as coming from an interested party. That is precisely why the clips and the exact commands are public, including the failures:
- Benchmark repo with the clips and commands: github.com/vaughnn/burned-in-subtitle-removal-benchmark
- The tool I work on: videotextremove.com
If you have a clip I did not test, run the two commands above against your own clean plate. It takes about a minute and it will tell you more than any comparison table, including mine.
Top comments (0)