From Burned-In Text to Bilingual SRT: How Subtitle OCR, Local Whisper and AI Translation Fit Together

If you have ever tried to get usable subtitles out of a video you did not make, you have probably hit one of two walls. Either the subtitles are burned into the pixels (hardcoded), so there is no subtitle track to extract, or there are no subtitles at all and you need to produce them from the audio. Then, more often than not, you also want them in another language.

This post walks through how each of those three steps actually works under the hood, where they tend to break, and a practical workflow for combining them. I'll use small command-line examples to show the moving parts, then show how I run the same pipeline day to day in a desktop app.

Part 1: Extracting hardcoded subtitles with OCR

Hardcoded subtitles are just pixels. To turn them back into an .srt file you need to answer two questions for every line: what does it say and when is it on screen. OCR only solves the first one. The second one is where most of the engineering lives.

A typical pipeline looks like this:

1. Sample frames. You do not need all 24 or 30 frames per second. Subtitles usually stay on screen for one to several seconds, so sampling a few frames per second is enough to catch every line while keeping the workload sane.

# Grab 4 frames per second, cropped to the bottom 20% of the picture
ffmpeg -i input.mp4 -vf "fps=4,crop=iw:ih*0.2:0:ih*0.8" frames/%06d.png

2. Restrict the region. Cropping to the subtitle area is the single biggest accuracy win. Without it, OCR happily reads shop signs, on-screen titles, and logos, and your output fills up with noise.

3. Preprocess. Subtitles are usually white or yellow text with a dark outline. Converting to grayscale, boosting contrast, and sometimes thresholding helps the recognizer separate text from a busy background. Over-aggressive thresholding can also erase thin strokes, which hurts CJK text in particular, so this step needs tuning per source.

4. Recognize text. Modern OCR engines run in two stages: a detector finds text boxes, and a recognizer reads each box. For subtitles you mostly get one or two lines per frame, which is an easy case for detection but a hard one for recognition when fonts are stylized or the video is low resolution.

5. Deduplicate and build timings. This is the part people underestimate. At 4 fps, a line that stays up for 2 seconds appears in about 8 consecutive frames, and OCR will not read it identically every time. You need to:

• compare consecutive results with a fuzzy similarity score instead of exact string equality,
• merge runs of "same" text into one cue,
• take the start time from the first frame and the end time from the last frame of the run,
• drop one-frame flickers that are usually misreads during fades.

The precision of your timestamps is bounded by your sampling rate. At 4 fps you can be off by up to 250 ms on each edge, which is usually fine for reading but noticeable if you are syncing to speech. Pixel-difference checks on the subtitle region can help you detect the exact frame where text appears or disappears without running OCR on every frame.

Part 2: Transcribing audio locally with Whisper

When there are no subtitles in the picture, you go to the audio. OpenAI's Whisper models made good speech recognition something you can run on a laptop, and running it locally has two obvious benefits: your footage never leaves your machine, and there is no per-minute bill.

A minimal run with the open-source CLI looks like this:

ffmpeg -i input.mp4 -ar 16000 -ac 1 audio.wav
whisper audio.wav --model medium --language ja --output_format srt

A few things worth knowing before you rely on it:

• Model size is a speed and accuracy trade-off. Smaller models are fast but stumble on accents, overlapping speech and uncommon names. Larger ones are noticeably better for non-English audio and need more memory and time.
• Set the language if you know it. Auto-detection works most of the time, but it decides from the opening seconds. A video that starts with music or a short English intro can get misdetected.
• Watch out for silence and music. Whisper is a sequence-to-sequence model, so on long stretches without speech it can "hallucinate" text, repeat the previous line, or produce things like "Thanks for watching." Voice activity detection (VAD) before transcription, which only feeds speech segments to the model, cuts this down a lot.
• Segment timestamps are not subtitle cues. Whisper's segments can be long or split mid-phrase. For readable subtitles you usually re-segment by sentence or by a maximum characters-per-line and duration rule, then nudge the boundaries so lines do not flash by too fast.

Part 3: AI translation without breaking the timing

Once you have a clean source-language SRT, translation looks trivial: send each line to a model, get a line back. Doing it line by line, though, produces the classic machine-subtitle feel, because subtitle lines are often fragments of a sentence that continues in the next cue.

Things that make a real difference:

• Translate in context windows. Send a block of consecutive cues together (with their indexes) so the model can see the full sentence and the surrounding conversation, then ask for the same number of cues back.
• Keep the cue count and order fixed. The timing comes from the source file. If the model merges or splits lines, you lose alignment. Validate the response: same count, same indexes, no empty lines.
• Respect reading speed. Translations into some languages run much longer than the source. A rough limit of two lines and a sensible characters-per-second ceiling keeps them readable.
• Use a glossary for names. Character names, places and recurring terms should be fixed up front, or the model will spell them three different ways across an episode.
• Keep both languages. For language learners and for anyone reviewing a translation, a bilingual subtitle (original on one line, translation on the next) is far more useful than the translation alone. It also makes mistakes easy to spot.

Part 4: A practical workflow

Gluing all of this together with scripts is a good learning exercise. I did it, and it worked, but the maintenance cost was real: per-video crop tuning, VAD parameters, retry logic for the translation calls, and then a separate editor to fix timings by hand.

These days I run the same pipeline in GeekLink, a desktop app for Mac and Windows that does each of these steps in one place:

1. Import the video. If it has burned-in subtitles, run OCR extraction on the subtitle region. If it does not, run local Whisper transcription. Both run on your own computer.
2. Review the source text first. Skim the extracted or transcribed lines before translating. Fixing a misread name once in the source is much cheaper than fixing it in every translated line.
3. Run AI translation to the target language. The result is laid out as a bilingual subtitle, original and translation per cue.
4. Polish in the bilingual editor. Adjust timings, split overlong lines, and correct anything that reads awkwardly, with the original right next to the translation.
5. Export the result as subtitles to use in your player or editing software.

Whether you script it yourself or use a tool, the checks are the same:

• Spot-check the first and last few minutes. That is where OCR misses fade-ins and Whisper misdetects language.
• Search the output for repeated lines. Long identical runs usually mean a Whisper hallucination over music or an OCR dedup failure.
• Read the bilingual version once at normal speed. If you cannot follow it, your viewers cannot either.

Pricing, for reference

GeekLink has a Free plan, and paid options at $12.99/month, $99/year, or a $129 Lifetime license. If you only need occasional extraction or transcription, start with the free plan and see whether it covers your case.

Wrapping up

OCR, speech recognition and translation each look like a solved problem in isolation. The hard parts are in the seams: turning per-frame OCR into clean cues, turning Whisper segments into readable subtitles, and translating fragments without losing the sentence or the timing. Once you understand where those seams are, you can debug any subtitle pipeline, whether it is your own Python script or a desktop app.

If you have built something similar, I'd love to hear how you handled deduplication and re-segmentation. Those two steps are where every pipeline I've seen differs the most.