Why Video Subtitle Translation Is Harder Than It Looks

Video subtitle translation sounds straightforward: get the subtitles, translate them, and display the result.

While building video subtitle translation for SelectTranslate, however, the translation step turned out to be only one part of the system. The harder problem is keeping three things aligned: what subtitle data the platform currently makes available, what has already been translated, and what should be visible at the current playback position.

This gets complicated quickly because video platforms do not expose subtitle data in the same way, and not every video has a usable subtitle track in the first place.

Two Subtitle Sources, Two Different Pipelines

Before translation starts, the first question is where the subtitle text comes from.

Dimension	Platform-provided subtitles	Videos without usable subtitles
Source	Existing subtitle data	Original video audio
Text generation	Already provided by the platform	Requires speech recognition
Timing	Usually provided, but may require adjustment	Generated together with recognized speech
Main difficulty	Availability, completeness, and timeline alignment	Recognition, segmentation, and timing
Shared stages	Translation and synchronized rendering	Translation and synchronized rendering

When native subtitles are available, SelectTranslate can work with the subtitle data already provided by the video platform. The original subtitle layer can then be hidden while the translated version is rendered against the same video timeline.

When no usable subtitle track exists, the process starts earlier: the video audio has to be recognized as text first, together with enough timing information to turn it into subtitles.

The two paths eventually meet at translation and rendering, but everything before that is fundamentally different.

Subtitle Data Is Not Always Available All at Once

One of the first assumptions to break is that a video player always exposes its entire subtitle track when playback begins.

Some platforms do make most or all subtitles available early. In that case, the workflow is relatively clean: retrieve the available subtitle data, organize it by timestamp, translate it as a batch, and associate the translated text with the corresponding subtitle segments.

Other platforms expose subtitles progressively.

Only part of the track may be available at first. As playback reaches later positions, additional subtitle data becomes accessible. The subtitle system therefore cannot always treat acquisition as a one-time operation.

For example, the state might look conceptually like this:

• At the beginning, subtitles for the first section are available.
• Playback continues and another group of subtitle segments becomes accessible.
• The newly retrieved segments are added to the existing timeline and translated.
• The same process may repeat as the video progresses.

This also means newly available subtitle data cannot simply be translated blindly. The system first has to determine what has already been processed and what is genuinely new, otherwise overlapping or repeated data can lead to redundant translation.

The important point is that subtitle availability is controlled by the video platform, not by the translation pipeline.

Acquisition, Translation, and Playback Are Separate States

This separation turned out to be one of the most important parts of the architecture.

Subtitle acquisition, translation, and playback may all be at completely different positions.

A video might currently be playing at 02:00, while subtitle data up to 20:00 has already been retrieved and translated. In another case, playback may reach a later section before the platform makes the next subtitle batch available.

So the system effectively has to track three independent states:

Acquisition — Which subtitle segments are currently available?

Translation — Which of those segments already have translated text?

Playback — Which subtitle should actually be visible right now?

A simplified internal representation can look like this:

ts
interface SubtitleSegment {
  text: string
  translatedText?: string
  startTime: number
  endTime: number
}