Video.js: frame-accurate subtitles

#javascript #video #postproduction #videojs

In the post-production and broadcast industry videos are processed frame by frame.
Subtitles follow the same rule : in and out timecodes are frame accurate (ex: 00:00:23:22 -> 23 seconds and 22 images).

Browsers process videos with millisecond timestamps, that's why subtitles have to be converted based on video framerate: for video at 25 images per second the previous timecode would look like 00:00:23.880 in milliseconds format (1000/25 * 22 = 880).

Frame accuracy is really important : subtitles need to disappear right before the next cut and appear right after the previous one.
Problem is that default HTML5 video's refresh rate is too low: subtitles appear and disappear too late creating a poor viewer's experience (and a non-broadcast compliant one).

Example below higlights the issue. Subtitles in the fixed video disappear right before the change of plan:

It might seem marginal, but the post production industry needs frame accuracy.

Due to browser limitations, timeupdate (event fired when the playing position of a video has changed) is fired every 150-250 milliseconds. It is not enough for frame-accuracy: 25fps means an update every 40ms.
We need to compute which subtitle has to be displayed every frame (instead of doing it every 4-5 frames by default). Video.js subtitles engine does the computation each time the text track's attribute activeCues getter is called.

text-track.js extract from Video.js source code:

Object.defineProperty(tt, 'activeCues', {
    get() {
        if (!this.loaded_) {
            return null;
        }
        // nothing to do
        if (this.cues.length === 0) {
            return activeCues;
        }
        const ct = this.tech_.currentTime();
        const active = [];
        for (let i = 0, l = this.cues.length; i < l; i++) {
            const cue = this.cues[i];

            if (cue.startTime <= ct && cue.endTime >= ct) {
                active.push(cue);
            } else if (cue.startTime === cue.endTime &&
                cue.startTime <= ct &&
                cue.startTime + 0.5 >= ct) {
                active.push(cue);
            }
        }
        changed = false;
        if (active.length !== this.activeCues_.length) {
            changed = true;
        } else {
            for (let i = 0; i < active.length; i++) {
                if (this.activeCues_.indexOf(active[i]) === -1) {
                    changed = true;
                }
            }
        }
        this.activeCues_ = active;
        activeCues.setCues_(this.activeCues_);
        return activeCues;
    },
    set() {}
});

Then you need to call trigger('cuechange') on the text track to make sure the video display is up to date:

player.textTracks()[0].activeCues; // computes the current subtitle based on current time
player.textTracks()[0].trigger('cuechange'); // updates the display

requestAnimationFrame is optimized for animations and has a lot less delay than setInterval or setTimeout, so we are going to use it for our time sensitive loop (Frame rate control source here).

Here is the complete source code:

var fps = 25;
var now;
var then = Date.now();
var interval = 1000/fps;
var delta;

function reloadCues() {
    requestAnimationFrame(reloadCues);

    now = Date.now();
    delta = now - then;

    if (delta > interval) {
        then = now - (delta % interval);

        if(videojs.players.player.textTracks().length == 1) {
            videojs.players.player.textTracks()[0].activeCues;
            videojs.players.player.textTracks()[0].trigger('cuechange')
        }
    }
}
reloadCues();