DEV Community: William Huster

Automatically Caption Your Videos with Whisper and ffmpeg

William Huster — Sat, 12 Aug 2023 14:00:00 +0000

Big video handlers like YouTube and even Slack use AI to automatically caption videos that you upload. This is a great win for accessibility, and the AI is extremely accurate. It sometimes stumbles on uncommon words and bad audio signals, but these can be easily fixed by a human. I think most would agree that the benefit and cost savings of generating 95% accurate captions greatly outweighs the cost of having none at all!

So at ThinkNimble we were talking about how we could build this feature ourselves using open source tools. I came up with this suggestion that uses OpenAI's Whisper and the venerable ffmpeg.

There are two steps to this (1) generate captions from the video as an SRT file, and (2) bundle the captions with the video.

Whisper in my experience is incredibly good at audio transcription. And it can directly transcribe mp4 files to SRT format. You don't have to extract the audio from the video or anything like that. Once you have an SRT file, ffmpeg can easily bundle an mp4 and SRT file. I've tried this on some personal videos, and the results have been near perfect.

Installation

Here is my project directory on GitHub. The README includes installation instructions, reproduced here.

https://github.com/whusterj/whisper-transcribe

The only two hard dependencies are the ffmpeg system package and the openai-whisper Python package.

This installation has been tested with Python 3.10.12 on Ubuntu 20.02. It should work on other Platforms as well, and OpenAI says Whisper should work with all Python versions 3.9 to 3.11.

First, you will need ffmpeg on your system, if you don't have it already:

# on Ubuntu or Debian
sudo apt update && sudo apt install ffmpeg

# on MacOS using Homebrew (https://brew.sh/)
brew install ffmpeg

For other platforms, see the [Whisper GitHub repo][1].

Now you can install the python requirements. Create a virtual environment, activate it, and pip install from requirements.txt:

python -m venv .venv/
source .venv/bin/activate
pip install -r requirements.txt

If you encounter errors during installation, you may need to install rust on your platform as well. If my requirements.txt fails to install for some reason, try just installing the openai-whisper package - this should install its Python sub-dependencies:

pip install openai-whisper

How to Do It

Step 1: Use whisper to generate an SRT transcription of the video:

whisper infile.mp4 \
    --model small.en \
    --language English \
    -f 'srt'

I tested this on a two minute video and it took less than 30 seconds to produce a complete transcription. This output is a subtitle file for your video - an SRT file. SRT is a plaintext file format, so you can edit it with your word processor of choice. Open the SRT, and you will see that each line of the file contains start and end timecodes and the caption text. This can be interpreted by video players to display the caption at the right time in sync with the video.

I used the small.en language model, which is about 400MB to download. It may be less accurate than the larger models, but in my experience it does a really good job. The option -f 'srt' specifies that you want an SRT file, but you have your choice of text output format.

You may want to quickly review the SRT file for any misinterpreted words or names

Step 2: Use ffmpeg to add the SRT as a subtitle track:

ffmpeg -i infile.mp4 \
       -i infile.srt \
       -c copy -c:s mov_text \
       outfile.mp4

This took less than 2 seconds to run. It doesn't have to re-render the video, it just bundles the SRT alongside the video track inside of the mp4 container.

You will need a player like VLC that can show you the subtitle tracks. It is also possible to use ffmpeg to "burn in" the subtitles from your SRT file, but this requires re-rendering the whole video.

Automate It All!

The above would be the "by hand" procedure, but perhaps you can see how easily this process might be automated in bulk.

The idea would be that you can drop the mp4 files you want subtitled into an S3 bucket. Then you'd have a scheduler script detect them and queue up jobs for one or more workers. This isn't strictly necessary, but would allow you to fan out the process to as many workers as you like to process more videos faster. The jobs would run the above commands and produce captioned video files automatically, which are then saved back to S3 somewhere.

Show Captions in HTML without Embedding

If you're captioning videos for viewing in web browsers, there's an even easier way.

After my initial write-up, I learned that it's also possible to attach the track to the video using HTML5's built-in <video> and <track> tags. This cuts out step two above, meaning you won't have to use ffmpeg to embed the track in the video file.

In this case, you'll want to generate a .vtt file instead of the .srt above. Fortunately, Whisper supports this format, too!

whisper infile.mp4 \
    --model small.en \
    --language English \
    -f 'vtt'

Once you've generated your captions, you can use them with the HTML5 video player like this:

<video src="myvideo.mp4">
  <source src="myvideo.mp4" type="video/mp4" />
  <track
    label="English"
    kind="captions"
    srclang="en"
    src="/path/to/captions.vtt"
    default
  />
</video>

Create a VueJS Currency Filter

William Huster — Wed, 03 Jan 2018 02:34:00 +0000

Displaying formatted currency amounts is a common requirement of web apps, but VueJS does not provide any filters out of the box. So here’s what you can do if you need to add a currency filter to your Vue project.

First, you'll need to know about VueJS filters and how they work. Read more here: https://vuejs.org/v2/guide/filters.html

Using the currency-formatter NPM Package

If you are using webpack or a similar build tool and have access to node.js packages, then you can install the currency-formatter package from npm. By the way, I highly recommend using the Vue CLI for your projects.

npm install currency-formatter --save

Then you can create a VueJS filter in your app code like so:

import Vue from 'vue'
import currencyFormatter from 'currency-formatter'

Vue.filter('currency', formatNumberAsUSD)

function formatNumberAsUSD (value) {
  if (!value) return ''
  value = Number(value)
  return currencyFormatter.format(value, { code: 'USD' })
}

Vue.filter registers a new global Vue filter called 'currency' and formatNumberAsUSD function does the work. The formatting function does three things, it (a) makes sure it has a truthy value and returns a blank string if not, (b) attempts to convert the value to a Number, and (c) uses currencyFormatter.format to format the number as USD.

Using Number.prototype.toLocaleString()

If you are not using a build tool with access to node.js packages, or you mistrust external dependencies, you might try using the Number.toLocalString() method below or take a peek at currency-formatter source code on Github and borrow from it. Note that while this is supported in modern browsers, it may not be available in older browsers you might need to target.

import Vue from 'vue'
import currencyFormatter from 'currency-formatter'

Vue.filter('currency', formatNumberAsUSD)

function formatNumberAsUSD (value) {
  if (!value) return ''
  return Number(value).toLocaleString('en', {
    style: 'currency', currency: 'USD'
  })
}

Both toLocaleString and the currency-formatter package can handle currencies besides USD, too. Check out the currency-formatter npm page for more details.

Summary

No matter the approach you chose, you have now created a 'currency' filter that you can use in your VueJS templates like this:

<template>
  <span>{{ 12 | currency }}</span>
</template>

And the output should look like this:

$12.00

Excellent!

Presenting: 🚨 Alert! Alert!

William Huster — Tue, 18 Nov 2014 12:00:00 +0000

I built a little, dependency-free JavaScript library to add UI notifications to any web project. I call it: "Alert! Alert!"

You’ll need to compile the LESS CSS before use … I’ll eventually add a Gruntfile or something.

It’s very simple to use:

Alert.alert('info', 'A message', {timeout: 7000});

The last param is optional. If a timeout is given, the notification will disappear after the given number of milliseconds.

It’s nothing groundbreaking, but I was frustrated by the options out there that seem to over-complicate such a trivial thing.

Complete Source Code

Here is the complete source code, which totals about 150 lines of JavaScript and CSS (LESS).

alert-alert.js

'use strict';

module.exports = (function () {

  var container,
      CONTAINER_ID = 'aa-container',
      ALERT_CLASS = 'aa-notification';

  exports = {
    alert: _alert
  };

  return exports;

  /////////////////////////////////////////
  // functions

  function _alert (type, message, config) {
    if (typeof(config) === 'undefined') { config = {}; }
    if (!container) { container = _genNotificationContainer(); }
    container.appendChild(
      _genAlertDiv(type, message, config.timeout)
    );
  }

  function _genNotificationContainer () {
    if (container) { return; }
    var containerDiv = document.createElement('div');
    containerDiv.id = CONTAINER_ID;
    document.body.appendChild(containerDiv);
    return containerDiv;
  }

  function _genAlertDiv (type, message, timeout) {
    var alertDiv = document.createElement('div');
    alertDiv.className = ALERT_CLASS + ' ' + type;
    alertDiv.innerHTML = message;

    //
    alertDiv.addEventListener('click', _alertClickHandler);

    //
    if (timeout) {
      alertDiv.timeout = setTimeout(
        function () {
          _removeAlert(alertDiv);
        }, timeout); 
    }

    return alertDiv;
  }

  function _removeAlert (alert) {
    window.clearTimeout(alert.timeout);
    container.removeChild(alert);
  }

  function _alertClickHandler (event) {
    _removeAlert(event.currentTarget);
  }

})();

alert-alert.css

#aa-container {
  position: fixed;
  top: 12px;
  right: 12px;
  z-index: 1001;
}

.aa-notification {
  width: 300px;
  min-height: 40px;
  padding: 12px;
  margin-bottom: 12px;
  background-color: white;
  border-left: 4px solid rgb(100, 100, 100);
  border-radius: 3px;
  cursor: pointer;
  box-shadow: 1px 1px 3px rgba(0, 0, 0, 0.6);
  transition: 0.3s linear all;
}

.aa-notification:hover {
  transform: scale(1.06);
  box-shadow: 3px 3px 4px gba(0, 0, 0, 0.6);
}

.aa-notification.info {
  border-left-color: rgb(100, 100, 200);
}

.aa-notification.success {
  border-left-color: rgb(100, 180, 100);
}

.aa-notification.warning {
  border-left-color: rgb(230, 215, 100);
}

.aa-notification.error {
  border-left-color: rgb(230, 100, 100);
}