DEV Community: alfg

I built the Postman for Video Engineers

alfg — Fri, 13 Mar 2026 13:18:07 +0000

I've worked in video engineering for the greater part of my career and one thing in common amongst many colleagues are the set of tools we use. ffmpeg, mp4box, gpac, bento4, ffprobe, mediainfo to name a few. Most of them are open source, some are commercial and some are web or desktop based. Some of them are scripts on top of these tools, and sometimes with text files with saved commands. Some of them have a 20+ year GUI.

This is in addition to our development stack and tooling.

These tools have no natural home together, so everyone builds their own patchwork. Why hasn't anyone built this yet?

What if we had a way to consolidate this type of workflow and tooling into a desktop application in the same respect Postman has for Web API developers? Or how JetBrains has for code?

That's why I made Video Commander.

The idea was simple: a familiar UI that wraps the tools we already know, a toolkit for video engineers that you can easily reach for when you need to inspect, encode, analyze, or package any media for testing or development.

Here's a preview of what it currently looks like:

If you're familiar with Postman, or any IDE layout, then you're probably already comfortable with the UI. You have a project sidebar to organize your files, multiple tab instances open, and a few tasks loaded up in the jobs queue.

You can load remote or local media files or streaming manifests.

Video Commander launches with a focus on inspection and analysis. You can dig into media tracks, metadata, and MP4/ISOBMFF box hierarchies with tree and graph views, the kind of structural detail you'd normally get by piping ffprobe and mp4box output into a terminal and reading JSON.

Track and sample analysis gives you timing and layout insight. For encoding and conversion, FFmpeg pipelines run through a jobs queue so you're not blocking your workspace or babysitting a terminal window.

Everything runs locally as a desktop app, keeping your media files on your machine.

It's not just about building a set of UI controls for tools, but also the workspace and integration to help empower these tools for productivity. The same reason you use Postman over curl, or VSCode over a text editor.

Video Commander is built on Tauri, using my own crate mp4box for media inspection. Bring your own FFmpeg or sideload it automatically within the app.

Today, Video Commander is in preview and available for macOS (Windows coming soon). Try it out and let me know what you think!

https://video-commander.com

Thanks for reading!

FFmpeg the Easy Way

alfg — Tue, 25 May 2021 18:55:00 +0000

FFmpeg has many simple and complex options, which can be intimidating at first. So I wanted to create a simple, web-based interface for generating common encoding operations for video and audio, inspired by HandBrake.

ffmpeg-commander is a web-based FFmpeg command generator written in JavaScript using the Vue Framework. It is open source and hosted on GitHub Pages.

https://alfg.github.io/ffmpeg-commander/

ffmpeg-commander supports generating most of the common operations for encoding video, such as the container, codec, video & audio settings, some basic filters and more.

Once the command is generated, you can easily copy the output to your clipboard or save it to the browser's local storage. A few presets are also available as an example.

I'm looking to expand on more general options and filters in the future.

`ffmpegd`

ffmpegd is an optional companion application that connects ffmpeg-commander to ffmpeg by providing a websocket server to send encode tasks and receive realtime progress updates back to the browser. This allows using ffmpeg-commander as a GUI for ffmpeg.

The goal is to provide a simple interface for sending FFmpeg tasks from the browser (and other supported clients in the future) to your local machine.

https://github.com/alfg/ffmpegd

ffmpegd is written in Go and implements the Gorilla Websockets library.

Please note, this is still considered experimental and a work-in-progress. Feel free to report any bugs at https://github.com/alfg/ffmpegd.

Hopefully you find these tools helpful for working with FFmpeg. ☺️

Thanks for reading!

Find me on GitHub at: https://github.com/alfg for more video-related projects!

FFmpeg for Instagram

alfg — Thu, 20 May 2021 18:01:51 +0000

Uploading high quality video to Instagram can result in lower quality video on posts and stories due to aggressive re-encoding to save on bandwidth and storage for users. You will notice this especially if uploading videos from GoPro or other devices captured in 4K or 1440p with larger file sizes. Although the video preview on Instagram looks high quality, the resulting output will be degraded drastically.

This guide describes various FFmpeg encoding presets that can be used to help improve your video quality output when uploading to Instagram. These settings can also be applied to other encoders, such as Handbrake.

Recommended Encoding Settings

Instagram does not provide an official list of encoding requirements. These settings are gathered from various sources and through trial and error. However, Instagram does provide some encoding requirements for IGTV.

Set your encoder to use these settings:

MP4 Container format
H.264 Video Codec
AAC Audio
3500kbps bitrate
30 FPS
60 seconds maximum in length
1080p 16:9 max (vertical or horizontal)

Also, if you inspect a story video using a tool like mediainfo, you'll see something like the following:

First video stream:
1 367 kb/s 640*1136 (0.563) at 30.000 FPS, AVC (Main@L3)(CABAC 4 Ref Frames)
64.0 kb/s 44.1 kHz, 1 channel, AAC LC

Stories without audio, don't have an audio stream within the container at all.

Optimize with FFmpeg

We can apply the settings above using FFmpeg. Here are some encoding profiles to try out.

`720p30 @ CRF 23`

Using CRF only without targetting a bitrate.

ffmpeg -i INPUT.MOV -vf scale=-2:720 -c:v libx264 -profile:v main -level:v 3.0 -x264-params scenecut=0:open_gop=0:min-keyint=72:keyint=72 -c:a aac -preset slow -crf 23 -r 30 -sn -f mp4 OUTPUT.mp4

`720p30 @ 3500k video bitrate + 256k audio bitrate`

Variable bitrate with a max of 3500k.

ffmpeg -i INPUT.MOV -vf scale=-2:720 -c:v libx264 -profile:v main -level:v 3.0 -x264-params scenecut=0:open_gop=0:min-keyint=72:keyint=72:ref=4 -c:a aac -crf 23 -maxrate 3500k -bufsize 3500k -r 30 -ar 44100 -b:a 256k -sn -f mp4 OUTPUT.mp4

`720p30 @ 3500k video bitrate + 256k audio bitrate - 2-pass encoding`

Using 2-pass encoding with a targetted bitrate of 3500k. This doesn't necessarily improve quality, but helps target a filesize more accurately.

ffmpeg -i INPUT.MOV -vf scale=-2:720 -c:v libx264 -profile:v main -level:v 3.0 -x264-params scenecut=0:open_gop=0:min-keyint=72:keyint=72:ref=4 -c:a aac -b:v 3500k -maxrate 3500k -bufsize 3500k -r 30 -ar 44100 -b:a 256k -pass 1 -sn -f mp4 NUL && \ 
 ffmpeg -i INPUT.MOV -vf scale=-2:720 -c:v libx264 -profile:v main -level:v 3.0 -x264-params scenecut=0:open_gop=0:min-keyint=72:keyint=72:ref=4 -c:a aac -b:v 3500k -maxrate 3500k -bufsize 3500k -r 30 -ar 44100 -b:a 256k -pass 2 OUTPUT.mp4

Try adjusting a few settings to get the optimal quality and filesize before uploading. Maybe try 1080p for IGTV, per their specification.

What do those options mean?

-i INPUT.MOV - Input movie file.
-vf scale=-2:720 - Video filter to scale to 720p while keeping aspect ratio.
-c:v libx264 - Use x264 video codec.
-profile:v main -level:v 3.1 - Mostly for H.264 compatibility for older devices. This seems to be the profile Apple devices export to.
-x264-params scenecut=0:open_gop=0:min-keyint=72:keyint=72:ref=4 - Some x264 codec options I use to improve video for streaming.
-c:a aac - Use AAC audio codec.
-preset slow - Slower presets will result in better quality. Feel free to also try slower and veryslow.
-crf 23 - Constant Rate Factor encoding mode that adjusts the file data rate up or down to achieve a selected quality level rather than a specific data rate.
-b:v 3500k - Targets video bitrate to 3500k.
-b:a 256k - Targets audio bitrate to 256k.
-ar 44100 - Sets audio sample rate to 44.1 kHz.
-r 30 - Sets the framerate to 30.
-f mp4 - Sets the MP4 container format. This is optional if you have .mp4 in the output filename.
OUTPUT.mp4 - The encoded output movie file.

Demo

Here's a before and after test I created using the above requirements. These were downloaded directly from Instagram after posting to compare results.

The source video file being a 3840x2160 @ 60fps (4K) file recorded from a GoPro Hero 8 at 567MB in size.

The optimized video encoded at 1280x720 @ 30fps using the 2-pass profile settings at 34.7MB in size.

Without optimizing video

With optimized video

Without optimizing as a story

Optimized video as a story

Notice the unoptimized videos have more jagged edge artifacts around the guard rails and curb, especially for video with a lot of motion.

The goal is basically to reduce your video to an optimal size and image quality so it doesn't get affected as much by Instagram's aggressive encoding.

Try adjusting some settings to find the best results for your use case and picture. Results will vary!

Thanks for reading!

Find me on GitHub at: https://github.com/alfg for more video-related projects!

Resources

FFmpeg + WebAssembly

alfg — Wed, 19 May 2021 18:20:22 +0000

FFmpeg is a powerful command line tool for handling video, audio and other multimedia files and streams. It is any video developer's utility for editing, transcoding, and remuxing virtually any format. It is developed in C and available for most platforms.

FFmpeg is not just a command line tool, though. It is powered by the FFmpeg libraries known as libav. These libraries enable FFmpeg to read, write and manipulate multimedia files. These libraries provide functionality for remuxing, encoding and decoding, filtering, scaling, colorspace conversion and device interfacing. You can use these libraries directly if you're writing an application in C/C++. There are also libav bindings available for most common languages.

What if you could use FFmpeg's libraries in the browser?

JavaScript in the browser is a different story. It is not designed to run system applications in the browser environment. So how can we run FFmpeg in the browser? WebAssembly!

WebAssembly (or Wasm) has been gaining popularity recently allowing us to run binary instructions in the browser, along with a compiler toolchain, Emscripten to help us build and port C/C++ code to Wasm.

This has already been done before. You can check out ffmpeg.wasm for running the FFmpeg CLI in the browser environment.

However, this guide's focus will step through the process on building FFmpeg's libav libraries for use in the browser via Web Assembly, rather than the FFmpeg CLI application.

Why? 🤔

FFmpeg in the browser won't have the greatest performance compared to running it natively on a system that can take advantage of threaded processing and hardware acceleration.

Normally, you would just build a back-end that wraps FFmpeg or libav and relay to the front-end to provide results.

However, we can still take advantage of the vast features of FFmpeg's libraries such as parsing format and codec information, decoding frames, applying filters and more. Imagine if we can do this purely in JavaScript, within the browser environment on a static webpage.

Hello World in `libav`

Let's start by writing a simple program in C to print out basic media information using libav. Let's name it mp4info.c.

If you are not familiar with libav, ffmpeg-libav-tutorial is a great introduction.

#include <stdio.h>
#include <libavformat/avformat.h>

static AVFormatContext *fmt_ctx;

int main(int argc, const char *argv[])
{
  if (argc < 2) {
    printf("Please specify a media file.\n");
    return -1;
  }

  // Open the file and read header.
  int ret;
  if ((ret = avformat_open_input(&fmt_ctx, argv[1], NULL, NULL)) < 0) {
      printf("%s", av_err2str(ret));
      return ret;
  }

  // Read container data.
  printf("format: %s, duration: %ld us, streams: %d\n",
    fmt_ctx->iformat->name,
    fmt_ctx->duration,
    fmt_ctx->nb_streams);

  return 0;
}

This is a basic program that takes a media file as an argument and prints out the format name, duration and stream count.

Compile and run the program:

Download Tears of Steel 10s Example

gcc src/mp4info.c -lavformat -lavutil -o bin/mp4info
./bin/mp4info tears-of-steel-10s.mp4

gcc, ffmpeg and ffmpeg-dev are required to build.

You should get the following output.

format: mov,mp4,m4a,3gp,3g2,mj2, duration: 10000000 us, streams: 2

Great! Now we can move onto the next step using Emscripten to compile FFmpeg, and build a wrapper for use within a JavaScript environment.

Emscripten

Emscripten is a compiler toolchain for for WebAssembly. We will use Docker and emscripten to build FFmpeg's libav and our custom wrapper code to Wasm. Let's start with FFmpeg first.

Compiling FFmpeg to Web Assembly

In our Dockerfile, we will use the base emscripten emsdk to build FFmpeg from source, along with the latest stable libx264 version.

Since we are compiling to WebAssembly, we will make a minimal build by disabling all programs and only enabling the features we are likely to use. This will help keep the binary size small and optimal.

FROM emscripten/emsdk:2.0.16 as build

ARG FFMPEG_VERSION=4.3.2
ARG X264_VERSION=20170226-2245-stable

ARG PREFIX=/opt/ffmpeg
ARG MAKEFLAGS="-j4"

# Build dependencies.
RUN apt-get update && apt-get install -y autoconf libtool build-essential

# Download and build x264.
RUN cd /tmp && \
  wget https://download.videolan.org/pub/videolan/x264/snapshots/x264-snapshot-${X264_VERSION}.tar.bz2 && \
  tar xvfj x264-snapshot-${X264_VERSION}.tar.bz2

RUN cd /tmp/x264-snapshot-${X264_VERSION} && \
  emconfigure ./configure \
  --prefix=${PREFIX} \
  --host=i686-gnu \
  --enable-static \
  --disable-cli \
  --disable-asm \
  --extra-cflags="-s USE_PTHREADS=1"

RUN cd /tmp/x264-snapshot-${X264_VERSION} && \
  emmake make && emmake make install 

# Download ffmpeg release source.
RUN cd /tmp/ && \
  wget http://ffmpeg.org/releases/ffmpeg-${FFMPEG_VERSION}.tar.gz && \
  tar zxf ffmpeg-${FFMPEG_VERSION}.tar.gz && rm ffmpeg-${FFMPEG_VERSION}.tar.gz

ARG CFLAGS="-s USE_PTHREADS=1 -O3 -I${PREFIX}/include"
ARG LDFLAGS="$CFLAGS -L${PREFIX}/lib -s INITIAL_MEMORY=33554432"

# Configure and build FFmpeg with emscripten.
# Disable all programs and only enable features we will use.
# https://github.com/FFmpeg/FFmpeg/blob/master/configure
RUN cd /tmp/ffmpeg-${FFMPEG_VERSION} && \
  emconfigure ./configure \
  --prefix=${PREFIX} \
  --target-os=none \
  --arch=x86_32 \
  --enable-cross-compile \
  --disable-debug \
  --disable-x86asm \
  --disable-inline-asm \
  --disable-stripping \
  --disable-programs \
  --disable-doc \
  --disable-all \
  --enable-avcodec \
  --enable-avformat \
  --enable-avfilter \
  --enable-avdevice \
  --enable-avutil \
  --enable-swresample \
  --enable-postproc \
  --enable-swscale \
  --enable-filters \
  --enable-protocol=file \
  --enable-decoder=h264,aac,pcm_s16le \
  --enable-demuxer=mov,matroska \
  --enable-muxer=mp4 \
  --enable-gpl \
  --enable-libx264 \
  --extra-cflags="$CFLAGS" \
  --extra-cxxflags="$CFLAGS" \
  --extra-ldflags="$LDFLAGS" \
  --nm="llvm-nm -g" \
  --ar=emar \
  --as=llvm-as \
  --ranlib=llvm-ranlib \
  --cc=emcc \
  --cxx=em++ \
  --objcc=emcc \
  --dep-cc=emcc

RUN cd /tmp/ffmpeg-${FFMPEG_VERSION} && \
  emmake make -j4 && \
  emmake make install

docker build -t mp4info .

This will build and install ffmpeg libraries to /opt/ffmpeg. However, we still need to write our wrapper using the Hello World example we wrote earlier.

Writing the Wrapper

Now that we can build the FFmpeg libraries to Wasm, we need to create a wrapper that integrates the functionality of emscipten so we can import and use in JavaScript within the browser.

We will write our custom wrapper in C++ to take advantage of Embind.

From the documentation:

Embind is used to bind C++ functions and classes to JavaScript, so that the compiled code can be used in a natural way by “normal” JavaScript.

This will allow us to easily create bindings from C++ constructs. So let's get started!

Update our Hello World example to C++:

mp4info-wrapper.cpp:

#include <emscripten.h>
#include <emscripten/bind.h>
#include <inttypes.h>

#include <string>
#include <vector>

using namespace emscripten;

extern "C"
{
#include <libavformat/avformat.h>
};

typedef struct Response
{
  std::string format;
  int duration;
  int streams;
} Response;

static AVFormatContext *fmt_ctx;

Response run(std::string filename)
{
  // Open the file and read header.
  int ret;
  if ((ret = avformat_open_input(&fmt_ctx, filename.c_str(), NULL, NULL)) < 0)
  {
    printf("%s", av_err2str(ret));
  }

  // Read container data.
  printf("format: %s, duration: %lld us, streams: %d\n",
         fmt_ctx->iformat->name,
         fmt_ctx->duration,
         fmt_ctx->nb_streams);

  // Initialize response struct with format data.
  Response r = {
      .format = fmt_ctx->iformat->name,
      .duration = (int)fmt_ctx->duration,
      .streams = (int)fmt_ctx->nb_streams,
  };

  return r;
}

EMSCRIPTEN_BINDINGS(structs)
{
  emscripten::value_object<Response>("Response")
      .field("format", &Response::format)
      .field("duration", &Response::duration)
      .field("streams", &Response::streams);
  function("run", &run);
}

Some notable changes:

C to C++
Included the emscripten headers
Created a Response typedef
Updated main to run, taking a filename argument and returns a Response struct with format, duration and streams.
Added EMSCRIPTEN_BINDINGS to export the run function with the Response type bindings.

Add a Makefile for emcc to build the wrapper:

dist/mp4info.wasm.js:
    mkdir -p dist && \
    emcc --bind \
    -O3 \
    -L/opt/ffmpeg/lib \
    -I/opt/ffmpeg/include/ \
    -s EXTRA_EXPORTED_RUNTIME_METHODS="[FS, cwrap, ccall, getValue, setValue, writeAsciiToMemory]" \
    -s INITIAL_MEMORY=268435456 \
    -s ASSERTIONS=1 \
    -s STACK_OVERFLOW_CHECK=2 \
    -s PTHREAD_POOL_SIZE_STRICT=2 \
    -s ALLOW_MEMORY_GROWTH=1 \
    -lavcodec -lavformat -lavfilter -lavdevice -lswresample -lpostproc -lswscale -lavutil -lm \
    -pthread \
    -lworkerfs.js \
    -o dist/mp4info.js \
    src/mp4info-wrapper.cpp

And add the following lines to our Dockerfile:

COPY ./src/mp4info-wrapper.cpp /build/src/mp4info-wrapper.cpp
COPY ./Makefile /build/Makefile

WORKDIR /build

Now let's build via Docker with a mounted volume. This is so we can easily build the artifact from the container to our host:

docker run -v ${pwd}:/build -it mp4info make

This will build the following files into dist/:

dist
├───mp4info.js
├───mp4info.wasm
└───mp4info.worker.js

Your WebAssembly artifacts are now ready for use within JavaScript! 🎉

Writing the JavaScript

In this next step, we'll be creating a static webpage using just HTML and JavaScript.

Let's create a basic index.html page with a file input and an empty results div:

<!DOCTYPE html>
<html>
<head>
    <meta charset='utf-8'>
    <meta http-equiv='X-UA-Compatible' content='IE=edge'>
    <title>FFmpeg WebAssembly Example</title>
    <meta name='viewport' content='width=device-width, initial-scale=1'>
</head>
<body>
    <div>
        <input id="file" type="file">
    </div>
    <div id="results"></div>
</body>
</html>

Since our mp4info wrapper is compiled from synchronous C++ code, it can block the browser's main thread when executing on larger files. This can lead to the browser's UI locking up and a bad experience for the user.

However, we can take advantage of Web Workers to handle background tasks and simply relay the response to the main thread to prevent any blocking code.

Let's create a new worker.js file with the following:

// Run this script as a Web Worker so it doesn't block the
// browser's main thread.
// See: index.html.
onmessage = (e) => {
    const file = e.data[0];
    let data;

    // Create and mount FS work directory.
    if (!FS.analyzePath('/work').exists) {
        FS.mkdir('/work');
    }
    FS.mount(WORKERFS, { files: [file] }, '/work');

    // Run the Wasm function we exported.
    const info = Module.run('/work/' + file.name);
    console.log(info);

    // Post message back to main thread.
    postMessage(info);

    // Unmount the work directory.
    FS.unmount('/work');
}

// Import the Wasm loader generated from our Emscripten build.
self.importScripts('mp4info.js');

This contains a message listener to create a virtual file system using the Emscripten Filesystem API, runs our Wasm module code, and send the results back using postMessage.

So let's also add the following code to your index.html right before the </html> line:

<script>
    // Create a worker for running Wasm code without blocking main thread.
    const worker = new Worker('worker.js');

    const input = document.querySelector('input');
    input.addEventListener('change', onFileChange);

    // Listen for messages back from worker and render to DOM.
    worker.onmessage = (e) => {
        const data = e.data;
        const results = document.getElementById('results');
        const ul = document.createElement('ul');
        const li = document.createElement('li');
        li.textContent = "format: " + data.format;
        const li2 = document.createElement('li');
        li2.textContent = "duration: " + data.duration;
        const li3 = document.createElement('li');
        li3.textContent = "streams: " + data.streams;
        ul.appendChild(li);
        ul.appendChild(li2);
        ul.appendChild(li3);
        results.appendChild(ul);
    }

    // Send file to worker.
    function onFileChange(event) {
        const file = input.files[0];
        worker.postMessage([ file ]);
    }

</script>

This creates a listener on the input form element and sends it to the Web Worker. It also creates a listener for messages relayed back from worker.js so we can render the results back to the DOM.

Let's add our index.html, worker.js and Wasm related files into a new directory www:

www
├───index.html
├───worker.js
├───mp4info.js
├───mp4info.wasm
└───mp4info.worker.js

Before we load up index.html into the browser, our Wasm build uses PTHREADS, which raises a security issue with some browsers. From the Emscripten documentation:

Browsers are currently shipping SharedArrayBuffer gated behind Cross Origin Opener Policy (COOP) and Cross Origin Embedder Policy (COEP) headers. Pthreads code will not work in deployed environment unless these headers are correctly set.

So we need to include the proper response headers when serving the webpage:

Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp

We can do this by running simple NodeJS server using Express, easily:

const express = require('express');
const app = express();
const port = process.env.PORT || 8080;

app.use((req, res, next) => {
    // CORS headers.
    res.append('Access-Control-Allow-Origin', ['*']);
    res.append('Access-Control-Allow-Methods', 'GET,PUT,POST,DELETE');
    res.append('Access-Control-Allow-Headers', 'Content-Type');

    // Required headers for SharedArrayBuffer.
    // See: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/SharedArrayBuffer#security_requirements
    res.append('Cross-Origin-Opener-Policy', 'same-origin');
    res.append('Cross-Origin-Embedder-Policy', 'require-corp');
    next();
});
app.use(express.static(__dirname));
app.listen(port);

However, keep in mind you will need to add these response headers if deploying to any static webpage provider.

Demo Time!

Run the server (requires NodeJS):

npm install express
node server.js

Load up http://localhost:8080/www in the browser and select an MP4 file.

And that's how you run FFmpeg's libav in the browser! ⚡

This just scratches the surface into the capabilities of FFmpeg + WebAssembly. You can use libav to do more than just read format details. Check out more libav examples and try porting them to WebAssembly.

Thanks for reading!

Check out https://github.com/alfg/ffmpeg-webassembly-example for the full demo files in this guide.

I also have a more advanced example of using FFProbe via Wasm:
https://github.com/alfg/ffprobe-wasm

Find me on GitHub at: https://github.com/alfg

References and Resources

Happy Hacking! 🎥

A Quick Dive Into MP4

alfg — Fri, 30 Apr 2021 17:15:41 +0000

What is an MP4? We all know it as a file format for playing video with sound. It's used for streaming video by Netflix, YouTube, Instagram, and also for capturing video on your iPhone, but how does it work? How is it used? What is the byte structure? What is a container?

This guide is an introduction and a quick dive into the MP4 file format, also known as the ISO Base Media File Format (ISO-BMFF MPEG-4 Part 14). Fancy name, I know.

I won't go into the playback details in this guide, but more of the MP4 byte format commonly known as the MP4 Box Structure.

Introduction

The MPEG-4 Part 14 (MP4) is one of the most common container formats for video and has an extension of .mp4. You may already know of other container formats, such as wav, mov, mp3 or more recently webm. A container just "contains" the video or audio track, or both. It can also support embedded subtitle tracks too.

MP4 is an extension of the ISO Base Media File Format (ISOBMFF, MPEG-4 Part 12), which is a format designed to contain timed media information.

The ISO-BMFF format is directly based on QuickTime, therefore the MP4 is essentially identical to the QuickTime file format.

     MPEG-4 Part 14 - MP4 File Format
 ┌──────────────────────────────────────┐
 ├────────────────────┐                 │
 │  ISO Base Media    │                 │
 │    File Format     │  MP4 Extension  │
 │  (MPEG-4 Part 12)  │                 │
 └────────────────────┴─────────────────┘

In order to fully understand the MP4 structure, you'll need to obtain a copy of the ISO documents:

14496-12 – MPEG-4 Part 12
14496-14 - MPEG-4 Part 14

A Google search should result in a few resources to get a copy of the PDF.

What's in a container?

Since MP4 is a container format, it doesn't actually handle the decoding of the video and audio streams, it just contains them as tracks along with their metadata.

A container can store some of the following information:

General metadata such as file type and compatibility
Video, audio and subtitle tracks and codec details
Metadata: duration, timescale, bitrate, width/height, etc
Progressive and fragmented metadata details
A series of video frames or audio samples known as "Sample Data"

This is all the information a player needs to decode and play the content.

At a high-level, this is what an MP4 structure typically looks like:

video.mp4
├───general file metadata
├───movie data
├───tracks
│   ├───video
│   │   ├───video metadata
│   │   └───video sample data
│   └───audio
│       ├───audio metadata
│       └───audio sample data
└───more metadata

Movie Boxes

The MP4 byte structure is composed of a series of boxes, also known as "atoms", according to the QuickTime specification. Each box describes and contains data to build the MP4 container format.

Boxes typically have a four letter name, also known as a FourCC. This is the shortened version of the full box name, enough to fit into 4 bytes. This is important for when you are reading and writing boxes into or from the byte format.

Before we jump into the byte structure details, here is a more technical view of the MP4 box tree, compared to the high-level view above:

video.mp4
├───ftyp -------------------> FileType Box
├───mdat -------------------> Movie Data Box
├───moov -------------------> Movie Boxes
│   ├───trak ---------------> Track Box
│   │   ├─── tkhd ----------> Track Header
│   │   └─── mdia ----------> Media Box
│   │        └─── ...
│   └───trak
│   │   ├─── tkhd ----------> Track Header
│   │   └─── mdia ----------> Media Box
│   │        └─── ...
└───udta -------------------> Userdata Box

This is just a simplified view. However, there are many more boxes defined in the MP4 specification.

What's in the box?

An MP4 "Box" contains just enough information to read and parse the box name, size and data.

Each of these boxes have a different purpose, containing a bit of information and details on a specific piece of data. Some boxes describe the file type, and others can describe codec detail, picture resolution, frame rate, duration, sample sizes and more. There's also boxes containing the encoded video and audio data too.

A box typically contains the following base information:

Size of the box (in bytes)
Box Name (FourCC)
Box Data

┌─────────────────────┐
|      Box Header     |
| Size (4) | Type (4) | Box Header = 8 Bytes
| --------------------|
|     Box Data (N)    | Box Data = N Bytes
└─────────────────────┘
           └─────────── Box Size = 8 + N bytes

This is just enough information we need to know how to parse a box, along with the MP4 specification document to understand the box fields.

Parse a Box

So let's parse our first box!

As mentioned above, the first 8 bytes of each box is known as the "Box Header", where the first 4 bytes are the size of the box, and the next 4 bytes are the box name. These are the two values you need to know to iterate and parse each box, byte by byte.

Here's a box header struct for example:

type Box struct {
  Size    int32
  Name    string
}

Reading the box data from each atom requires the box size, name and byte structure of each box you are parsing. You can refer to the MPEG-4 Part 14 specification for the byte structure of each known box, or just refer to some existing MP4 parsing open-source code.

According to the specification, the ftyp box has the following structure:

aligned(8) class FileTypeBox
  extends Box(‘ftyp’) {
  unsigned int(32) major_brand;
  unsigned int(32) minor_version;
  unsigned int(32) compatible_brands[]; // to end of the box
}

major_brand – is a brand identifier
minor_version – is an informative integer for the minor version of the major brand
compatible_brands – is a list, to the end of the box, of brands

For example, reading the FileTypeBox (ftyp) would look something like the following (in Golang):

type FtypBox struct {
  *Box
  MajorBrand       string
  MinorVersion     uint32
  CompatibleBrands []string
}

func (b *FtypBox) parse() {
  data := b.ReadBoxData() // Read box header.
  b.MajorBrand = string(data[0:4])
  b.MinorVersion = binary.BigEndian.Uint32(data[4:8])
  if len(data) > 8 {
    for i := 8; i < len(data); i += 4 {
      b.CompatibleBrands = append(b.CompatibleBrands, string(data[i:i+4]))
    }
  }
}

Going over the above:

Reading the first 4 bytes of the box header as an unsigned 32 bit integer (big endian) gives us the box size: 32 bytes.
The next 4 bytes gives us: 0x66747970 in hexidecimal, or ftyp as a string.
Next 4 bytes gives us the Major Brand: 0x69736F6D in hexidecimal, or isom as a string.
Next 4 bytes gives us the Minor Version: 512
The next 16 bytes, read as uint32be (into a string) at a time gives us an array of compatible brands: isom, iso2, avc1, and mp41.
We have read a total of 32 bytes as defined in the box header.

See a minimal example:
https://gist.github.com/alfg/7375aee32fda490de4bf62fbced49d2e#file-mp4_example-go

$ go run mp4.go tears-of-steel.mp4
ftyp.name:  ftyp
ftyp.major_brand:  isom
ftyp.minor_version:  512
ftyp.compatible_brands:  [isom iso2 avc1 mp41]

If you were to open the mp4 into a hex editor, it would look something like this for the ftyp box:

0x00 00 00 00 20 66 74 79 70 | 69 73 6F 6D 00 00 02 00 ... ftypisom....
0x10 69 73 6F 6D 69 73 6F 32 | 61 76 63 31 6D 70 34 31 isomiso2avc1mp41

Next Box!

Now that we've read the ftyp box, we can move on to the next box header, which happens to be the moov box:

type MoovBox struct {
  *Box
  Mvhd  *MvhdBox
}

func (b *MoovBox) parse() {
  boxes := readBoxes(b.Reader, b.Start+BoxHeaderSize, b.Size-BoxHeaderSize)

  for _, box := range boxes {
    switch box.Name {
    case "mvhd":
    b.Mvhd = &MvhdBox{Box: box}
    b.Mvhd.parse()
  }
}

The moov box contains a nested mvhd box, so we also need to define mvhd too:

type MvhdBox struct {
  *Box
  Flags            uint32
  Version          uint8
  CreationTime     uint32
  ModificationTime uint32
  Timescale        uint32
  Duration         uint32
  Rate             Fixed32
  Volume           Fixed16
}

func (b *MvhdBox) parse() {
  data := b.ReadBoxData()
  b.Version = data[0]
  b.Timescale = binary.BigEndian.Uint32(data[12:16])
  b.Duration = binary.BigEndian.Uint32(data[16:20])
  b.Rate = fixed32(data[20:24])
  b.Volume = fixed16(data[24:26])
}

See the example with moov and mvhd box included:
https://gist.github.com/alfg/7375aee32fda490de4bf62fbced49d2e#file-mp4_example_2-go

$ go run mp4.go tears-of-steel.mp4
ftyp.name:  ftyp
ftyp.major_brand:  isom
ftyp.minor_version:  512
ftyp.compatible_brands:  [isom iso2 avc1 mp41]
moov.name:  moov 3170
moov.mvhd.name:  mvhd
moov.mvhd.version:  0
moov.mvhd.volume:  1

EOF

Now that we've parsed 3 boxes, hopefully you have an idea on how to implement more. The process is iterative when using a reader:

Read the box header, containing the box size and name.
Refer to the specification to read and/or skip fields.
Skip any remaining bytes left in the box size.
Read the next box (or skip).

Some things to keep in mind:

Some boxes have multiple versions, and therefore can differ in the struct and overall size of the box.
You can skip properties, but the reader must know how many bytes to skip.
There are various MP4 specifications beyond MPEG-4 Part 14 as more boxes are being added throughout the years.
Fragmented MP4 files (fMP4) are segmented as a series of moof and mdat boxes. This is more common and optimial for streaming delivery. I'll cover this in a future post.

Thanks for reading!

For a more complete example of reading MP4 boxes in Go, check out:
https://github.com/alfg/mp4

I also have a more advanced MP4 reader and writer in Rust:
https://github.com/alfg/mp4rs

I highly suggest some of the following tools for inspecting MP4 files:

Find me on GitHub at: https://github.com/alfg

Happy Hacking! 🎥

DEV Community: alfg

I built the Postman for Video Engineers

FFmpeg the Easy Way

ffmpegd

Thanks for reading!

FFmpeg for Instagram

Recommended Encoding Settings

Optimize with FFmpeg

720p30 @ CRF 23

720p30 @ 3500k video bitrate + 256k audio bitrate

720p30 @ 3500k video bitrate + 256k audio bitrate - 2-pass encoding

What do those options mean?

Demo

Without optimizing video

With optimized video

Without optimizing as a story

Optimized video as a story

Thanks for reading!

Resources

FFmpeg + WebAssembly

Why? 🤔

Hello World in libav

Emscripten

Compiling FFmpeg to Web Assembly

Writing the Wrapper

Writing the JavaScript

Demo Time!

Thanks for reading!

References and Resources

A Quick Dive Into MP4

Introduction

What's in a container?

Movie Boxes

What's in the box?

Parse a Box

Next Box!

EOF

Some things to keep in mind:

Thanks for reading!

References and Resources

`ffmpegd`

`720p30 @ CRF 23`

`720p30 @ 3500k video bitrate + 256k audio bitrate`

`720p30 @ 3500k video bitrate + 256k audio bitrate - 2-pass encoding`

Hello World in `libav`