DEV Community: Alan Allard

Exploring video generators in FFMPEG...

Alan Allard — Fri, 24 Feb 2023 11:05:04 +0000

...in which the author creates entirely inappropriate amounts of eye candy by testing the ranges of most of the video generation options within ffmpeg - but also manages, as luck would have it, to identify some valuable testing tools in the process...

If you haven't started following Eyevinn's ffmpeg of the day series on Instagram (of which I am the author), perhaps you should! Anyway, the very first command that I looked at in the series was an example of the cellauto filter in ffmpeg. While trawling through the long, long list of ffmpeg filters available for audio and video, I noticed that video generators have their own quite sizeable section in the list. There are so many of these generators available, but many of them may turn out to be useful for various purposes. Not only are there many separate generators in the list, several of them are extensible with code or plugins in a specific format. In this article, I will try and get a notion of what all of these generators can do, by examining their various parameter ranges.

A caveat, first. I am on macOS, so I will be focusing on the filters that are readily available for me - not least of all the coreimagesrc generator with it's considerable list of built-in effects.

In fact, coreimagesrc seems like a pretty good place to start...

The `coreimagesrc` generator

Several of these generators we will look at are also available as more general filters, with some slight variation in options. Generators are more specific in that they are designed to be first in the filter chain, as inputs. This coreimagesrc has a more general counterpart in the filter coreimage, for example.

The first thing we want to do is identify which particular generators are available on our Mac:

ffmpeg -f lavfi -i coreimagesrc=list_generators=true null

which will give you a long list of generators with all their parameters. Let's grep it (notice the little pirouette of piping everything from stderr to stdout, which is necessary in order to pipe the ffmpeg output to grep successfully). We may as well cut the first part of the line too for a nice tidy list:

ffmpeg -f lavfi -i coreimagesrc=list_generators=true 2>&1 | grep Filter: | cut -c 32-

which gives us the following list (in my case):

 Filter: CIAttributedTextImageGenerator
 Filter: CIAztecCodeGenerator
 Filter: CIBarcodeGenerator
 Filter: CICheckerboardGenerator
 Filter: CICode128BarcodeGenerator
 Filter: CIConstantColorGenerator
 Filter: CILenticularHaloGenerator
 Filter: CIMeshGenerator
 Filter: CIPDF417BarcodeGenerator
 Filter: CIQRCodeGenerator
 Filter: CIRandomGenerator
 Filter: CIRoundedRectangleGenerator
 Filter: CIStarShineGenerator
 Filter: CIStripesGenerator
 Filter: CISunbeamsGenerator
 Filter: CITextImageGenerator

Quite a mixed bag. How about a lenticular halo generator?
With many filters and/or generators in ffmpeg, you can just give the filter name and the default parameters will be used. That's only partially true with the coreimage generators; there are default parameters available, but all of these generators have one or more parameters that have no default, so you will get an error if you try something like:

ffmpeg -f lavfi -i \
"coreimagesrc=s=600x600:\
filter=CILenticularHaloGenerator" \
-t 30 -pix_fmt yuv420p output.mov

which will give:

[coreimagesrc @ 0x7fa3fc205a00] Parsing of filters failed.
[lavfi @ 0x7fa3fb7045c0] Error initializing filter 'coreimagesrc' with args 's=600x600:filter=CIMeshGenerator'
coreimagesrc=s=600x600:filter=CIMeshGenerator: Input/output error

Let's examine the parameters of the lenticular halo generator in the list from earlier (we filtered them out last time):

ffmpeg -f lavfi -i coreimagesrc=list_generators=true 2>&1 | grep -A 8 'Filter: CILenticularHaloGenerator' | cut -c 32-

giving

Option: inputCenter     [CIVector]
Option: inputColor      [CIColor]
Option: inputHaloRadius [NSNumber]      [0 1000][70]
Option: inputHaloWidth  [NSNumber]      [0 300][87]
Option: inputHaloOverlap        [NSNumber]      [0 1][0.77]
Option: inputStriationStrength  [NSNumber]      [0 3][0.5]
Option: inputStriationContrast  [NSNumber]      [0 5][1]
Option: inputTime       [NSNumber]      [0 1][0]

which is very useful in that we can see which parameters have defaults (the second pair of square brackets are the defaults). So at the very least we need a CIVector and a CIColor. Next problem, how do we represent a CIVector and CIColor as parameters to the generator within an ffmpeg command? It took me a while to work this out, but with the help of the very sparse ffmpeg documentation, the ffmpeg source code and a quick look at the CoreImage apple documentation, I came up with the answer, as follows:

ffmpeg -f lavfi -i \
"coreimagesrc=s=600x600:\
filter=CILenticularHaloGenerator\
@inputCenter=200.0 200.0\
@inputColor=0.0 0.0 1.0" \
-t 5 -pix_fmt yuv420p output.mov

Which gives us five seconds of a rather nice sort of focusing iris effect:

Ok, how about a star shine generator:

ffmpeg -f lavfi -i \
"coreimagesrc=s=600x600:\
filter=CIStarShineGenerator\
@inputCenter=300.0 300.0\
@inputRadius=25\
@inputCrossAngle=0\
@inputColor=1.0 0.0 1.0" \
-t 5 -pix_fmt yuv420p starshine.mov

which yields:

The `gradients` generator

There are several more of these coreimagesrc generators to explore, maybe we'll come back to them in a bit. As there are so many more types of ffmpeg generator, let's move on for the moment. Let's examine the gradients generator next. This is a powerful filter that can generate several different types of gradient, with random or designated colours and even rotate them if you wish (merely a convenience of course, as it's pretty trivial to add a rotation filter to any filter graph). An example:

ffmpeg -f lavfi -i \
gradients=duration=5\
:nb_colors=5:x0=320:y0=240\
:type=spiral:speed=0.1 \
-pix_fmt yuv420p gradients.mov

Here, in fact, it becomes apparent that rotation is very much meant to be part of the gradient effect - the lowest speed is not 0 but 1e-05. So in fact it always rotates even at the lowest setting, just very slowly. You could always use the rotate filter in the opposite direction to keep it stationary of course...for that you would need to know what exactly that speed parameter represents. Here's a clue: in the ffmpeg source code, video sources are sensibly prefixed with vsrc_, so here we can take a look in vsrc_gradients.c and see that the speed parameter is used to calculate an angle as follows: float angle = fmodf(s->pts * s->speed, 2.f * M_PI); So basically the angle is in radians and is a function of pts. We'll leave that topic there this time round, just as a little note on how to compensate for the occasional shortfalls in the documentation by taking a quick look at the source code. Anyway, this is what we got from the last command:

Certainly pleasing to the eye, but the back and forth partial rotation is a bit odd. I haven't yet found a combination of gradients parameters to make a spiral gradient rotate around its centre (and I welcome suggestions), but I did notice that setting parameters x1 and y1 to the same as x0 and y0is a handy way to stop all rotation. After that we can simply apply the rotate filter instead, which looks...pretty good, I suppose:

When I'm going through all the options of a filter or generator to get a feel for them, I often want to see some kind of overview of all the options together. In those cases, I typically reach for javascript and build an ffmpeg command in that way, one that results in either a grid showing all the different options or one option after another in a concatenated video. For example, here is a grid of all four gradient types in the gradients generator. In this case, some of the default options are used, so the colours are randomly chosen and will vary each time the video is generated:

Take a look at the output ffmpeg command to understand why you might like to offload things like this to a custom command! :

ffmpeg -y -filter_complex \
"gradients=duration=10:type=spiral[out0];\
gradients=duration=10:type=radial[out1];\
gradients=duration=10:type=linear[out2];\
gradients=duration=10:type=circular[out3];\
[out0]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=spiral:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out0];\
[out1]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=radial:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out1];\
[out2]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=linear:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out2];\
[out3]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=circular:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out3];\
[text_out0][text_out1][text_out2][text_out3]xstack=inputs=4:grid=2x2" \
-t 10 allGradients.mp4

There is clearly a whole load of repetition, so this should be fairly easy to build and parameterise. Essentially this will all just be string building so we won't need to use any particular libraries for most of this script. We will need a way to call ffmpeg though - and ffmpeg will need to be present too, of course. To call a CLI command we can use the package commander.

We can see from a glance at the above command that we need:

The start line: ffmpeg -y -filter_complex
A line for each gradient type with a uniquely-named output at the end: gradients=duration=10:type=spiral[out0];\
A line for each text label with the matching inputs from 2. and uniquely-named outputs:

[out0]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=spiral:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out0];\

An xstack of the appropriate size that collects all of the uniquely-named drawtext-outputs as inputs to it.

Apart from that, there will be a few syntax issues to deal with as regards adding line ends, quotation marks and then building the final command. When it's built we can call exec() from commander to invoke ffmpeg. I tend to build in line breaks to the final commands so that they are easy to debug in the console.

Without further ado here is the code for that:

// gradientsDemo.js

const { exec } = require("child_process");

// Set up variables used to construct lines
const allGradientsTypes = [
    'spiral',
    'radial',
    'linear',
    'circular',
];

const TEXT_FONT = '\'fonts/crystal-radio-kit/crystal radio kit.ttf\'';
const TEXT_POS = 'x=(w-text_w)/2:y=(h-text_h)/10';
const TEXT_COLOR = 'black';
const FONT_SIZE = 35;
const TIME = 10;
// Setup lines
let filterLinesStart= "ffmpeg -y -filter_complex \\";
let filterLinesGenerator = ""; // generator lines
let filterLinesText = ""; // text lines
let filterLinesXStackInputs = "";
let filterLinesStacker = "";

let filterLinesLast = ` -t ${TIME} -an -pix_fmt yuv420p gradients.mp4`;

// Loop through lines
allGradientsTypes.forEach((gradient, index, gradients) => {
    filterLinesGenerator += `gradients=duration=${TIME}:type=${gradient}[out${index}];`;
    filterLinesText += `[out${index}]drawtext=fontfile=${TEXT_FONT}\
:text=${gradient}:fontsize=${FONT_SIZE}:${TEXT_POS}:fontcolor=${TEXT_COLOR}[text_out${index}];`;
    filterLinesXStackInputs += `[text_out${index}]`;

    if (Object.is(gradients.length - 1, index)) {
        // Finish up lines
        filterLinesGenerator += `\\`;
        filterLinesText += '\\';
        filterLinesStacker = `xstack=inputs=${allGradientsTypes.length}:grid=${2}x${2}`;
        filterLinesXStackInputs += ``;
    } else {
        filterLinesGenerator += `\\\n`;
        filterLinesText += `\\\n`;
    }
});

// Build command...
const command = `${filterLinesStart}
"${filterLinesGenerator}
${filterLinesText}
${filterLinesXStackInputs}${filterLinesStacker}"\\
${filterLinesLast}`;

console.info("Command is: ", command);

exec(command, (error, stdout, stderr) => {
    if (error) {
        console.log(`error: ${error.message}`);
        return;
    }

    console.log(`stdout: ${stdout}`);

    if (stderr) {
        console.log(`stderr: ${stderr}`);
    }
    console.info(`Generated a file with ${allGradientsTypes.length} gradient demos in a ${2} by ${2} grid`);
    return;
});

And as we get completely different colours each time, let's test that code and produce another instance of the video:

Already at this stage we are able to alter some of the text properties (as you have probably noticed you will want to have a path to the font in question on your machine). It's now fairly easy to add other parameters. For example, if the intention is to use these generated videos for analytic purposes, you may want to understand how the colours look for each of the gradients, so why not make the colours settable. By default, gradients sets only 2 random colours, but if we pass in the colour list as an array we can set the colour count parameter nb_colors too.

// gradientsDemo2.js

const { exec } = require("child_process");

// Set up variables used to construct lines
const allGradientsTypes = [
    'spiral',
    'radial',
    'linear',
    'circular',
];

// NEW - define the colours
const colours = [
    'blue',
    'red',
    'yellow'
];

const TEXT_FONT = '\'fonts/crystal-radio-kit/crystal radio kit.ttf\'';
const TEXT_POS = 'x=(w-text_w)/2:y=(h-text_h)/10';
const TEXT_COLOR = 'black';
const FONT_SIZE = 35;
const TIME = 10;
// Setup lines
let filterLinesStart= "ffmpeg -y -filter_complex \\";
let filterLinesGenerator = ""; // generator lines
let filterLinesText = ""; // text lines
let filterLinesXStackInputs = "";
let filterLinesStacker = "";

let filterLinesLast = ` -t ${TIME} -an -pix_fmt yuv420p gradients2.mp4`;

// NEW - Build colours list
let colourParams = "";
colours.forEach((colour, index, colours) => {
    colourParams += `:c${index}=${colour}`;
});

// Loop through lines
allGradientsTypes.forEach((gradient, index, gradients) => {
    filterLinesGenerator += `gradients=duration=${TIME}:type=${gradient}${colourParams}[out${index}];`; // NEW - add colours to command
    filterLinesText += `[out${index}]drawtext=fontfile=${TEXT_FONT}\
:text=${gradient}:fontsize=${FONT_SIZE}:${TEXT_POS}:fontcolor=${TEXT_COLOR}[text_out${index}];`;
    filterLinesXStackInputs += `[text_out${index}]`;

    if (Object.is(gradients.length - 1, index)) {
        // Finish up lines
        filterLinesGenerator += `\\`;
        filterLinesText += '\\';
        filterLinesStacker = `xstack=inputs=${allGradientsTypes.length}:grid=${2}x${2}`;
        filterLinesXStackInputs += ``;
    } else {
        filterLinesGenerator += `\\\n`;
        filterLinesText += `\\\n`;
    }
});

// Build command...
const command = `${filterLinesStart}
"${filterLinesGenerator}
${filterLinesText}
${filterLinesXStackInputs}${filterLinesStacker}"\\
${filterLinesLast}`;

console.info("Command is: ", command);

exec(command, (error, stdout, stderr) => {
    if (error) {
        console.log(`error: ${error.message}`);
        return;
    }

    console.log(`stdout: ${stdout}`);

    if (stderr) {
        console.log(`stderr: ${stderr}`);
    }
    // NEW - Show the colours
    console.info(`Generated a file with ${allGradientsTypes.length} gradient demos in a ${2} by ${2} grid with colours: ${colours}`);
    return;
});

This, to me, makes it a lot easier to understand how each gradient uses the colours it is given...well, I'm sure code like this will be useful again soon. Perhaps we should move on to the next generator - mandelbrot.

The `mandelbrot` generator

This filter will take you right back to the 80s or 90s with it's classic (and slightly cheesy) psychedelic effects. Here's one example of a rapid descent into the world of chaos fractals:

ffmpeg -f lavfi -i \
mandelbrot=end_pts=100 \
-pix_fmt yuv420p -t 30 mandelbrot1.mov

(Here it is the end_pts value that speeds up the journey).

Let's do something similar to last time in javascript to explore the different shading effect presets for the parameter inner:

const { exec } = require("child_process");

// Set up variables used to construct lines
const allInnerPresets = [
    'black',
    'convergence',
    'mincol',
    'period',
];

const TEXT_FONT = '\'fonts/crystal-radio-kit/crystal radio kit.ttf\'';
const TEXT_POS = 'x=(w-text_w)/2:y=(h-text_h)/10';
const TEXT_COLOR = 'black';
const FONT_SIZE = 35;
const TIME = 30;
// Setup lines
let filterLinesStart= "ffmpeg -y -filter_complex \\";
let filterLinesGenerator = ""; // generator lines
let filterLinesText = ""; // text lines
let filterLinesXStackInputs = "";
let filterLinesStacker = "";

let filterLinesLast = ` -t ${TIME} -an -pix_fmt yuv420p mandelbrotDemo.mp4`;

// Loop through lines
allInnerPresets.forEach((innerPreset, index, allInnerPresets) => {
    filterLinesGenerator += `mandelbrot=inner=${innerPreset}:end_pts=100[out${index}];`;
    filterLinesText += `[out${index}]drawtext=fontfile=${TEXT_FONT}\
:text=${innerPreset}:fontsize=${FONT_SIZE}:${TEXT_POS}:fontcolor=${TEXT_COLOR}[text_out${index}];`;
filterLinesXStackInputs += `[text_out${index}]`;

    if (Object.is(allInnerPresets.length - 1, index)) {
        // Finish up lines
        filterLinesGenerator += `\\`;
        filterLinesText += '\\';
        filterLinesStacker = `xstack=inputs=${allInnerPresets.length}:grid=${2}x${2}`;
        filterLinesXStackInputs += ``;
    } else {
        filterLinesGenerator += `\\\n`;
        filterLinesText += `\\\n`;
    }
});

// Build command...
const command = `${filterLinesStart}
"${filterLinesGenerator}
${filterLinesText}
${filterLinesXStackInputs}${filterLinesStacker}"\\
${filterLinesLast}`;

console.info("Command is: ", command);

exec(command, (error, stdout, stderr) => {
    if (error) {
        console.log(`error: ${error.message}`);
        return;
    }

    console.log(`stdout: ${stdout}`);

    if (stderr) {
        console.log(`stderr: ${stderr}`);
    }

    console.info(`Generated a file with ${allInnerPresets.length} mandelbrot demos in a ${2} by ${2} grid`);
    return;
});

This yields even more trippiness:

As a next step, we could use a similar process to handle the outer parameter for example, using a nested loop. This is also something we might return to later. Moving on though, we can use very similar code for the next filter which is mptestsrc.

The `mptestsrc` filter

mptestsrc is yet another option for generating useful test patterns and is apparently based on the Mplayer test filter. It only generates patterns of size 256x256 and has an option to cycle through all of them which is handy (and otherwise something we might do using the concat filter):

ffmpeg -f lavfi -i \
mptestsrc=d=40\
:max_frames=100 \
-pix_fmt yuv420p mptestsrc.mov

That gives us 4 seconds of each test source:

It becomes apparent that the size is in fact different for some of the tests, which could be useful to know. We could try applying a similar approach to the above javascript, but using this list of filters instead:

const allPresets = [
    'dc_luma',
    'dc_chroma',
    'freq_luma',
    'freq_chroma',
    'amp_luma',
    'amp_chroma',
    'cbp',
    'mv',
    'ring1',
    'ring2'
];

which produces:

The mysterious section '15.10'...

We could theoretically do a similar thing with another generator - a generator that is basically another list of test sources. It doesn't seem to have a group name, but at the time of writing it is under section 15.10. This is the list:

const allPresets = [
    'allrgb', 
    'allyuv',
    'color',
    'colorchart',
    'colorspectrum',
    'haldclutsrc',
    'nullsrc',
    'pal75bars',
    'pal100bars',
    'rgbtestsrc',
    'smptebars',
    'smptehdbars',
    'testsrc',
    'testsrc2',
    'yuvtestsrc'
];

However, in this particular case several of the filters are differently sized and best viewed in full resolution, so we can't really present them all at once (well, I'm sure we could really and maybe we will, although not right now). This is an example that is probably best suited to the approach of concatenating the test source outputs (much like the mptestsrc). That, too, is an exercise for a later article. Here are a couple of the available test sources though:

ffmpeg -f lavfi -i \
allrgb \
-update 1 -frames:v 1 testSrc1.png

(this one is static so I present it as an image here).

ffmpeg -f lavfi -i \
testsrc \
-pix_fmt yuv420p -t 6 testSrc2.mov

Various other generators...

So we have several more generators to explore... Just to round things off nicely here are a couple more - there is an implementation of The Game of Life as source video in which you can alter various parameters:

ffmpeg -f lavfi -i \
life=ratio=0.1:death_color=#FF0000:\
life_color=#00FF00:mold_color=yellow:mold=1:s=400x400 \
-t 30 life3.mp4

And there are also some other simpler Fractal generators - the sierpinski and cellauto test sources shown here:

All three of the above have many parameters to test out, so would be ideal to present with different parameters in an xstack demo. Also something to explore in the future.

Generators with plug-in capability

Beyond that, we still have yet to explore two larger generators. These are the openclsrc generator which is a way of using OpenCL code to generate video sources, and the frei0r_src generator which enables using frei0r scripts as video generators. As you might expect, there is a lot to examine there. And that is what we'll do in a later article.

Finally, all of the videos in this article were converted to gifs with code such as:

ffmpeg -t 6 -i testSrc2.mov \
-vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" \
-loop 0 testSrc2.gif

Alan Allard is a developer at Eyevinn Technology, the European leading independent consultancy firm specializing in video technology and media distribution.

If you need assistance in the development and implementation of this, our team of video developers are happy to help out. If you have any questions or comments just drop us a line in the comments section to this post.

Exploring OCR and text-to-speech in FFMPEG...

Alan Allard — Fri, 20 Jan 2023 13:13:10 +0000

...in which the author pursues a hare-brained idea to create a system to recognise text in any given image, reproduce the text, place it next to the original image and have the text be spoken aloud - all within ffmpeg, in order to explore several techniques and features within said tool. How far did the author get with all this madness? Read on to find out...

While spending an unhealthy amount of time browsing through the various filters available in ffmpeg, I discovered a couple that I really wanted to investigate. The ocr filter (as in Optical Character Recognition) is not documented in great detail - as is the case in several parts of the ffmpeg documentation. (On the other hand, some of the filters are extremely well-documented with several intriguing examples. But not this one!) Nonetheless, I wanted to try it out. Also, while writing my earlier blogs about audio in ffmpeg, I had come across the flite filter. I recognised the name (from my days in the past working with the open-source audio software Pure Data) to be text-to-speech generation software. This was also interesting to me, so I started experimenting...

Using the ocr filter

The ocr filter in ffmpeg is powered by the Tesseract library. As you will often find in ffmpeg, the build within ffmpeg has only a subset of the functionality of the original library - at least, for the moment. There's always the possibility of APIs being expanded in later ffmpeg releases. And it is open source of course, so there's the option of instigating those changes yourself - or using the original library in conjunction with ffmpeg if that suits your needs better.

To use the ocr filter, we need to be sure that ffmpeg was built with Tesseract enabled. You can check this by running ffmpeg without any options. If you see --enable-libtesseract in the configuration line in the first few lines, like this:

 % ffmpeg
ffmpeg version N-109469-g62da0b4a74 Copyright (c) 2000-2023 the FFmpeg developers
  built with Apple clang version 14.0.0 (clang-1400.0.29.202)
  configuration: --enable-libx264 --enable-gpl --enable-lv2 --enable-libfreetype --enable-libflite --enable-cross-compile --enable-libtesseract --enable-libfontconfig --enable-libfribidi

then you don't need to do anything. Otherwise add --enable-libtesseract to and run your ./configure command, then make and make install.

To test the ocr filter, a video containing text would be useful. We can generate that fairly easily in ffmpeg:

ffmpeg -f lavfi -i color=c=red:size=400x400:duration=5 -filter_complex \
drawtext=fontfile='fonts/lazenby-computer/LazenbyCompSmooth.ttf'\
:text='SYNTHESIZER':fontsize=40:x=25:y=20:fontcolor=white,\
drawtext=fontfile='fonts/lazenby-computer/LazenbyCompSmooth.ttf'\
:text='DRUM MACHINE':fontsize=40:x=25:y=150:fontcolor=white,\
drawtext=fontfile='fonts/lazenby-computer/LazenbyCompSmooth.ttf'\
:text='REVERB':fontsize=40:x=25:y=310:fontcolor=white\
 ocrTest1.mov

This is probably fairly self-explanatory. It:

generates a single colour video of dimensions 400x400 for 5 seconds
draws three texts over this, using a specified font (one that I downloaded to a local folder). We do this with the drawtext filter.

youtube.com

This was done using only the most basic font support available in ffmpeg. This installation of ffmpeg was built with only --enable-libfreetype configured. If we also configure --enable-libfontconfig and --enable-libfribidi, things get a little easier. Among other things, we can then use the default font option, meaning that we don't have to pass in a long, unwieldy path to the drawtext filter (unless we have specific font requirements:

ffmpeg -f lavfi -i color=c=red:size=400x400:duration=5 -filter_complex \
drawtext=text='SYNTHESIZER':fontsize=40:x=25:y=20:fontcolor=white,\
drawtext=text='DRUM MACHINE':fontsize=40:x=25:y=150:fontcolor=white,\
drawtext=text='REVERB':fontsize=40:x=25:y=310:fontcolor=white\
 ocrTest2.mov

So we have a video with text in, in a default font:

youtube.com

Now it would be nice to feed that through the ocr filter and see what text it recognises. Before we do that, let's also stagger the display of the texts over time but in the same position, which will allow us to demonstrate the OCR output clearly by printing it below the source text:

ffmpeg -f lavfi -i color=c=red:size=400x400:duration=9.5 -filter_complex \
"[0]drawtext=text='SYNTHESIZER':fontsize=40:x=(w-text_w)/2:y=20:\
fontcolor=white:enable='between(t,1,3)'[1];\
[1]drawtext=text='DRUM MACHINE':fontsize=40:x=(w-text_w)/2:y=20:\
fontcolor=white:enable='between(t,4,6)'[2];\
[2]drawtext=text='REVERB':fontsize=40:x=(w-text_w)/2:y=20:\
fontcolor=white:enable='between(t,7,9)'"\
 ocrTest3.mov

Notice that we centred the texts horizontally (using the text width text_w, which is available within the drawtext filter). Also, if you take a moment to run ffmpeg -filters, you will see a list of all the available filters with three characters preceding each filter. Here is the one for drawtext:

T.C drawtext          V->V       Draw text on top of video frames using libfreetype library.

The T is an indication that drawtext has timeline support. This means that we can enable the filter based on the time we are at in the video. There are several expressions available for evaluating this and we are using a common one, the between expression. For example, the first drawtext instance is only enabled between 1 and 3 seconds in the video. This is how it looks for all three:

youtube.com

So now we have something that our ocr filter can respond to. This is how we arrange that:

ffmpeg -i ocrTest3.mov -filter_complex \
"ocr,drawtext=fontsize=40:fontcolor=black:x=(w-text_w)/2:y=(h-text_h)/2:\
text='%{metadata\:lavfi.ocr.text}'" ocrOutput1.mov

resulting in this video:

youtube.com

Here we are using drawtext to display the data that the ocr filter outputs to the metadata that accompanies each frame of the video. This is why the OCR output changes; it's being updated per frame. (We will examine this metadata shortly).

To explore things further, let's use a text pulled from the internet as a source. We will then position it next to the ocr output in an output image. Unfortunately for this particular (surely atypical) use-case, the ocr stream leaves the source input unchanged, which means our recognised text would be output on top of the existing image. So let's cover the stream with a white colour overlay before outputting the ocr
results using drawtext on top:

ffmpeg -i test.png -f lavfi -i "color=white" -filter_complex \
"split[textOut][origOut];\
[textOut]ocr[ocrOut];\
[1:v][ocrOut]scale2ref=w=iw:h=ih[textScaled][colorScaled];\
[colorScaled][textScaled]overlay[overlayOut];\
[overlayOut]drawtext=fontsize=40:fontcolor=black:x=(w-text_w)/2:y=(h-text_h)/2:\
text='%{metadata\:lavfi.ocr.text}'[drawtextOut];\
[origOut][drawtextOut]hstack" -t 1 -update 1 ocrOutput2.png

Useful things to note here are that we resize the white color image to the original .png using the scale2ref filter with the (source video) input height and input width constants ih and iw. Then we overlay the OCR output text onto that white background. Finally, we line up the two video streams horizontally using the hstack filter. If we didn't add the -t 1, this stream would continue until we shut it down manually. And the -update 1 is necessary for outputting a single image instead of a video, video being what ffmpeg is primarily designed to work with, of course. Here is ocrOutput2.png:

I was curious as to how the ocr filter handled angled text - and if it didn't really, then how much of an angle can cause distortion of the recognised text:

ffmpeg -i test.png -f lavfi -i "color=red" -filter_complex \
"rotate='PI/24:ow=hypot(iw,ih):oh=ow'[rotOut];\
[rotOut]split[textOut][origOut];\
[textOut]ocr[ocrOut];\
[1:v][ocrOut]scale2ref=w=oh*mdar:h=ih[textScaled][colorScaled];\
[colorScaled][textScaled]overlay[overlayOut];\
[overlayOut]drawtext=fontsize=40:fontcolor=black:x=(w-text_w)/2:y=(h-text_h)/2:\
text='%{metadata\:lavfi.ocr.text}'[drawtextOut];\
[origOut][drawtextOut]hstack" -t 1 -update 1 ocrOutput2.png

What's new here is the rotate filter, which rotates the test image by 7.5 degrees (expressed in radians). The parameters after the angle expression - ow=hypot(iw,ih):oh=ow - control the output height and width and ensure that the image is contained within its frame when we rotate it (this will be particularly useful when we animate the rotation shortly). I coloured the text underlay red for clarity, giving us this for ocrOutput2.png

And rotating by PI/12 (15 degrees) gives us:

And here you can see, perhaps unsurprisingly, that the OCR results are starting to break up a bit. 30 degrees looks like this:

Now let's test with a couple of real-world images. First this (the image shown here is much smaller than the large output file I got):

That's quite impressive - as well as recognising the neon sign, it did actually manage to capture some of the text on the synthesizers too!

Now this (same here, much-reduced version of the resulting image):

Pretty extreme, the texture of the book and/or the resulting pixels got recognised as characters.

Much better, especially considering that the ocr filter defaults to interpreting the English language (this is Swedish). (It can be changed using the language parameter though).

For a quick overview of how rotating the text affects the ocr output - and mostly because it's instructive - let's animate the rotation in a 12 second video:

ffmpeg -loop 1 -i test.png -f lavfi -i "color=white" -filter_complex \
"rotate='PI/6*t:ow=hypot(iw,ih):oh=ow'[rotOut];\
[rotOut]split[textOut][origOut];\
[textOut]ocr[ocrOut];\
[1:v][ocrOut]scale2ref=w=oh:h=ih[textScaled][colorScaled];\
[colorScaled][textScaled]overlay[overlayOut];\
[overlayOut]drawtext=fontsize=40:fontcolor=black:\
x=(w-text_w)/2:y=(h-text_h)/2:text='%{metadata\:lavfi.ocr.text\:NA}'[drawtextOut];\
[origOut][drawtextOut]hstack" -t 12 ocrOutput8.mov

Here we are animating by multiplying the rotation by time t. Otherwise most of this is the same as previously, except for the fact that we have a default value of "NA" for when the lavfi.ocr.text data is empty or non-existent. It's quite clear that most of the OCR output is junk, but the effect is quite entertaining (I think so, in any case). If you did want to use this text in some way, we can analyse the output video in ffprobe like so:

ffprobe -show_entries frame_tags=lavfi.ocr.text,lavfi.ocr.confidence -f lavfi -i "movie=ocrOutput8.mov,ocr"  > ocrText.txt

Here's a couple of the frame outputs from that text file:

[FRAME]
TAG:lavfi.ocr.text=It was the best of

times, it was the worst Fe REG oT
of times, it was the age St eee
of wisdom, jas the age of foolishness...
age of foolishness...

TAG:lavfi.ocr.confidence=96 96 96 96 96 96 96 95 96 96 31 4 5 97 96 96 96 96 96 1 0 96 96 21 96 96 96 96 95 95 96 
[SIDE_DATA]
[/SIDE_DATA]
[/FRAME]
[FRAME]
TAG:lavfi.ocr.text=It was the best of

times, it was the worst times it was the worst
of times, it was the age oF Wego twas they.
of wisdom, eae fie age of foolishness...
age of foolishness...

TAG:lavfi.ocr.confidence=96 96 96 96 96 96 95 96 96 96 70 80 80 96 96 96 96 96 96 96 96 73 27 26 23 96 96 41 46 95 95 96 96 97 95 
[/FRAME]

This is quite early in the rotation and most of the text is intact. The ocr filter also outputs a confidence rating for each word, which is fairly high at this stage. Later on it drops to 0 at worst:

FRAME]
TAG:lavfi.ocr.text=aa QB
s oO a omMmMs 4
2 m3 gf =o a0 F
man &
% 24Gb & 04,9
4.3 2 a 04.520
oC n ae Cofnae
mm OY 4 oO m OY a)
a oS a oO OA2e4Cc
a> 327% 6 $3 2% 9
a. Se % 5 DP
Ce eS a,
Rae et Eee &S
O
S40 & S470 &
Lo) eo & 4 Bec4
%% 2 4 %& 2S
Pah GY Paet %
, TP we , De we
@ 9 aj "00 &

TAG:lavfi.ocr.confidence=59 95 0 8 74 90 96 67 37 95 57 57 75 64 74 79 16 6 47 48 58 47 79 0 0 67 90 11 51 94 26 92 96 96 43 22 53 53 90 42 45 36 89 89 96 42 9 0 96 95 18 27 49 47 41 93 92 0 0 0 91 91 0 38 27 89 90 13 61 81 36 90 0 37 91 92 37 56 56 90 95 95 48 32 43 81 81 
[/FRAME]

So now we have a good idea of how the ocr filter works and can now move on to...speech!

Introducing the flite filter

As I mentioned, it is the flite filter that can generate speech in ffmpeg. The name flite derives from the fact that the library is designed as a lightweight and portable version of the Festival speech synthesis library. In the ffmpeg implementation there are six speech synthesis voices available. Flite itself has support for downloading voices on the fly, but this is currently unavailable within ffmpeg. Here is an example of text being spoken in the flite filter:

ffmpeg -filter_complex \
"flite=text='This is the slt voice':\
voice=slt" fliteOut1.wav

Text can also be spoken from a text file:

ffmpeg -filter_complex \
"flite=textfile=speech.txt:\
voice=rms" fliteOut2.wav

NB: You may need to re-configure ffmpeg in order to use it - in this case, we'll also need to clone the flite repo to get it working:

fetch the repo: git clone https://github.com/festvox/flite.git
move to the directory: cd flite/
configure, make and install:
- ./configure
- make
- sudo make install

Then in ffmpeg:

./configure --enable-libflite --enable-cross-compile
make install

(These are macOS Monterey instructions, others are here)

Combining OCR and text-to-speech...

So could we build an ffmpeg command that takes in an image of text, recognises the text and then speak it aloud? Well, there are limitations. For example, if the flite filter had had the ability to handle text expansion we could simply build it into the filtergraph in the same way as the drawtext filter.
Instead we are going to have to read the text from a file. This means that we will have to resort to some sort of compound command. Here's one possibility:

ffmpeg -y -i test.png -f lavfi -i "color=red" -filter_complex \
"[0]split[textOut][origOut];\
[textOut]ocr[ocrOut];\
[1:v][ocrOut]scale2ref=w=iw*mdar:h=ih[textScaled][colorScaled];\
[colorScaled][textScaled]overlay[overlayOut];\
[overlayOut]metadata=mode=print:key=lavfi.ocr.text:file=metadata.txt:direct=1[metadataOut];\
[metadataOut]drawtext=fontsize=40:fontcolor=black:x=(w-text_w)/2:y=(h-text_h)/2:\
text='%{metadata\:lavfi.ocr.text}'[drawtextOut];\
[origOut][drawtextOut]hstack" -update 1 -frames:v 1 textOutSpeak.png \
&& ffmpeg -y -loop 1 -i textOutSpeak.png -filter_complex \
"flite=textfile=metadata.txt:\
voice=rms[speakOut]" -t 15 -map 0 -map "[speakOut]" ocrAndFliteOutput1.mov

and the other way to do this would be to pipe the output from the first ffmpeg command to the next:

ffmpeg -y -i test.png -f lavfi -i "color=red" -filter_complex \
"[0]split[textOut][origOut];\
[textOut]ocr[ocrOut];\
[1:v][ocrOut]scale2ref=w=iw*mdar:h=ih[textScaled][colorScaled];\
[colorScaled][textScaled]overlay[overlayOut];\
[overlayOut]metadata=mode=print:key=lavfi.ocr.text:file=metadata.txt:direct=1[metadataOut];\
[metadataOut]drawtext=fontsize=40:fontcolor=black:x=(w-text_w)/2:y=(h-text_h)/2:\
text='%{metadata\:lavfi.ocr.text}'[drawtextOut];\
[origOut][drawtextOut]hstack" -update 1 -frames:v 1 -f image2 - \
| ffmpeg -y -loop 1 -i pipe:0 -filter_complex \
"flite=textfile=metadata.txt:\
voice=rms[speakOut]" -t 15 -map 0 -map "[speakOut]" ocrAndFliteOutput2.mov

Both of these will give us the following, which sort of works:

The problem is that flite is reading the whole output of metadata.txt rather than just the text itself. We need to trim this text data before it is read. One way is to use the grep and tail in the following way:

grep -A50 '=' metadata.txt | tail -c+16 > massagedData.txt

This will work up to a fairly reasonable maximum of 50 lines. It trims away the first part of the text, giving us the whole readable text. And so, finally, we have the following three-part compound command that takes in an image and produces OCR output and text-to-speech, tidily packaged in a short video:

ffmpeg -y -i test.png -f lavfi -i "color=red" -filter_complex \
"[0]split[textOut][origOut];\
[textOut]ocr[ocrOut];\
[1:v][ocrOut]scale2ref=w=iw*mdar:h=ih[textScaled][colorScaled];\
[colorScaled][textScaled]overlay[overlayOut];\
[overlayOut]metadata=mode=print:key=lavfi.ocr.text:file=metadata.txt:direct=1[metadataOut];\
[metadataOut]drawtext=fontsize=40:fontcolor=black:x=(w-text_w)/2:y=(h-text_h)/2:\
text='%{metadata\:lavfi.ocr.text}'[drawtextOut];\
[origOut][drawtextOut]hstack" -update 1 -frames:v 1 textOutSpeak.png \
&& grep -A50 '=' metadata.txt | tail -c+16 > massagedData.txt \
&& ffmpeg -y -loop 1 -i textOutSpeak.png -filter_complex \
"flite=textfile=massagedData.txt:\
voice=rms[speakOut]" -t 15 -map 0 -map "[speakOut]" ocrAndFliteOutput3.mov

And the result, a tidied-up version of the last video:

Obviously, this is a rather cumbersome and fragile end result. We can summarise by stating that there are better ways to do this. However, it has been quite an educational investigation into some useful tools within ffmpeg.

About the cover image

The cover image of this post was also generated with ffmpeg. I used two GLSL shaders (captured to .png files) as input. These are the shaders used for the design of two cards from the Pixelspirit Deck. The first card is called "Vision" and the second card is called "Judgement". They seemed appropriate ;)

I used the ffmpeg filter minterpolate to morph from the "Vision" image to the "Judgement" image (the morph effect is pretty basic, but it works):

ffmpeg -y -r 0.3 -stream_loop 1 -i 33_vision.png -r 0.3 -stream_loop 2 -i 31_Judgement.png -filter_complex "[0][1]concat=n=2:v=1:a=0[v];[v]minterpolate=fps=24:scd=none,trim=3:7,setpts=PTS-STARTPTS" -pix_fmt yuv420p visionToJudgement.mp4

I first captured the output images from the visionToJudgement.mp4:

ffmpeg -i visionToJudgement.mp4 -r 1 morphtest1/test-%09d.png

and tiled them horizontally (new filter alert: tile) at the right size to create the header image

ffmpeg -i morphtest1/test-%09d.png -filter_complex "scale=594:1496,tile=6x1" -update 1 -frames:v 1 output.png

Alan Allard is a developer at Eyevinn Technology, the European leading independent consultancy firm specializing in video technology and media distribution.

Working with audio in ffmpeg - Part 2

Alan Allard — Fri, 06 Jan 2023 16:26:30 +0000

Last year I examined the basics of working with audio in ffmpeg and promised a follow-up in which I demonstrate how to use LV2 plugins to further extend the possibilities for audio manipulation in every video streaming professional's favourite command line tool. And here, at last, is Part 2 of "Working with audio in ffmpeg"!

This demonstration, similarly to last time, uses a short song excerpt which I have made available for you on freesound.org.

To pick up where we left off, we established that ffmpeg already has a wealth of audio manipulation tools built right in as audio filters. There is however much more that you can do without too much effort. By using LV2 plugins, for example, you can take advantage of any number of free plugins and even code your own.

LV2 plugins are the second generation version of an earlier plugin system known as LADSPA - LV2 is an acronym for LADSPA Version 2 plugins. And it's also worth noting that LADSPA (version 1) plugins are also usable inside ffmpeg (this may well be a topic for a future blog too ;) ) and there are plenty of those available too, of course.

You may find that LV2 is not immediately available within your current ffmpeg build. In that case, you will need to build from source. Luckily, this process is well-documented in ffmpeg. I am on macOS so I tried my hand at following those specific instructions, detailed here. They were fairly straightforward, so I may as well document them here. This is also the process for enabling any of the other filters that are not included by default in any particular build. The documentation will usually indicate when this is necessary. There will be occasions when the documentation falls short though, as we will see, so be aware of that.

If you, like me, already had a version of ffmpeg on your machine, getting rid of it is probably the best option. For me, that meant doing a homebrew uninstall ffmpeg.

Next, grab ffmpeg from GitHub:

git clone https://git.ffmpeg.org/ffmpeg.git ffmpeg

then you'll want to get all the necessary dependencies to successfully build ffmpeg. I chose the simplest option in my case; installing those dependencies via Homebrew:

brew install automake fdk-aac git lame libass libtool libvorbis libvpx \
opus sdl shtool texi2html theora wget x264 x265 xvid nasm

then:
cd ffmpeg

At which point it's probably best to try a vanilla build with the following trinity of commands:

./configure
which will take a while then:
make
which can also take a bit then:
make install
which usually doesn't take too long...

If all is well, and you can run something like, say:

ffmpeg -filters

and see a nice long list of filters and their features, then we're ready to setup LV2!

First we also need to install lv2 from Homebrew:

brew install lv2

To use LV2 plugins then, we need to build with the -enable-lv2 flag:

./configure --enable-lv2

then once again:

make
and
make install

Now here you may run into an issue:

ERROR: lilv-0 not found using pkg-config

Don't stress, though. Have a sip of tea/beer and rest easy in the knowledge that all you need to do is another brew install:

brew install lilv

so after this...

make
and
make install

....you'll have had time to finish your tea/beer and get right down to testing LV2 plugins...

Let's dive into an example in the ffmpeg LV2 filter documentation:

ffmpeg -y -i "a different day intro excerpt.wav" -af \
'lv2=p=http\\://calf.sourceforge.net/plugins/Vinyl:c=drone=0.2\
|aging=0.5' vinylizedADD.wav

BUT - notice that I found it necessary to alter the escaped characters in the url for it be correctly formed. So that's one thing to think about. The other is that this doesn't work at all...because you need to install the Vinyl plugin!

The Vinyl plugin, as you may have guessed from the url, is part of the Calf plugin suite, an impressive selection of free - in all senses - professional audio tools. Let's grab them:

brew tap david0/homebrew-audio
and
brew install calf

Now you will be able to run the last ffmpeg command and hear a vinyl effect on the audio excerpt. Nice! Let's immediately make that more drastic with our own version.

I noticed in the source code documentation that the plugin GUI has several other parameters, some of them controlled by switches. So how on earth would we access those in the command line? Let's find out more about our parameters with:

ffmpeg -y -i "a different day intro excerpt.wav" -af\
'lv2=p=http\\://calf.sourceforge.net/plugins/Vinyl:c=help' \
vinylizedADD.wav

Which unpolitely - but, admittedly, helpfully - spews out the following:

The 'http://calf.sourceforge.net/plugins/Vinyl' plugin has the following input controls:
[Parsed_lv2_0 @ 0x7faa2c005d00] bypass          <float> (from 0.000000 to 1.000000) (default 0.000000)             Bypass
[Parsed_lv2_0 @ 0x7faa2c005d00] level_in                <float> (from 0.015625 to 64.000000) (default 1.000000)            Input Gain
[Parsed_lv2_0 @ 0x7faa2c005d00] level_out               <float> (from 0.015625 to 64.000000) (default 1.000000)            Output Gain
[Parsed_lv2_0 @ 0x7faa2c005d00] drone           <float> (from 0.000000 to 1.000000) (default 0.000000)             Drone
[Parsed_lv2_0 @ 0x7faa2c005d00] speed           <float> (from 33.000000 to 78.000000) (default 33.000000)          Speed
[Parsed_lv2_0 @ 0x7faa2c005d00] aging           <float> (from 0.000000 to 1.000000) (default 0.000000)             Aging
[Parsed_lv2_0 @ 0x7faa2c005d00] freq            <float> (from 600.000000 to 1800.000000) (default 1000.000000)             Frequency
[Parsed_lv2_0 @ 0x7faa2c005d00] gain0           <float> (from 0.000016 to 1.000000) (default 0.007812)             Vol 0
[Parsed_lv2_0 @ 0x7faa2c005d00] pitch0          <float> (from -1.000000 to 1.000000) (default 0.000000)            Pitch 0
[Parsed_lv2_0 @ 0x7faa2c005d00] active0         <float> (from 0.000000 to 1.000000) (default 0.000000)             Activate 0
[Parsed_lv2_0 @ 0x7faa2c005d00] gain1           <float> (from 0.000016 to 1.000000) (default 0.007812)             Vol 1
[Parsed_lv2_0 @ 0x7faa2c005d00] pitch1          <float> (from -1.000000 to 1.000000) (default 0.000000)            Pitch 1
[Parsed_lv2_0 @ 0x7faa2c005d00] active1         <float> (from 0.000000 to 1.000000) (default 0.000000)             Activate 1
[Parsed_lv2_0 @ 0x7faa2c005d00] gain2           <float> (from 0.000016 to 1.000000) (default 0.015625)             Vol 2
[Parsed_lv2_0 @ 0x7faa2c005d00] pitch2          <float> (from -1.000000 to 1.000000) (default 0.000000)            Pitch 2
[Parsed_lv2_0 @ 0x7faa2c005d00] active2         <float> (from 0.000000 to 1.000000) (default 0.000000)             Activate 2
[Parsed_lv2_0 @ 0x7faa2c005d00] gain3           <float> (from 0.000016 to 1.000000) (default 0.007812)             Vol 3
[Parsed_lv2_0 @ 0x7faa2c005d00] pitch3          <float> (from -1.000000 to 1.000000) (default 0.000000)            Pitch 3
[Parsed_lv2_0 @ 0x7faa2c005d00] active3         <float> (from 0.000000 to 1.000000) (default 0.000000)             Activate 3
[Parsed_lv2_0 @ 0x7faa2c005d00] gain4           <float> (from 0.000016 to 1.000000) (default 0.031250)             Vol 4
[Parsed_lv2_0 @ 0x7faa2c005d00] pitch4          <float> (from -1.000000 to 1.000000) (default 0.000000)            Pitch 4
[Parsed_lv2_0 @ 0x7faa2c005d00] active4         <float> (from 0.000000 to 1.000000) (default 0.000000)             Activate 4
[Parsed_lv2_0 @ 0x7faa2c005d00] gain5           <float> (from 0.000016 to 1.000000) (default 0.062500)             Vol 5
[Parsed_lv2_0 @ 0x7faa2c005d00] pitch5          <float> (from -1.000000 to 1.000000) (default 0.000000)            Pitch 5
[Parsed_lv2_0 @ 0x7faa2c005d00] active5         <float> (from 0.000000 to 1.000000) (default 0.000000)             Activate 5
[Parsed_lv2_0 @ 0x7faa2c005d00] gain6           <float> (from 0.000016 to 1.000000) (default 0.062500)             Vol 6
[Parsed_lv2_0 @ 0x7faa2c005d00] pitch6          <float> (from -1.000000 to 1.000000) (default 0.000000)            Pitch 6
[Parsed_lv2_0 @ 0x7faa2c005d00] active6         <float> (from 0.000000 to 1.000000) (default 0.000000)             Activate 6

So let's try that more drastic version of the same:

ffmpeg -y -i "a different day intro excerpt.wav" -af\
'lv2=p=http\\://calf.sourceforge.net/plugins/Vinyl:c=drone=1.0|aging=1.0' \
vinylizedADD.wav

Hmm, needs to be louder (not much though, those resonant frequencies can be a bit nasty). And let's add a speed change too (this changes the LFO speed of the drone, in fact).

ffmpeg -y -i "a different day intro excerpt.wav" -af \
'lv2=p=http\\://calf.sourceforge.net/plugins/Vinyl:c=drone=1.0|aging=1.0\
|level_out=2.0|speed=45.0' vinylizedADD.wav

So what about these several gain, pitch and activate parameters? It seems from examining the GUI of this plugin that these are basically seven file players with various different vinyl-style effects supplied as looping audio files. Let's add a couple:

ffmpeg -y -i "a different day intro excerpt.wav" -af \
'lv2=p=http\\://calf.sourceforge.net/plugins/Vinyl:c=drone=0.6|aging=0.6\
|level_out=2.0|speed=45.0|active5=1.0|gain5=1.0|pitch5=1.0|active6=1.0\
|gain6=1.0|pitch6=-1.0' vinylizedADD.wav

Oddly enough, those audio effects don't loop. Why is that? Well, who knows...but you can track that bug here. From reading that, it transpires that one of them does actually loop correctly. By trial and error, I found it - gain/pitch/active1:

ffmpeg -y -i "a different day intro excerpt.wav" -af \
'lv2=p=http\\://calf.sourceforge.net/plugins/Vinyl:c=drone=0.6|aging=0.6\
|level_out=2.0|speed=45.0|active1=1.0|\
gain1=0.1|pitch1=1.0' vinylizedADD.wav

Well that's lovely, how about some reverb?

ffmpeg -y -i "a different day intro excerpt.wav" -af \
'lv2=p=http\\://calf.sourceforge.net/plugins/Reverb:c=help' \
vinylizedADD.wav

Ok then, these are the params this time:

The 'http://calf.sourceforge.net/plugins/Reverb' plugin has the following input controls:
[Parsed_lv2_0 @ 0x7f954b00c700] decay_time              <float> (from 0.400000 to 15.000000) (default 1.500000)            Decay time
[Parsed_lv2_0 @ 0x7f954b00c700] hf_damp         <float> (from 2000.000000 to 20000.000000) (default 5000.000000)           High Frq Damp
[Parsed_lv2_0 @ 0x7f954b00c700] room_size               <float> (from 0.000000 to 5.000000) (default 2.000000)             Room size
[Parsed_lv2_0 @ 0x7f954b00c700] diffusion               <float> (from 0.000000 to 1.000000) (default 0.500000)             Diffusion
[Parsed_lv2_0 @ 0x7f954b00c700] amount          <float> (from 0.000000 to 2.000000) (default 0.250000)             Wet Amount
[Parsed_lv2_0 @ 0x7f954b00c700] dry             <float> (from 0.000000 to 2.000000) (default 1.000000)             Dry Amount
[Parsed_lv2_0 @ 0x7f954b00c700] predelay                <float> (from 0.000000 to 500.000000) (default 0.000000)           Pre Delay
[Parsed_lv2_0 @ 0x7f954b00c700] bass_cut                <float> (from 20.000000 to 20000.000000) (default 300.000000)              Bass Cut
[Parsed_lv2_0 @ 0x7f954b00c700] treble_cut              <float> (from 20.000000 to 20000.000000) (default 5000.000000)             Treble Cut
[Parsed_lv2_0 @ 0x7f954b00c700] on              <float> (from 0.000000 to 1.000000) (default 1.000000)             Active
[Parsed_lv2_0 @ 0x7f954b00c700] level_in                <float> (from 0.015625 to 64.000000) (default 1.000000)            Input Gain
[Parsed_lv2_0 @ 0x7f954b00c700] level_out               <float> (from 0.015625 to 64.000000) (default 1.000000)            Output Gain

So how about:

ffmpeg -y -i "a different day intro excerpt.wav" -af \
'lv2=p=http\\://calf.sourceforge.net/plugins/Reverb:c=amount=0.9\
|amount=1.0|room_size=5.0|hf_damp=2000.0|predelay=5.0\
|diffusion=1.0' reverbedADD.wav

It would be nice to hear the reverb tail on that, but I haven't yet given thought on how to do that. (Please let me know if you work it out). There is a lot more to explore within just the Calf audio suite, let alone all of the other free alternatives out there. Perhaps one day you might find you need one of these tools, be it for a quick fix or a professional editing repair...

One final note: the image for this blog post is a frequency plot generated from exactly the audio excerpt that I have been using in this blog post. Naturally, this was also generated within ffmpeg:

ffmpeg -i "a different day intro excerpt.wav" -lavfi \
showspectrumpic=s=1280x480:scale=log:color=fiery:legend=0 add.png

(Chances are I'll look closer at showspectrumpic in a future blog post...)

Alan Allard is a developer at Eyevinn Technology, the European leading independent consultancy firm specializing in video technology and media distribution.

Working with audio in ffmpeg

Alan Allard — Fri, 25 Feb 2022 15:20:11 +0000

Everyone working in video streaming development knows about ffmpeg and most of us use it from time to time in our daily assignments. It's also well-known that ffmpeg has a whole bunch of functionality for audio. Not as many of us delve into that aspect I suspect though, beyond perhaps converting the occasional audio track format in some way. Ffmpeg does go way beyond that though, offering a plethora of conversion, analysis and even sound generation tools.

Re-capping simply and very briefly, a typical ffmpeg command consists of one or more inputs (-i), any number of operations and a number of outputs. (There are many more aspects to the command syntax though - see the ffmpeg docs for the full spec).

ffmpeg -i bunny.mp4 -c:v libvpx-vp9 -c:a libvorbis bunny.webm

The power lies in what the inputs can be and in the operations, which are extremely numerous, chainable - and also extensible (that's something we will adress in the next article...). Here is a slightly more complex example, by a colleague of mine:

# Low Latency ScreenGrab
`ffmpeg -f avfoundation -capture_cursor 1 -i 3 -r 30 -c:v libx264 -an -tune zerolatency -preset ultrafast -pix_fmt yuv420p -fflags flush_packets -f mpegts - | mpv - --no-cache --untimed --no-demuxer-thread --video-sync=audio --vd-lavc-threads=1`

There, we are using the screen as an input. (This is quite a specific command and will need some adjustment when running on another device though).

Download FFmpeg and try the following examples. I am using a track snippet that you can download from Freesound.

Audio conversion

A video streaming developer might work with encoding or transcoding various audio tracks for video. For example, you might want to convert from mono to stereo:

ffmpeg -i critics.flac -ac 1 critics_mono.flac

or 5.1 to LR stereo (here is the same file mixed as 5.1 so you can try it out):

ffmpeg -i critics_surround.wav -ac 2 critics.wav

or converting formats, which can be even simpler, for example to mp3 or flac:

ffmpeg -i critics.wav critics.mp3
ffmpeg -i critics.wav critics.flac

It's also worth mentioning the ffprobe tool that is typically bundled with ffmpeg. Use it on any of the above files to see all the available file info. In the case of the surround example, it reveals my song metadata from Logic Pro X:

Input #0, wav, from 'critics_surround.wav':
  Metadata:
    encoded_by      : Logic Pro X
    date            : 2022-02-25
    creation_time   : 12:04:39
    time_reference  : 163250181
    umid            : 0x0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004500FA50
    coding_history  : 
  Duration: 00:00:15.02, bitrate: 4237 kb/s
  Chapters:
    Chapter #0:0: start 0.000000, end 15.018662
      Metadata:
        title           : Tempo: 132.0
    Chapter #0:1: start 0.000000, end 0.113628
      Metadata:
        title           : Bridge 1
    Chapter #0:2: start 0.113628, end 14.545442
      Metadata:
        title           : Chorus
    Chapter #0:3: start 14.545442, end 15.018662
      Metadata:
        title           : Chorus w/ Vox
  Stream #0:0: Audio: pcm_s16le ([1][0][0][0] / 0x0001), 44100 Hz, 5.1, s16, 4233 kb/s

While we're playing around with various channels, what happens if you want to convert from left-right stereo to mid-sides stereo...? Well, then you'll want to make use of a filter...

filters

Simple audio filters are invoked with the -af flag. A filter in this case is a tool with an input and output that manipulates the incoming data in some way. Some examples of when you can make use of audio filters are as follows.

When you want to fade out some audio:

ffmpeg -i critics.wav -a 'afade=t=out:st=4:d=9' critics_faded.wav

To create a fairly naive stereo-widening effect by delaying the left and right channels by different, small amounts:

ffmpeg -i critics.wav -af 'adelay=5|0|10' critics_wide.wav

To create a flange effect using an echo algorithm with a very short delay time and regeneration:

ffmpeg -i critics.wav -af 'aecho=0.8:0.88:6:0.9' critics_echo_flange.wav

To pitch down by 2 octaves and filter audio (this one sounds wild...):

ffmpeg -i critics.wav -af 'afreqshift=shift=-2400' critics_shifted.wav

So back to the stereo conversion we wanted to do. We could use the stereotools audio filter for that:

ffmpeg -i critics.wav -af 'stereotools=mode=lr>ms' critics_ms.wav

Complex filters

If you want to work with several inputs and/or outputs, you will need to use complex filters instead, using the -filter_complex option instead of -af. Here is an example that outputs three separately filtered audio files from the original source:

ffmpeg -i critics.wav -filter_complex 'acrossover=split=900 7000:order=8th[LOW][MID][HIGH]' -map '[LOW]' critics_low.wav -map '[MID]' critics_mid.wav -map '[HIGH]' critics_high.wav

And an example - with multiple inputs - of performing a crossfade of two of the earlier files:

ffmpeg -i critics.wav -i critics_shifted.wav -filter_complex acrossfade=d=10:c1=exp:c2=exp critics_xfade.wav

You can chain complex_filters one after the other using an array-like syntax:

ffmpeg -i critics.wav -filter_complex 'afreqshift=shift=-1200[shifted];[shifted]chorus=0.5:0.9:50|60|40:0.4|0.32|0.3:0.25|0.4|0.3:2|2.3|1.3[chorused];[chorused]areverse' critics_destroyed.wav

To break this down into its three parts:

afreqshift=shift=-1200[shifted] - shifts the audio down an octave
[shifted]chorus=0.5:0.9:50|60|40:0.4|0.32|0.3:0.25|0.4|0.3:2|2.3|1.3[chorused] - takes the output of the shifted filter and applies chorus to it
[chorused]areverse - takes the output of the chorused filter and reverses it

You can do things like this on parallel inputs and outputs, then mix them together:

ffmpeg -i critics.wav -i critics_low.wav -filter_complex '[0]afreqshift=shift=-1200[shifted];[shifted]chorus=0.5:0.9:50|60|40:0.4|0.32|0.3:0.25|0.4|0.3:2|2.3|1.3[chorused];[chorused]areverse[reversed];[1]afreqshift=shift=-3600[shifted2];[reversed][shifted2]amix=inputs=2' critics_annihilated.wav

Notice the similarity to the previous statement, but here we have added two input files and a new filter path for the second file. We add a [0] to the first audio path to capture the input from the first file then:

[1]afreqshift=shift=-3600[shifted2]; - captures the input from the second audio file and shifts it down 3 octaves.
[reversed][shifted2]amix=inputs=2 - combines the result of both audio flows into one file.

These kinds of flows are called audiographs and can be much more complex than this if necessary. There are even tools included in the ffmpeg suite to represent such audiographs graphically. I hope to take a more detailed look at this in the future.

There are many more built-in audio filters available in ffmpeg. Beyond that though, it's also possible to use audio plugins within ffmpeg. One of the formats you can use for this is the widely used LV2 format, the same format used in some recent audio effects hardware (eg. Mod Devices' Mod Duo X). Support for this is not necessarily included in a typical ffmpeg installation though. In the next article I will look at custom ffmpeg builds and getting LV2 plugins working within them.

I recommend another article on this subject that was very much the inspiration for my own. I've tried not to repeat what was said there and instead provided yet more useful examples as a complement to that article.

UPDATE: Similarly to Part 2 of this blog, which is now available, I added a frequency plot of the audio file as the blog post image using ffmpeg. I also converted the file to 16 bit, 44.1 kHz prior to plotting:

ffmpeg -i "critics (Excerpt).wav" -acodec pcm_s16le -ar 44100 critics16bit.wav

ffmpeg -i "critics16bit.wav" -lavfi \
showspectrumpic=s=1000x420:scale=sqrt:color=cividis:legend=0:gain=20 c.png
ffmpeg -i c.png -vf crop=900:600:0:0 -update 1 c1.png

Alan Allard is a developer at Eyevinn Technology, the European leading independent consultancy firm specializing in video technology and media distribution.

Building React Native UI Components and Modules in 2021

Alan Allard — Sun, 12 Dec 2021 19:07:29 +0000

Intro

React Native has been around for some time now. And as many of us know, one of the advantages of the platform are the many ready-made components available for use. Another is of course, the other side of that same coin - that you can build your own components for those more specific needs that so often arise in a project.

In a recent assignment, I had occasion to build a wrapper around two separate proprietary video players for iOS and Android respectively. The players in question were mature and well-built but the interfaces had led their own lives, so to speak, and were quite different to each other - due mostly to the platform-specific differences. Also, the team that were intending to use the component in their React Native project had specific needs of their own, not necessarily met by the existing components, these having been developed previously for other apps with different requirements.

This is a fairly common scenario when building a React Native component wrapper, where the development needs to occur across several different teams - in this case, the app team, the iOS team and the android team. Also, while the official React Native documentation is ever-improving in this area, there are still large gaps that you will often come across as soon as you dig deeper into component development. In this article, based on my recent work in the area, I will discuss some useful tools and processes as well as some of the undocumented techniques that you will often want to use when wrapping platform-specific components.

I will also mention some interesting and radical changes in the codebase that will become the new way to develop React Native components in future.

Creating the project

A typical React Native component project will benefit from having a simple example project embedded into the repo. This can simplify development of the component as well as provide a decent demo of what it can do when it's ready for publishing. In our case, we also wanted typescript set up and ready to go. There is usually a fair amount of boilerplate code needed on both the native sides and in javascript. Given all of this, I chose to generate the component project with a tool by Callstack, called create-react-native-library. This is a CLI that will create a project for you that meets all the above demands (and more):

There is of course the usual caveat that, as with any wizard-type tool, you will want to at least have some idea of what it's doing behind the scenes. And most likely you will need to tweak your typescript and linting setup according to what you need. As you can see from the above GIF though, another plus is that you also have the opportunity to choose which languages to set up your boilerplate for. As Android has both Java and Kotlin available for React Native development and iOS has both Objective-C and Swift available, you can really save some time here and fit smoothly in to whatever project you're building for.

To get up and running with create-react-native-library is as simple as:

npx create-react-native-library your-new-project

This will give you iOS and Android modules in the languages of your choice and a fully functional embedded example project, so you can just focus on building out what you need.

Integrating your platform-specific packages

The next step is to get your project building with the packages you intend to provide a wrapper around. Perhaps you are only wrapping a component for a specific platform, or two platform-specific components from the same vendor or perhaps - as in the case of my most recent project - two separate platform-specific components that do not even share the same common interface, beyond a more general use-case. If you are developing for both platforms here, there will be some focus later on on developing a common interface.

This step is where you will need to work in the platform-specific part of the generated project - in XCode or Android Studio typically. As these components are designed specifically for integration into other projects, you will likely simply need to follow the iOS or Android guides for the component in question. In my case, it was necessary to collect some information from the respective development teams, as there were new use-cases planned for the React Native component (for example, videos were going to be expected to play in a newsfeed-style list instead of fullscreen). The example project can be very useful at this stage, to show that each integration is working properly in its most basic form. In my case, this meant showing and autoplaying on focus a list of videos in both platforms.

Module or UI Component

You may need to build one or both of the two possible types of component in React Native - a module or a UI component. Modules are good wrappers for API components, while components or packages with a user interface need of course to be UI Components.

Here, I set out to build a UI Component for each platform with the understanding that certain parts of the project may benefit from being separated from the UI as an API module. In fact, while this may be planned for the future, what actually happened was that it turned out that the only way to get the functionality we were after in Android was to provide both a module part and a UI component part in the same package.

View controller vs view

When wrapping a UI component in iOS, you will quite often find that the component you are wrapping makes use of a view controller. In a React UI Component the RCTViewManager (and more often its SimpleViewManager descendent) is responsible for all instances of the component in your app. The thing is the view manager manages views (obviously) and not view controllers. To deal with this, you need to attach the view controller's view to the component's view (as a subview). That is done in the following way:

In the view you will be providing to the view manager, add a variable of the type of the view controller you want to use:

final class RNPlayerView: UIView {
    var viewController: RNPlayerViewController
    ...(rest of class here)...

Init that view controller var in your view and add the view controller's view as a subview:

    override init(frame: CGRect) {
        self.viewController = RNPlayerViewController.init()
        super.init(frame: frame)
        self.addSubview(viewController.view)
    }

Finally, make sure that the view controller's view matches the view's dimensions. In your view class:

    override func layoutSubviews() {
        super.layoutSubviews()
        self.viewController.view.frame = self.bounds
    }

That is all you need.

Properties and methods in iOS

It's worth recapping that the view mentioned in the last section is where you will typically handle all of the properties that you expect your component to expose in react native. If you are using a view controller as above most, or probably all, of these properties (apart from any event callbacks you may be handling) will need forwarding to the view controller.

A colleague of mine and I came up with a fairly tidy way to handle this.

Take for example an assetId property for a video player UI component.

Provide the usual react native property setter:

    @objc func setAssetId(_ val: String) {
        assetId = val
    }

Then respond to changes in the variable:


    var assetId: String? = nil {
        didSet {
            self._updateAsset()
        }
    }

And finally, update the variable in your view controller:

    @objc func _updateAsset() {
        if assetId != nil {
            self.viewController.updateAssetId(assetId: assetId!)
        }
    }

The nice thing is that this adapts well to the situation where you have several properties that need setting together in a group. This is fairly common in the components I have developed, where you might, for example, have a "setup" or "config" function in your component that takes several parameters, and if any one of those parameters changes, the function should be called anew. The above pattern allows you to group variables together before sending them off to the view controller in question:

    @objc func setParam1(_ val: String) {
        param1 = val
    }

    @objc func setParam2(_ val: String) {
        param2 = val
    }

    var param1: String? = nil {
        didSet {
            self._updateParams()
        }
    }

    var param2: String? = nil {
        didSet {
            self._updateParams()
        }
    }

    @objc func _updateParams() {
        if param1 != nil && param2 != nil {
            self.viewController.updateParams(param1: param1!, param2: param2!))
        }
    }

Simple and maintainable.

You will often want to react to events in your component on the react native side. The official docs have an objective-c example, but we prefer to operate in Swift, given the choice ;) Your view should have an obj-c property for each event you are responding to:

    @objc var onReady: RCTDirectEventBlock?
    @objc var onError: RCTDirectEventBlock?

And a method for each:

    func sendOnReadyToRN() {
        if onReady != nil {
            onReady!(["Ready": "Ready"])
        }
    }

    func sendOnErrorToRN(error: String) {
        if onError != nil {
            onError!(["Error": error])
        }
    }

and you will typically have some switch statement or similar in your view controller that calls back to these view methods:

    private func eventMonitoring() {
        self.uiComponent
            .onError{  [weak self] uiComponent, source, error in
                let errorMsg = "uiComponent Error : \(error.message)"
                self?.sendErrorToView(error: errorMsg)
            }

            .onReady{ [weak self] uiComponent, source in
                self?.sendOnReadyToView()
            }
    }

    private func sendErrorToView(error: String) {
        let componentView = self.view.reactSuperview() as! ComponentView
        componentView.sendErrorToRN(error: error)
    }


    private func sendOnReadyToView() {
        let componentView = self.view.reactSuperview() as! ComponentView
        componentView.sendPlaybackReadyToRN()
    }

Finally on the iOS/tvOS side, you will want to know how to call methods on your component. In your ViewManager class:

    @objc func start(_ node: NSNumber) {
        DispatchQueue.main.async {
            let componentView = self.bridge.uiManager.view(forReactTag: node) as! ComponentView;
            componentView.viewController.component?.start();
        }
    }

    @objc func stop(_ node: NSNumber) {
        DispatchQueue.main.async {
            let componentView = self.bridge.uiManager.view(forReactTag: node) as! ComponentView;
            componentView.viewController.component?.stop();
        }
    }

See below for an idea of what the javascript side could look like.

Setup iOS/tvOS/Android

Occasionally, it's necessary to configure a component before viewing it. One way to make that possible is to add a method similar to the start/stop methods above (for iOS/tvOS), but containing several parameters, which can then be consumed in the view controller or the view as required.

In Android, you are not immediately able to call such a setup method on a view component. Instead you will need to implement a small Android native module that forwards the necessary setup info to the view component (or at least, this is one way to do it that we successfully implemented). This might look something like this (in Kotlin, also our language of choice...):


class RNComponentSetupManager(reactContext: ReactApplicationContext, viewManager: RNComponentViewManager) : ReactContextBaseJavaModule(reactContext) {

  var viewManager: RNComponentViewManager = viewManager

  override fun getName(): String {
    return "RNComponentSetupManager"
  }

  @ReactMethod
  fun setup(param1: String, param2: String) {
    viewManager.setup(param1, param2)
  }
}

Then we can set up our ReactPackage class to initialise either the view manager or the setup manager, whichever is called first:

lass RNComponentPackage : ReactPackage {

  inner class Modules(val setupManager: RNComponentSetupManager, val viewManager: RNComponentViewManager) {
  }

  var modules: Modules? = null

  private fun getInitiatedModules(reactContext: ReactApplicationContext) : Modules {
    val currentModules = modules
    return if(currentModules != null) {
      currentModules
    } else {
      val viewManager = RNComponentViewManager(reactContext)
      val setupManager = RNComponentSetupManager(reactContext, viewManager) //<-- Send in viewManager so we can use it.
      val initializedModules = Modules(setupManager, viewManager)
      this.modules = initializedModules
      initializedModules
    }
  }

  override fun createNativeModules(reactContext: ReactApplicationContext): List<NativeModule> {
    return listOf(getInitiatedModules(reactContext).setupManager)
  }

  override fun createViewManagers(reactContext: ReactApplicationContext): List<ViewManager<*, *>> {
    return listOf(getInitiatedModules(reactContext).viewManager)
  }
}

Other properties and methods in Android

Setting properties is a little simpler in Android, where there is no view controller as such.

  @ReactProp(name = "assetId")
  fun setAssetId(view: View, assetId: String) {
    val componentView = view as ComponentView
    componentView.setAssetId(assetId)
  }

and in your view class, handle that as necessary.

For methods on the view, you will need to override the receiveCommand method, the equivalent of the two iOS/tvOS methods above looking like this in the view manager:

  override fun receiveCommand(root: View, commandId: String?, args: ReadableArray?) {
    when (commandId) {
      "startFromManager" -> start(root)
      "stopFromManager" -> stop(root)
    }
  }

  private fun start(view: View) {
    val componentView = view as ComponentView
    componentView.start()
  }

  private fun stop(view: View) {
    val componentView = view as ComponentView
    componentView.stop()
  }

And similarly in the view, handle those start() and stop() commands within your native component as appropriate.

Javascript side

Finally, the typescript for our theoretical component would look something like this...notice in particular the way we handle the typing of iOS/tvOS and Android setup/view managers.

import {
    requireNativeComponent,
    ViewStyle,
    UIManager,
    findNodeHandle,
    Platform,
    NativeModules,
  } from 'react-native';

  import * as React from 'react';
  import {
    useRef,
    useImperativeHandle,
    forwardRef,
    SyntheticEvent,
  } from 'react';

  export interface RNComponentManagerProps {
    assetId: string;
    style: ViewStyle;
    start(): void;
    stop(): void;
    onReady(event: SyntheticEvent): void;
    onError(event: SyntheticEvent): void;
  }

  interface RNComponentProps {
    assetId: string;
    style: ViewStyle;
    onReady?(): void;
    onError?(error: String): void;
  }

  interface RNComponentSetupManagerProps {
    setup(
      param1: string,
      param2: string
    ): void;
  }

  export const RNComponentSetupManager =
    Platform.OS === 'ios'
      ? (NativeModules.RNComponentViewManager as RNComponentSetupManagerProps)
      : (NativeModules.RNComponentSetupManager as RNComponentSetupManagerProps);

  const RNComponentViewManager = requireNativeComponent<RNComponentManagerProps>(
    'RNComponentView'
  );

  const RNComponent = forwardRef((props: RNComponentProps, extRef) => {
    useImperativeHandle(extRef, () => ({
      stop,
      start
    }));
    const componentRef = useRef(null);


    const start = () => {
      var command = 'startFromManager';
      if (componentRef && componentRef.current) {
        UIManager.dispatchViewManagerCommand(
          findNodeHandle(componentRef.current),
          // @ts-ignore: Issue in RN ts defs 
          Platform.OS === 'ios'
            ? UIManager.getViewManagerConfig('RNComponentView').Commands
                .startFromManager
            : command,
          undefined
        );
      }
    };

    const stop = () => {
      var command = 'stopFromManager';
      if (componentRef && componentRef.current) {
        UIManager.dispatchViewManagerCommand(
          findNodeHandle(componentRef.current),
          // @ts-ignore: Issue in RN ts defs 
          Platform.OS === 'ios'
            ? UIManager.getViewManagerConfig('RNComponentView').Commands
                .stopFromManager
            : command,
          undefined
        );
      }
    };

    const getErrorString = (nativeEvent: any) => {
      let error = '';
      if (nativeEvent.hasOwnProperty('error')) {
        error = nativeEvent.error;
      }
      return error;
    };

    return (
      <RNComponentViewManager
        assetId={props.assetId}
        style={props.style}
        ref={componentRef}
        start={() => start()}
        stop={() => stop()}
        onReady={(_event: SyntheticEvent) =>
          props.onReady && props.onReady()
        }
        onError={(event: SyntheticEvent) =>
          props.onError && props.onError(getErrorString(event.nativeEvent))
        }
      />
    );
  });

  RNComponent.displayName = 'RNComponent';
  export default RNComponent;

Cocoapods tips

As well as being aware of general iOS/tvOS and Android best practices it is as well to be aware of the power of Cocoapods on the iOS/tvOS side. Your iOS/tvOS project will already have a .podfile and you will occasionally have cause to make some specific changes here that can solve issues you may be having.

User defined build types

You may find yourself in the position of having to integrate several different types of third-party tools in order to get a plugin up and running. This may mean that you need to mix dynamic and/or static frameworks and/or libraries in your project. Unfortunately, there is no support for this at present in Cocopods. But, fortunately, there is a Cocopods plugin available that will allow you to do just this! This means you can write things like this in your pod file:

plugin 'cocoapods-user-defined-build-types'

enable_user_defined_build_types!

target "PlayerApp" do
  pod 'PlayerUI', :build_type => :static_framework
  pod 'Player', :build_type => :static_framework
  pod "SwiftyJSON", :build_type => :dynamic_framework

Combining package managers

Occasionally you might end up being forced to use both Cocoapods and another package manager, such as Carthage. It's not actually that complicated but you can get some confusing error messages along the way.

To take Carthage as an example, you can simply set up your Carthage installation in the iOS root directory of the generated plugin according to the standard way of working for Carthage, but be sure to (in your .podspec):

exclude certain Carthage files such as:

  s.exclude_files = "ios/Carthage", "ios/cartfile", "ios/Cartfile.resolved", "ios/carthage.sh"

Make sure that the generated frameworks are indicated in your .podspec:

s.vendored_frameworks = "ios/Frameworks/Player.xcframework"

(Here we have copied the Carthage framework from the build directory to a Frameworks directory in our plugin).

Turbomodules

There are changes coming in React Native that will have a big impact on how to implement native modules...in fact, these changes are already in place to an extent that it is possible to build modules using this technology. At React Native EU this year, a talk was given that described a camera library that utilises the power of JSI. This gives some insight into how turbo modules will work when they are released. This promises great things for the future of React Native modules.

Alan Allard is a developer at Eyevinn Technology, the European leading independent consultancy firm specializing in video technology and media distribution.

Working with VideoToolbox for more control over video encoding and decoding - Part 2.

Alan Allard — Fri, 27 Aug 2021 15:49:16 +0000

In the last article, I presented the structure of a MacOS app to encode and decode video using VideoToolbox. This time, I would like to focus on the actual encoding process. I will also look at ways to improve on and restructure the existing code, bearing in mind the various ways you could build such a project.

As we left the project in the last article, we had a working macOS app that encoded video from the camera and then sent it off directly to be encoded. In that naive implementation, we are essentially doing the same thing as - or even less than, really - what AVFoundation can do for you. AVFoundation provides access to hardware-accelerated compression and decompression by default. What you don't get there, though, is the ability to fine-tune and customise the encoding and decoding. That is the whole point of VideoToolbox. So let's see how to access the details of the encoding process. The changes I have made in conjunction with this article were merged in from my branch encoder-improvements. Once again, I have referred to existing projects for some of the approach I have taken. In this case, I have updated the encoder code in this iOS project from Objective-C to Swift 5.

The first improvement to make is to set up our encoder before we actually start using it.

func prepareToEncodeFrames() {
        let encoderSpecification = [
            kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder: true as CFBoolean
        ] as CFDictionary
        let status = VTCompressionSessionCreate(allocator: kCFAllocatorDefault, width: self.width, height: self.height, codecType: kCMVideoCodecType_H264, encoderSpecification: encoderSpecification, imageBufferAttributes: nil, compressedDataAllocator: nil, outputCallback: outputCallback, refcon: Unmanaged.passUnretained(self).toOpaque(), compressionSessionOut: &session)
        print("H264Coder init \(status == noErr) \(status)")
        // This demonstrates setting a property after the session has been created
        guard let compressionSession = session else { return }
        VTSessionSetProperty(compressionSession, key: kVTCompressionPropertyKey_RealTime, value: kCFBooleanTrue)
        VTSessionSetProperty(compressionSession, key: kVTCompressionPropertyKey_ProfileLevel, value: kVTProfileLevel_H264_Main_AutoLevel)
        VTSessionSetProperty(compressionSession, key: kVTCompressionPropertyKey_AllowFrameReordering, value: kCFBooleanFalse)
        VTSessionSetProperty(compressionSession, key: kVTCompressionPropertyKey_ExpectedFrameRate, value: CFNumberCreate(kCFAllocatorDefault, CFNumberType.intType, &self.fps))
        VTCompressionSessionPrepareToEncodeFrames(compressionSession)
    }

Here we are wrapping the method VTCompressionSessionPrepareToEncodeFrames(compressionSession) which you can read more about in the Apple docs.
We've also taken the opportunity to set some session properties before we begin encoding. Most are set inline here, but frames-per-second is an example of a property exposed in the class, set in appDelegate when creating the encoder:

    // Create encoder here (at the expense of dynamic setting of height and width)
    encoder = H264Encoder(width: 1280, height: 720, callback: { encodedBuffer in
      // self.sampleBufferNoOpProcessor(encodedBuffer) // Logs the buffers to the console for inspection
      // self.decodeCompressedFrame(encodedBuffer) // uncomment to see decoded video
    })
    encoder?.delegate = self
    encoder?.fps = 15
    encoder?.prepareToEncodeFrames()

(As commented here, in the previous code we could dynamically set the width and height based on the incoming buffer of data. I sacrificed that here for the sake of other demonstrations, but you may need to find a way to keep that functionality in another application.)

I want to receive the compressed data in my appDelegate, so I can do something with it later. To this end I created an extension for two delegate functions I created in the encoder. First, the encoder protocol:

protocol H264EncoderDelegate: AnyObject {
    func dataCallBack(_ data: Data!, frameType: FrameType)
    func spsppsDataCallBack(_ sps:Data!, pps: Data!)
}

and the extension:

extension AppDelegate : H264EncoderDelegate {
    func dataCallBack(_ data: Data!, frameType: FrameType) {
        let byteHeader:[UInt8] = [0,0,0,1]
        var byteHeaderData = Data(byteHeader)
        byteHeaderData.append(data)
        // Could decode here
        // H264Decoder.decode(byteHeaderData)
    }

    func spsppsDataCallBack(_ sps: Data!, pps: Data!) {
        let spsbyteHeader:[UInt8] = [0,0,0,1]
        var spsbyteHeaderData = Data(spsbyteHeader)
        var ppsbyteHeaderData = Data(spsbyteHeader)
        spsbyteHeaderData.append(sps)   
        ppsbyteHeaderData.append(pps)
        // Could decode here
        // H264Decoder.decode(spsbyteHeaderData)
        // H264Decoder.decode(ppsbyteHeaderData)
    }
}

We'll discuss those byte headers shortly ;)

While we're there, to keep things tidy, we may as well make the existing AVManagerDelegate into an extension too:

// MARK: - AVManagerDelegate
extension AppDelegate : AVManagerDelegate {
    func onSampleBuffer(_ sampleBuffer: CMSampleBuffer) {
        cameraView.render(sampleBuffer)
        encoder?.encode(sampleBuffer)
    }
}

In short, we are going to keep the existing buffer-to-buffer encoding for now and extend the callback method such that it will also call the above callbacks each time. And just to keep an eye on the overall plan, what we are doing here is preparing (encoding) the data as an elementary stream. The data we receive in our sampleBuffer is in AVCC format, whereas the format we want out is an elementary stream in the so-called Annex B format. Everything we do in the callback has to do with converting from the AVCC format to the Annex B format, while allowing us to tweak the details of that process in various ways.

If a sample buffer contains a keyframe we also know it will contain data describing how the decoder should handle these frames when it receives them.

So the first part of our callback looks like this:

let outputCallback: VTCompressionOutputCallback = { refcon, sourceFrameRefCon, status, infoFlags, sampleBuffer in
        guard let refcon = refcon,
              status == noErr,
              let sampleBuffer = sampleBuffer else {
            print("H264Coder outputCallback sampleBuffer NULL or status: \(status)")
            return
        }

        if (!CMSampleBufferDataIsReady(sampleBuffer))
        {
            print("didCompressH264 data is not ready...");
            return;
        }
        let encoder: H264Encoder = Unmanaged<H264Encoder>.fromOpaque(refcon).takeUnretainedValue()
        if(encoder.shouldUnpack) {
            var isKeyFrame:Bool = false

    //      Attempting to get keyFrame
            guard let attachmentsArray:CFArray = CMSampleBufferGetSampleAttachmentsArray(sampleBuffer, createIfNecessary: false) else { return }
            if (CFArrayGetCount(attachmentsArray) > 0) {
                let cfDict = CFArrayGetValueAtIndex(attachmentsArray, 0)
                let dictRef: CFDictionary = unsafeBitCast(cfDict, to: CFDictionary.self)

                let value = CFDictionaryGetValue(dictRef, unsafeBitCast(kCMSampleAttachmentKey_NotSync, to: UnsafeRawPointer.self))
                if(value == nil) {
                    isKeyFrame = true
                }
            }

(Note the encoder property shouldUnpack, which simply wraps all this unpacking code in an if-statement so you can activate it as required).
The callback receives a sample buffer on each call, which we need to check is ready to be processed. Next we need to know what kind of data is in the buffer. To do this, we need to take a look at what are known as "attachments" on the buffer - essentially an array of dictionaries providing information about the data. You can see a list of the many attachment keys used here. The one we need is kCMSampleAttachmentKey_NotSync, the absence of which indicates that the data we are dealing with is a keyframe. (And as you can see, it can get a bit messy working with CFDictionaries in Swift...)

So to the data contained in a keyframe sample data; once we know that we are dealing with a keyframe, we can extract two sets of data from our buffer. These are the Sequence Parameter Set and the Picture Parameter Set. A buffer with a keyframe will have both of these sets of data.

This is how we go about extracting these and sending them to the SPS and PPS callback:

            if(isKeyFrame) {
                var description: CMFormatDescription = CMSampleBufferGetFormatDescription(sampleBuffer)!
                // First, get SPS
                var sparamSetCount: size_t = 0
                var sparamSetSize: size_t = 0
                var sparameterSetPointer: UnsafePointer<UInt8>?
                var statusCode: OSStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, parameterSetIndex: 0, parameterSetPointerOut: &sparameterSetPointer, parameterSetSizeOut: &sparamSetSize, parameterSetCountOut: &sparamSetCount, nalUnitHeaderLengthOut: nil)

                if(statusCode == noErr) {
                    // Then, get PPS
                    var pparamSetCount: size_t = 0
                    var pparamSetSize: size_t = 0
                    var pparameterSetPointer: UnsafePointer<UInt8>?
                    var statusCode: OSStatus = CMVideoFormatDescriptionGetH264ParameterSetAtIndex(description, parameterSetIndex: 0, parameterSetPointerOut: &pparameterSetPointer, parameterSetSizeOut: &pparamSetSize, parameterSetCountOut: &pparamSetCount, nalUnitHeaderLengthOut: nil)
                    if(statusCode == noErr) {
                        var sps = NSData(bytes: sparameterSetPointer, length: sparamSetSize)
                        var pps = NSData(bytes: pparameterSetPointer, length: pparamSetSize)
                        encoder.delegate?.spsppsDataCallBack(sps as Data, pps: pps as Data)
                    }
                }
            }

The decoder will know how to handle these correctly once it receives them (the idea being that we call the decoder from said callback, as indicated earlier). We can also come back to the byte headers I referred to earlier - for elementary streams, all so-called NAL units (each packet of data, basically) must begin with a byte header array of [0,0,0,1]. Thus, we prepend the data with that header.

After that we can handle the actual image data, which we send to our callback with an indication whether it is a keyframe (a.k.a. PFrame) or not (Iframe):

            var dataBuffer: CMBlockBuffer = CMSampleBufferGetDataBuffer(sampleBuffer)!
            var length: size_t = 0
            var totalLength: size_t = 0
            var bufferDataPointer: UnsafeMutablePointer<Int8>?
            var statusCodePtr: OSStatus = CMBlockBufferGetDataPointer(dataBuffer, atOffset: 0, lengthAtOffsetOut: &length, totalLengthOut: &totalLength, dataPointerOut: &bufferDataPointer)
            if(statusCodePtr == noErr) {
                var bufferOffset: size_t = 0
                let AVCCHeaderLength: Int = 4
                while(bufferOffset < totalLength - AVCCHeaderLength) {
                    // Read the NAL unit length
                    var NALUnitLength: UInt32 = 0
                    memcpy(&NALUnitLength, bufferDataPointer! + bufferOffset, AVCCHeaderLength)
                    //Big-Endian to Little-Endian
                    NALUnitLength = CFSwapInt32BigToHost(NALUnitLength)

                    var data = NSData(bytes:(bufferDataPointer! + bufferOffset + AVCCHeaderLength), length: Int(Int32(NALUnitLength)))
                    var frameType: FrameType = .FrameType_PFrame
                    var dataBytes = Data(bytes: data.bytes, count: data.length)
                    if((dataBytes[0] & 0x1F) == 5) {
                        // I-Frame
                        print("is IFrame")
                        frameType = .FrameType_IFrame
                    }

                    encoder.delegate?.dataCallBack(data as Data, frameType: frameType)
                    // Move to the next NAL unit in the block buffer
                    bufferOffset += AVCCHeaderLength + size_t(NALUnitLength);
                }
            }
        }

There are a few important details here. When building an elementary stream, we need to include the length of each packet as a requirement. So we need to get that length - see the comment "Read the NAL unit length" in the above code to see how we do that with memcpy...then we also need to convert the data from Big-endian to Little-endian - for which Core Foundation provides the method CFSwapInt32BigToHost(NALUnitLength). Further down (the last line) we can use that length to move to the next NAL unit in the buffer.

One final detail is the analysis of the dataBytes variable as a convenient way top know if we are dealing with an iframe or not - info that we use to update the frameType variable with for use in the dataCallback.

So at this stage, we have acquired everything we need to process the elementary stream of data. We could send this data to a decoder of our choice now. In the next article we will decode the data using an updated version of our decoder class. Part 3 is on its way...

Those of you keen to dig deeper may find this SO discussion and this one useful.

Here, again, is a link to the previous article on this subject.

The repository accompanying this post is available here.

Alan Allard is a developer at Eyevinn Technology, the European leading independent consultancy firm specializing in video technology and media distribution.

Working with VideoToolbox for more control over video encoding and decoding.

Alan Allard — Fri, 20 Aug 2021 09:18:51 +0000

Apple's framework VideoToolbox has been around for many years now. This year it had an update to handle low-latency encoding, among other things, so it's certainly worth getting familiar with the framework both generally and in order to make use of the newer features. Unfortunately, there are not a great number of examples on how to do this and the official documentation is not much more than a collection of header files. Hopefully we can provide some useful examples in this and future articles.

VideoToolbox is a fairly low-level framework, positioned as it is below AVKit and AVFoundation but on top of CoreMedia and CoreVideo.

You can do many things with both AVKit and AVFoundation. AVKit provides a lot of functionality at the view level. AVFoundation can also encode and decode streams of compressed video data (in the example accompanying this post, there is a display view based on AVFoundation's AVSampleBufferDisplayLayer). Sometimes, however, you will want greater control over the processes of compression and decompression. This is where VideoToolbox can be useful.

As an introduction to VideoToolbox, I will walk through the process of building a simple camera capture view that can display the captured data using either a Metal view layer or an AVSampleBufferDisplayLayer. The captured data is then also encoded with the help of VideoToolbox. Once a stream of data is being produced in this way, a next step would be to use that stream in some fashion - streaming it over a network, for example. And on the receiving side, VideoToolbox can then decode the data for display on that device. In fact, there is an example of the decoding process included in the code, which simply takes the encoded buffers and decodes them immediately. Not a realistic example, but useful as a reference.

Credit where it's due before we go further; I based my code on this repo which I initially forked and adjusted to taste. I've since rebuilt the project from scratch but the basic inspiration for the code remains. There is a useful WWDC video on VideoToolbox (seven years old now!) and the more recent one on low-latency encoding can be found here . I will keep the descriptions here mostly to the VideoToolbox-related code and just provide a high-level description of the steps. The accompanying code should suffice as a more detailed example.

This project can be built using SwiftUI, so the first step is to create a new Xcode macOS app project selecting SwiftUI as the interface, but AppKit App Delegate for the Life cycle option. If you're taking the step-by-step approach, you can then delete the generated ContentView as we will be using our own VideoView.swift:

struct VideoView: View {

    let displayView = RenderViewRep() // Metal or AVDisplayViewRep to use AVSampleBufferDisplayLayer
    // let displayView = AVDisplayViewRep()

    var body: some View {
        displayView
            .frame(maxWidth: .infinity, maxHeight: .infinity)
    }

    func render(_ sampleBuffer: CMSampleBuffer)  {
        displayView.render(sampleBuffer)
    }
}

struct CameraView_Previews: PreviewProvider {
    static var previews: some View {
        VideoView()
    }
}

Notice that we have the option to use either a metal layer or an AVSampleBufferDisplayLayer, both are available in the sample code. We use this VideoView in our AppDelegate and create a window in which to use the view, like so:

let cameraView = VideoView()
var cameraWindow: NSWindow!

and

    // Create the windows and set the content view.
    cameraWindow = NSWindow(
        contentRect: NSRect(x: 0, y: 0, width: 480, height: 300),
        styleMask: [.titled, .closable, .miniaturizable, .resizable, .fullSizeContentView],
        backing: .buffered, defer: false)
    cameraWindow.isReleasedWhenClosed = false
    cameraWindow.center()
    cameraWindow.setTitleWithRepresentedFilename("Camera view")
    cameraWindow.contentView = NSHostingView(rootView: cameraView)
    cameraWindow.makeKeyAndOrderFront(nil)

Here we set the contentView to a NSHostingView in order to use SwiftUI.

To handle the camera capture, we can build an AVManager class that implements AVCaptureVideoDataOutputSampleBufferDelegate:

import AVFoundation

protocol AVManagerDelegate: AnyObject {

    func onSampleBuffer(_ sampleBuffer: CMSampleBuffer)

}

class AVManager: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {

    // MARK: - Properties

    weak var delegate: AVManagerDelegate?

    var session: AVCaptureSession!

    private let sessionQueue = DispatchQueue(label:"se.eyevinn.sessionQueue")

    private let videoQueue = DispatchQueue(label: "videoQueue")

    private var connection: AVCaptureConnection!

    private var camera: AVCaptureDevice?

    func start() {
        requestCameraPermission { [weak self] granted in
            guard granted else {
                print("no camera access")
                return
            }
            self?.setupCaptureSession()
        }
    }

    private func setupCaptureSession() {
        sessionQueue.async {
            let deviceTypes: [AVCaptureDevice.DeviceType] = [
                .builtInWideAngleCamera
            ]
            let discoverySession = AVCaptureDevice.DiscoverySession(
                deviceTypes: deviceTypes,
                mediaType: .video,
                position: .front)
            let session = AVCaptureSession()
            session.sessionPreset = .vga640x480

            guard let camera = discoverySession.devices.first,
                  let format = camera.formats.first(where: {
                    let dimens = CMVideoFormatDescriptionGetDimensions($0.formatDescription)
                    return dimens.width * dimens.height == 640 * 480
                  }) else { fatalError("no camera of camera format") }

            do {
                try camera.lockForConfiguration()
                camera.activeFormat = format
                camera.unlockForConfiguration()
            } catch {
                print("failed to set up format")
            }
            self.camera = camera

            session.beginConfiguration()
            do {
                let videoIn = try AVCaptureDeviceInput(device: camera)
                if session.canAddInput(videoIn) {
                    session.addInput(videoIn)
                } else {
                    print("failed to add video input")
                    return
                }
            } catch {
                print("failed to initialized video input")
                return
            }

            let videoOut = AVCaptureVideoDataOutput()
            videoOut.videoSettings = [ kCVPixelBufferPixelFormatTypeKey as String:  Int(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange)
            ]
            videoOut.alwaysDiscardsLateVideoFrames = true
            videoOut.setSampleBufferDelegate(self, queue: self.videoQueue)
            if session.canAddOutput(videoOut) {
                session.addOutput(videoOut)
            } else {
                print("failed to add video output")
                return
            }
            session.commitConfiguration()
            session.startRunning()
            self.session = session
        }
    }

    private func requestCameraPermission(handler: @escaping (Bool) -> Void) {
        switch AVCaptureDevice.authorizationStatus(for: .video) {
            case .authorized: // The user has previously granted access to the camera.
                handler(true)

            case .notDetermined: // The user has not yet been asked for camera access.
                AVCaptureDevice.requestAccess(for: .video) { granted in
                    handler(granted)
                }

            case .denied, .restricted: // The user can't grant access due to restrictions.
                handler(false)
        @unknown default:
            fatalError()
        }
    }

    // MARK: - AVCaptureVideoDataOutputSampleBufferDelegate

    func captureOutput(_ output: AVCaptureOutput,
                       didOutput sampleBuffer: CMSampleBuffer,
                       from connection: AVCaptureConnection) {
        delegate?.onSampleBuffer(sampleBuffer)
    }

    // MARK: - Types

    enum AVError: Error {

        case noCamera
        case cameraAccess

    }
}

This means, by the way, that we will need to add an entitlement to our entitlements file for Camera set to YES, and we will also need to add the usual usage text for camera to the info.plist file, stating the reason why we need to use the camera.

Notice the method onSampleBuffer, where we render the current buffer to the camera view and then send it on to the encoder to compress the data according to our needs. The encoder in this case is the H264Encoder class. The buffers we are working with here are, as you can see, CMSampleBuffers. These are typically, but not necessarily, compressed sample data. In the case of the camera capture output they are CoreVideo pixel buffers, but these are compressed after running through the H264Encoder. Taking a look at the H264Encoder class, this is where VideoToolbox is used. To encode we need to create a "compression session" using VideoToolbox's VTCompressionSessionCreate:

    init(width: Int32, height: Int32, callback: @escaping (CMSampleBuffer) -> Void) {
        self.callback = callback
      let encoderSpecification = [
          kVTVideoEncoderSpecification_RequireHardwareAcceleratedVideoEncoder: true as CFBoolean
      ] as CFDictionary
        let status = VTCompressionSessionCreate(allocator: kCFAllocatorDefault, width: width, height: height, codecType: kCMVideoCodecType_H264, encoderSpecification: encoderSpecification, imageBufferAttributes: nil, compressedDataAllocator: nil, outputCallback: outputCallback, refcon: Unmanaged.passUnretained(self).toOpaque(), compressionSessionOut: &session)
        print("H264Coder init \(status == noErr) \(status)")
    }

Here, the session, of type VTCompressionSession, is a member of the H264Encoder class. We can pass in an encoderSpecification on creation - this gives us control over hardware acceleration and, in the latest update, low-latency control. The init also assigns the necessary callback of type VTCompressionOutputCallback which provides the compressed result.

With these building blocks in place, we are able to capture camera output and encode the result. I have also included an example of setting properties on the compression session, after it has been created. In the example I indicate that the stream is in realtime:

        guard let compressionSession = session else { return }
        status = VTSessionSetProperty(compressionSession, key: kVTCompressionPropertyKey_RealTime, value: kCFBooleanTrue)

Next steps

There are additional methods available in VideoToolbox for preparing CMSampleBuffers for network transmission as NAL units (Network Abstraction Layer). This would be a good next step for this project, as would an investigation into the details of decoding sessions in a similar fashion. Watch this space...

The repository accompanying this post is available here.

Alan Allard is a developer at Eyevinn Technology, the European leading independent consultancy firm specializing in video technology and media distribution.

Introducing hls-truncate, the new tool in our collection of VOD manipulators!

Alan Allard — Wed, 24 Feb 2021 10:17:46 +0000

Over the last year or so, we at Eyevinn Technology have been developing a series of tools for the manipulation of HLS files. There is an increasing interest for presenting VOD content as live channels. That approach makes it possible for companies to provide themed channels and fixed schedules based on existing video material. Among other things, this can be a way to make use of older content in new packaging that interests consumers.

We have built several open source tools to provide every aspect of delivering a livestream from pre-existing content. As we develop these tools further, we often see that a need arises for convenient helper utilities. A previous example was hls-repeat which takes a HLS VOD as an input and outputs a new VOD that is repeated the required number of times. This can be useful when working dynamically with slates, for example, for display during schedule gaps. Another example, hls-splice is discussed in one of our previous articles.

Last week we released hls-truncate, a similar tool which also takes a HLS VOD as an input. In this case, it shortens the VOD in question to the nearest segment, providing that as the output. Again, this has proven to be useful for adjusting slate lengths on-the-fly for customers.

Wherever one wishes to use a VOD, it's as simple as making a call to create the new VOD:

const hlsVod = new HLSTruncateVod('http://testcontent.eyevinn.technology/slates/30seconds/playlist.m3u8', 4);
hlsVod.load()
.then(() => {
  const mediaManifest = hlsVod.getMediaManifest(3496000);
  console.log(mediaManifest);
});

Now all bandwidths are available at the truncated length. This all simplifies a code base considerably and has the added advantage of you knowing that a number of fiddly edge cases are taken care of and backed up by a good suite of tests, that can be developed in isolation over time.

Read more on our repo
Release 0.1.0 is up on npm now: @eyevinn/hls-truncate

DEV Community: Alan Allard

Exploring video generators in FFMPEG...

The coreimagesrc generator

The gradients generator

The mandelbrot generator

The mptestsrc filter

The mysterious section '15.10'...

Various other generators...

Generators with plug-in capability

Exploring OCR and text-to-speech in FFMPEG...

Using the ocr filter

Introducing the flite filter

Combining OCR and text-to-speech...

About the cover image

Working with audio in ffmpeg - Part 2

Working with audio in ffmpeg

Audio conversion

filters

Complex filters

Building React Native UI Components and Modules in 2021

Intro

Creating the project

Integrating your platform-specific packages

Module or UI Component

View controller vs view

Properties and methods in iOS

Setup iOS/tvOS/Android

Other properties and methods in Android

Javascript side

Cocoapods tips

User defined build types

Combining package managers

Turbomodules

Working with VideoToolbox for more control over video encoding and decoding - Part 2.

Working with VideoToolbox for more control over video encoding and decoding.

Introducing hls-truncate, the new tool in our collection of VOD manipulators!

The `coreimagesrc` generator

The `gradients` generator

The `mandelbrot` generator

The `mptestsrc` filter