DEV Community: Eyevinn Video Dev-Team Blog

Creating a web video application

Jonas Birmé — Wed, 05 Feb 2025 20:12:57 +0000

As a continuation to the blog post Stream Video with Open Web Services we will in this post develop a web application for streaming the video we have created.

In this guide we will use the build tool Vite to setup a web project for a web application based on the React framework.

Setup the React Typescript project

Run the following command to scaffold a new project based on React template.

% npm create vite@latest my-video-app -- --template react-ts

Then run:

% cd my-video-app
% npm install
% npm run dev

Open your browser to http://localhost:5173/ and you will see the following template.

Create a Video Player component

Now it is time to add a video player component. We will use the open source web player from Eyevinn in this example.

First install the library.

% npm install --save @eyevinn/web-player

Then we will create a folder for our components

% mkdir src/components

In this folder we will create a file called Player.tsx containing the following code.

import webplayer from '@eyevinn/web-player';
import { useEffect, useRef } from 'react';

import '@eyevinn/web-player/dist/webplayer.css';

export default function Player({
  src,
  autoplay = false
}: {
  src: string;
  autoplay?: boolean;
}) {
  const elRef = useRef<HTMLDivElement>(null);
  useEffect(() => {
    if (elRef.current) {
      const instance = webplayer(elRef.current, {});
      instance.player.load(src, autoplay);
      return () => {
        instance.destroy();
      };
    }
  }, [src]);

  return <div ref={elRef} className="h-full" />;
}

Use the Player component

Now we can use this Player component on our main page. Replace the contents in the file src/App.tsx and replace the src property with the URL to the video you created in the previous blog.

import { useState } from 'react'
import './App.css'
import Player from './components/Player'

function App() {
  const [count, setCount] = useState(0)

  return (
    <>
      <h1>Vite + React + Eyevinn OSC</h1>
      <div className="card">
        <Player src="https://eyevinnlab-devguide.minio-minio.auto.prod.osaas.io/devguide/VINN/52e124b8-ebe8-4dfe-9b59-8d33abb359ca/index.m3u8"/>
        <button onClick={() => setCount((count) => count + 1)}>
          count is {count}
        </button>
      </div>
    </>
  )
}

export default App

Now when updating the browser you should see the video player with your video.

Make the web application available online

Now you have a working application running locally on your computer and it is time to make it available online. To make it available online we will be using Eyevinn Open Source Cloud to host this website as described in a blog post on Eyevinn Open Source Cloud blog.

First modify the vite config file vite.config.ts to use a relative base url (default expects the root /).

import { defineConfig } from 'vite'
import react from '@vitejs/plugin-react'

// https://vite.dev/config/
export default defineConfig({
  base: './',
  plugins: [react()],
})

Then we can build the application by running the following command.

% npm run build

Go to the web console of Eyevinn Open Source Cloud and obtain the access token available under Settings. Copy the token and store it in the environment variable OSC_ACCESS_TOKEN.

% export OSC_ACCESS_TOKEN=YOUR_TOKEN

Now run the following command to deploy the build. We name the website “video” which will be the bucket where the files are placed.

% npx @osaas/cli@latest web publish -s video dist/

Now your web application is available online at the address returned by the command above. In our example it is available at https://eyevinnlab-video.minio-minio.auto.prod.osaas.io/video/index.html

Conclusion

This gives you an example of how you can build a video streaming application based on open source without having to host everything in-house. From video processing to the deployment of the web application based on open web services in Eyevinn Open Source Cloud.

As everything is based on open source you always have the option to bring it in-house if you want to.

Stream video with open web services

Jonas Birmé — Thu, 30 Jan 2025 23:24:44 +0000

Upload and play back your video files in your application using open web services in Eyevinn Open Source Cloud in five minutes or less.

Eyevinn Open Source Cloud was developed to reduce the barrier to getting started with open source and at the same time contribute to a sustainable model for open source by giving back a share of the revenue to the creator.

In this guide

Get an API Access Token and setup project.
Setup a video processing pipeline for streaming.
Upload and process video

Prerequisites

An Eyevinn Open Source Cloud account.
An active Business plan or higher. See pricing for more details.

Get an API Access Token and setup project

Navigate to Settings / API in the Eyevinn Open Source Cloud web console.

Copy this token and store in your shell's environment in the environment variable OSC_ACCESS_TOKEN.

% export OSC_ACCESS_TOKEN=<access-token-copied-above>

Setup a NodeJS project.

% mkdir vod
% cd vod
% npm init

Install the Javascript client SDK.

% npm install --save @osaas/client-core @osaas/client-transcode

Create a file called vod.js and open it in your favorite editor.

Setup video processing pipeline

Add the following code to your file to setup the video processing pipeline.

const { Context } = require('@osaas/client-core');
const { createVodPipeline, createVod } = require('@osaas/client-transcode');

async function setup(context) {
  const pipeline = await createVodPipeline('devguide', context);
  return pipeline;
}

async function main() {
  const ctx = new Context();
  const pipeline = await setup(ctx);
}

main();

Run the script.

% node vod.js

After a few minutes it will have created a video processing pipeline.

Upload and process video

Now we can use the pipeline we created to upload and process a video. Here is a demo video you can use:

https://testcontent.eyevinn.technology/mp4/VINN.mp4

We will add the following to the main function.

  const vod = await createVod(pipeline,
    'https://testcontent.eyevinn.technology/mp4/VINN.mp4',
    ctx
  );
  console.log(vod);

Now when we run the script it will return the following.

% node vod.js
{
  id: '52e124b8-ebe8-4dfe-9b59-8d33abb359ca',
  vodUrl: 'https://eyevinnlab-devguide.minio-minio.auto.prod.osaas.io/devguide/VINN/52e124b8-ebe8-4dfe-9b59-8d33abb359ca/index.m3u8'
}

When the video processing has completed we can now paste the vodURL in a video player.

Next post: Creating a web video application

Transcode video with open web services

Jonas Birmé — Thu, 30 Jan 2025 22:49:38 +0000

Video file transcoding in your media supply chain using open web services in Eyevinn Open Source Cloud in five minutes or less.

In this guide

Get an API Access Token and setup project.
Setup a storage for storing the transcoded video files.
Setup a video transcoder.
Transcode a video file

Prerequisites

An Eyevinn Open Source Cloud account. Sign up for free if you don't already have an account. No credit card required for free plan.

Access to 1 open web service is included in the free plan. If you choose to use an open web service for storing the transcoded video files you need startup plan that includes access to 2 open web services. See pricing for more details.

Get an API Access Token and setup project

Navigate to Settings / API in the Eyevinn Open Source Cloud web console.

Copy this token and store in your shell's environment in the environment variable OSC_ACCESS_TOKEN.

% export OSC_ACCESS_TOKEN=<access-token-copied-above>

Setup a NodeJS project.

% mkdir transcode
% cd transcode
% npm init

Install the Javascript client SDK.

% npm install --save @osaas/client-core @osaas/client-services @osaas/client-transcode

Create a file called transcode.js and open it in your favorite editor.

Add the following code in the file.

const { Context } = require('@osaas/client-core');

const ctx = new Context();

You can now test running this script.

% node transcode.js

Setup a storage for storing transcoded files

You can skip this step if you already have an S3 compatible storage that you want to use.

Install MinIO client library

% npm install --save minio

Add a new function called setup() that takes the Context as argument.

const { Context, waitForInstanceReady } = require('@osaas/client-core');
const { getMinioMinioInstance, createMinioMinioInstance } = require('@osaas/client-services');
const Minio = require('minio');

const delay = (ms) => new Promise((res) => setTimeout(res, ms));

async function setup(context) {
  console.log('Setting up storage');

  // Check if we already have a storage service running
  let storage = await getMinioMinioInstance(context, 'devguide');
  if (!storage) {
    storage = await createMinioMinioInstance(context, {
      name: 'devguide',
      RootUser: 'admin',
      RootPassword: 'abC12345678'
    });
    await waitForInstanceReady('minio-minio', 'devguide', context);
    await delay(2000);
  }
  // Create a bucket on the storage service if it does not exists
  const client = new Minio.Client({
    endPoint: new URL(storage.url).hostname,
    accessKey: 'admin',
    secretKey: 'abC12345678',
  });
  const buckets = await client.listBuckets();
  if (!buckets.find((bucket) => bucket.name === 'output')) {
    await client.makeBucket('output');
  }
}

async function main() {
  const ctx = new Context();
  await setup(ctx);
}

main();

When we now run the script again it will create a storage service and a bucket.

Setup video transcoder

Next step is to add to the setup function to create a video transcoder instance. If you already have an existing S3 bucket you replace the S3 credentials below with the access credentials to your bucket.

const { getEncoreInstance } = require('@osaas/client-services');
...

async function setup(context) {
  ...
  let transcoder = await getEncoreInstance(context, 'devguide');
  if (!transcoder) {
    transcoder = await createEncoreInstance(context, {
      name: 'devguide',
      s3AccessKeyId: 'admin',
      s3SecretAccessKey: 'abC12345678',
      s3Endpoint: storage.url
    });
  }
}

Transcode a video file

The last step is to transcode a video file and here is a demo video you can use:

https://testcontent.eyevinn.technology/mp4/VINN.mp4

Add to the main function after setup the following code

const { transcode } = require('@osaas/client-transcode');
...
async function main() {
  const ctx = new Context();
  await setup(ctx);
  await transcode(ctx, {
    encoreInstanceName: 'devguide',
    inputUrl: new URL('https://testcontent.eyevinn.technology/mp4/VINN.mp4'),
    externalId: 'vinn',
    outputUrl: new URL('s3://output/')
  });
}

And then we wait for the transcoding process to complete. To be notified of the progress of the transcoding process you can provide a webhook URL, for example:

await transcode(ctx, {
    encoreInstanceName: 'devguide',
    inputUrl: new URL('https://testcontent.eyevinn.technology/mp4/VINN.mp4'),
    externalId: 'vinn',
    outputUrl: new URL('s3://output/')
    callBackUrl: 'https://example.com/webhook'
  });

We can verify that the transcoded files are available in the output bucket with an S3 client. For example:

% AWS_ACCESS_KEY_ID=admin AWS_SECRET_ACCESS_KEY=abC12345678 \
  aws --endpoint https://eyevinnlab-devguide.minio-minio.auto.prod.osaas.io \
  s3 ls s3://output/
2025-01-30 23:46:23     531182 vinn_12x20_160x90_thumbnail_map.jpg
2025-01-30 23:46:24    1755553 vinn_STEREO.mp4
2025-01-30 23:46:24      71512 vinn_thumb01.jpg
2025-01-30 23:46:23      45780 vinn_thumb02.jpg
2025-01-30 23:46:24      48038 vinn_thumb03.jpg
2025-01-30 23:46:25   18736955 vinn_x264_1312.mp4
2025-01-30 23:46:24   29213093 vinn_x264_2069.mp4
2025-01-30 23:46:25   42571864 vinn_x264_3100.mp4
2025-01-30 23:46:23    5627650 vinn_x264_324.mp4
2025-01-30 23:46:24   12033920 vinn_x264_806.mp4

Sever-Guided Ad Insertion Made Easy.

Kun Wu — Tue, 05 Nov 2024 14:31:04 +0000

In today’s streaming landscape, delivering personalized, seamless ad experiences is key, and Server-Guided Ad Insertion (SGAI) is leading the charge. Unlike traditional Client-Side Ad Insertion (CSAI), where the user’s device manages ads, SGAI centralizes control at the server level. It decides the timing, format, and content of ads based on user data, ensuring ad relevance without relying on the client’s system. By connecting to ad decision servers (ADS), SGAI generates tailored ad breaks in real-time, keeping viewers engaged with ads that suit their preferences and demographics.

How SGAI Works

Servers can schedule interstitials by placing EXT-X-DATERANGE tags in Media Playlists, allowing flexibility in ad placements. These interstitials, which must be VOD assets, can be inserted at any point in a primary asset, whether VOD or live, including Low-Latency HLS streams.

#EXTM3U
#EXT-X-TARGETDURATION:4
#EXT-X-MEDIA-SEQUENCE:11
#EXT-X-PROGRAM-DATE-TIME:2024-10-30T12:52:27.853+0100
#EXTINF:4,
fileSequence11.ts
#EXT-X-PROGRAM-DATE-TIME:2024-10-30T12:52:31.853+0100
#EXTINF:4,
fileSequence12.ts
#EXT-X-PROGRAM-DATE-TIME:2024-10-30T12:52:35.853+0100
#EXTINF:4,
fileSequence13.ts
#EXT-X-PROGRAM-DATE-TIME:2024-10-30T12:52:39.853+0100
#EXTINF:4,
fileSequence14.ts
#EXT-X-PROGRAM-DATE-TIME:2024-10-30T12:52:43.853+0100
#EXTINF:4,
fileSequence15.ts
#EXT-X-DATERANGE:ID="ad_slot0",CLASS="com.apple.has.interstitial",START-DATE="2024-10-30T12:52:47.207+01:00",DURATION=10,X-ASSET-LIST="http://localhost:3333/interstitials.m3u8?_HLS_interstitial_id=ad_slot0",X-RESTRICT="SKIP,JUMP",X-RESUME-OFFSET=10,X-SNAP="IN,OUT"
#EXT-X-PROGRAM-DATE-TIME:2024-10-30T12:52:47.853+0100
#EXTINF:4,
fileSequence16.ts
#EXT-X-PROGRAM-DATE-TIME:2024-10-30T12:52:51.853+0100
#EXTINF:4,
fileSequence17.ts
#EXT-X-PROGRAM-DATE-TIME:2024-10-30T12:52:55.853+0100
#EXTINF:4,
fileSequence18.ts

In this playlist, an EXT-X-DATERANGE tag schedules a 10-second ad break to play at the specified time point. The player loads the resource specified by the URL after buffering the primary asset to the scheduled interstitial playback time, and then resumes playback of the primary asset at the live edge. Seeking and scanning forward will be disabled during interstitial playback. It is also possible to include custom attributes to be processed by the client.

{
   "ASSETS":[
      {
         "URI": "http://localhost:3333/interstitials.m3u8?_HLS_interstitial_id=ad_slot0&_HLS_primary_id=40FE1829-438E-49B0-8B3A-A285DD4A8154&_HLS_start_offset=0&_HLS_follow_id=361434bf-05e7-4e17-83ca-690452e1cb33",
         "DURATION":5
      },
      {
         "URI": "http://localhost:3333/interstitials.m3u8?_HLS_interstitial_id=ad_slot0&_HLS_primary_id=40FE1829-438E-49B0-8B3A-A285DD4A8154&_HLS_start_offset=5&_HLS_follow_id=eb805a34-1d61-4217-9632-deab8790c30d",
         "DURATION":5
      }
   ]
}

In this example, the returned JSON object from the X-ASSET-LIST URL includes two 5-second ads. Their URI includes query parameters to identify ad slots, individual ads, and user sessions, making late binding to ad inventory possible and ad personalization easier.

Compared to SSAI and CSAI

SGAI vs. SSAI: Both SGAI and SSAI stitch ads server-side, but SGAI offers finer control over targeting and adapts dynamically to real-time user data. While SSAI lacks some targeting precision, SGAI’s server-level adjustments allow for improved ad personalization.
SGAI vs. CSAI: CSAI places ads on the client side, allowing for tracking data but also making ads prone to blockers and playback glitches. SGAI, on the other hand, provides a seamless experience and improved protection from ad blockers, enhancing both viewer engagement and revenue.

Our SGAI Proxy Tool

As a Proof of Concept, Eyevinn's SGAI Ad Proxy serves as a practical tool that facilitates the integration of ads into HLS content. This lightweight HTTP proxy server intercepts the video stream and inserts ads at defined time points.

Steps to Use the Ad Proxy Tool

Set Up the Environment:
- Create an HLS Stream: Use ffmpeg to set up a basic HLS streaming server.
- Ad Server: Ensure access to a compatible ad server serving VAST-compliant ads.
Choose Ad Insertion Mode:
- Select either static mode (fixed intervals) or dynamic mode (on-demand insertion).
Run the Proxy Server.
To Insert Ads Dynamically:
- Send a GET request to the proxy.
Access the new Stream:
- Use a compatible video player (e.g., AVPlayer) with the proxy URL.

As the streaming landscape evolves, tools like the SGAI Ad Proxy are pivotal in enhancing the viewer experience and driving ad revenue. For more information and setup details, visit the tool’s GitHub repository.

Client Side Ad Insertion Crash Course

Martin Stark — Wed, 29 May 2024 12:43:28 +0000

This guide acts as a quick overview of concepts relevant to CSAI, it is not meant to act as a complete tutorial.

What is CSAI?

Client Side Ad Insertion means that the app on the user's device is responsible for fetching and displays advertisement as part of the streaming experience. This is in contrast to Server Side Ad Insertion, where the ads are stitched into the video on the server.

VMAP

The VMAP is fetched or polled by the client, it is parsed to check when an ad break should occur.

VAST

The VAST document is fetched and parsed by the client, it represents the content of an ad break.

The VAST document can contain several types of ads. For example: video, images, and companion links.

The VAST specifies how an ad should be tracked. It says which events need to be sent to a tracking server in order to verify if and how much of the ad was watched and/or interacted with.

Displaying the Ads

When it is time to show the ad to the user, it is usually handled by pausing the content stream, creating a new video element, overlaying it on the content stream, and playing the ad(s). When the ad break is finished, remove the ad specific video element and resume the content stream.

How to quickly setup a continue watching endpoint for your video players

Jonas Birmé — Wed, 28 Feb 2024 21:26:05 +0000

In this blog post we will walk through how you can quickly setup a continue watching micro service for your video player applications using Open Source Cloud and Redis Cloud.

Today a user of a video streaming service expects to be able to pick up where they left off, on any of the episodes they watched and on any of the devices they have. To handle that you would need to develop a backend service with endpoints for the video player and device application to write to and read from.

Open Source Cloud by Eyevinn Technology is a service that enables you to build software solutions on on open and detachable ready-to-run cloud components based on open source, and where revenue from this service is contributed back to the open source community.

Redis Cloud is the easiest way to build and manage fast, scalable apps–fast. It reduces total cost of ownership and helps organizations fight database sprawl. It also empowers architects and developers like no other cloud-based service.

In this post we will create a continue watching service using one of the available components in Open Source Cloud and Redis Cloud for storage. As both components are based on open source there is nothing preventing you from bringing these components "home" to your own infrastructure.

Create a database on Redis Cloud

Once created a database note down the Public endpoint and port as seen above. You will also need the password that you find further down on the same page.

Create service on Open Source Cloud

Create an account on Open Source Cloud unless you already have one. Once logged in to Open Source Cloud you will look for the component called Continue Watching Service.

When you have started a subscription you can create a service by clicking on the "Create service button".

When you have clicked this button you will be presented with a dialog where you will enter the connection details to the Redis database you created. Replace Host, Port and Password with the configuration details from your Redis database. Then click on Create.

Wait a few seconds and you will see the service up and running.

Now your continue watching service is up and running and you can copy the endpoint URL available above and try the following in a new tab in your browser.

https://demo-jonas.eyevinn-continue-watching-api.auto.prod.osaas.io/position/jonas

This will give the saved positions for the user jonas and now it should be just an empty array.
To save a position your video player or application would issue an HTTP POST request to /position/jonas/12345/120 where 12345 is the id of the episode and 120 is the position in seconds. For example:

curl -v -X POST \
  -H 'x-jwt: Bearer <SAT>' \
  https://demo-jonas.eyevinn-continue-watching-api.auto.prod.osaas.io/position/jonas/12345/120

And now when you retrieve the positions for user jonas you would get.

curl -v -X GET \
  -H 'x-jwt: Bearer <SAT>' \
  https://demo-jonas.eyevinn-continue-watching-api.auto.prod.osaas.io/position/jonas

[{"assetId":"12345","position":"120","expiration":31535812}]

How to get the SAT (service access token) you can read about in the API documentation and the serviceId in this case is eyevinn-continue-watching.

curl -X 'POST' \
  'https://token.svc.prod.osaas.io/servicetoken' \
  -H 'accept: application/json' \
  -H 'x-pat-jwt: Bearer <PAT>' \
  -H 'Content-Type: application/json' \
  -d '{
  "serviceId": "eyevinn-continue-watching"
}'

The PAT (personal access token) you find under Settings in the menu to the left.

And that's all you need to provide a continue watching micro service in your solution.

Generate subtitles with OpenAI Whisper and Eyevinn OSC

Oscar Nord — Thu, 08 Feb 2024 15:21:01 +0000

With AI and LLMs being on (almost) everyone's mind these days, we at Eyevinn thought that it would be interesting to explore the possibilities of using OpenAI's speech-to-text model to generate subtitles for video/audio content.

The initial idea was that it would be cool if we could generate subtitles on the fly when a user requests them for a movie, they won't be perfect of course but maybe they can be just good enough to provide basic translations for content that otherwise wouldn't have subtitles in that specific language available.

With that idea in mind, we have created a small POC that is available on GitHub. It's a super simple API that makes it possible to provide a link to a video/audio segment where the response is the transcribed content. We've also added the possibility of uploading the transcribed file to an S3 bucket.

To transcribe content you will do a POST to the /transcribe endpoint:

{
  "url": "https://example.net/vod-audio_en=128000.aac"
  "language": "en" // ISO 639-1 language code (https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) (optional)
  "format": "vtt" // Supported formats: json, text, srt, verbose_json, or vtt (optional)
}

The response will then be the transcribed content:

{
  "workerId": "BFabbcCi3IYuWOj6LfsgK",
  "result": "WEBVTT\n\n00:00:00.000 --> 00:00:04.180\nor into transcoding I mean, I could probably add just the keyframe in the start and just\n\n00:00:04.180 --> 00:00:06.920\nskip I-frames and the rest of that.\n\n"
}

Formatted output:

WEBVTT

00:00:00.000 --> 00:00:01.940
So into transcoding, I mean, I could

00:00:01.940 --> 00:00:03.700
probably add just a keyframe at the start

00:00:03.700 --> 00:00:06.700
and then just skip iFrames in the rest of the scenes.

The API uses the OpenAI API so an OpenAI account and API key are required.

The easiest and best way to try this out is to use our newly released (currently in beta) platform that we call Eyevinn Open Source
as a service. It's a cloud platform that makes it super easy to prototype, monetize, and take open-source services to production. The idea behind this is that when you sign up, which is completely free, you can easily try out and deploy a selection of open-source services with a click of a button. What makes this platform special is that you only pay for the cost of running the services and a small community fee that will be shared with the maintainers of the open-source service that you use.

Take the auto-subtitles API as an example, when you have created an account you will be presented with a dashboard containing multiple services.

When creating an instance of auto-subtitles you'll need to input your OpenAI key (this is only used when making calls with the OpenAI API)

That's it now you are up and running and can try it out!

Eyevinn OSaaS is currently in beta so let us know what you think about it.

As always if you need assistance in the development and implementation of this, our team of video developers are happy to help you out. If you have any questions or comments just drop a line in the comments section to this post.

Stream from OBS to Twitch and Preview With SRT-WHEP

Kun Wu — Mon, 28 Aug 2023 10:30:37 +0000

This guide will take you through the step-by-step process of setting up OBS (Open Broadcaster Software) to stream to Twitch and preview the output in browser using our SRT-WHEP tool and Web Player.

Prerequisites

Installed OBS: Download and install OBS from the official website: OBS Project.

Steps

1.Twitch Account Setup:

If you don't have a Twitch account, create one by visiting: Twitch Signup.
Log in to your Twitch account.

2.Install and Run SRT-WHEP:

You need to run the SRT-WHEP application on your computer. You can either build it from source or use the Docker image (for Ubuntu). Detailed building instructions can be found in the GitHub repository.
To run it in listener mode and wait on port 1234, execute the following commands:
- If building from source: srt-whep -i 127.0.0.1:1234 -o 127.0.0.1:8888 -p 8000 -s listener | bunyan
- If using Docker image: docker run --rm --network host eyevinntechnology/srt-whep -i 127.0.0.1:1234 -o 127.0.0.1:8888 -p 8000 -s listener
This will forward the SRT stream to port 8888.

3.Create Input Stream with OBS:

Launch OBS on your computer.
In OBS, go to the "Sources" tab and add a new source (e.g., macOS Screen Capture).
To connect to SRT-WHEP, navigate to the "Settings" menu in OBS and select the "Stream" tab.
Choose "Custom" as your streaming service.
In the "Server" field, enter the input SRT URL: srt://127.0.0.1:1234?mode=caller
Start streaming by clicking the "Start Streaming" button. OBS will begin sending the stream to SRT-WHEP as the caller.
You'll see in the SRT-WHEP console that the stream is received and ready for playback.

4.Stream to Twitch using SRT-RTMP Tool:

Twitch accepts RTMP streams directly but not SRT stream.
We have a tool that accepts SRT stream and outputs as RTMP stream.
Once it's installed, set your Stream Key as environmental variable STREAM_KEY and then run the srt-rtmp application:

export STREAM_KEY=$YOUR_STREAM_KEY

srt-rtmp -i 127.0.0.1:1234 -s listener -o live.twitch.tv/app | bunyan

Here the Stream KEY is the Stream Key from your Twitch account. Find it under Creator Dashboard -> Settings -> Stream -> Primary Stream Key.

5.Preview the Stream in Browser:

To preview the stream, open the WHEP Player in your web browser.
Enter the URL: http://localhost:8000/channel.
Click "Play," and within a few seconds, you should see the OBS stream in your browser.

6.Viewing the Stream on Twitch:

Navigate to your Twitch account dashboard to view your live stream.

7.Ending the Stream:

To stop previewing the stream, click the "Stop" button in the WHEP Web Player.
When you're ready to end your stream, return to OBS and click the "Stop Streaming" button.

Conclusion

Congratulations! You've successfully configured OBS to stream to Twitch and previewed the output using your web browser. Enjoy your seamless streaming experience!

Seamlessly Streaming SRT Content in Web Browsers

Kun Wu — Fri, 25 Aug 2023 11:25:01 +0000

In the ever-evolving realm of content delivery, the convergence of SRT (Secure Reliable Transport) streaming and WebRTC (Web Real-Time Communication) marks a significant milestone. This fusion addresses the complex challenge of delivering content seamlessly across diverse devices and platforms, opening up new avenues for real-time interactions. Our exploration into bridging the gap between SRT streaming and browser playback is a testament to our commitment to solving these challenges.

Background: The Nexus of SRT and WebRTC

RT streaming redefines how video, audio, and data are transmitted across unpredictable networks. Operating on a single UDP socket, SRT seamlessly integrates reliability, ordering, and congestion control, crucial for real-time applications like video conferencing. Its automatic repeat request (ARQ) mechanism ensures content delivery integrity, particularly essential for maintaining seamless real-time interactions.

In contrast, the conventional HLS (HTTP Live Streaming) protocol relies on TCP, which can introduce latency complexities and hinder real-time interactions. However, HLS excels in device compatibility, making it accessible on a wide range of devices without the need for specialized tools. Recognizing the strengths of SRT in terms of reliability and efficiency, coupled with HLS's device compatibility edge, our exploration led us to the world of WebRTC.

Our Endeavor: Forging with Tools and Architecture

Our mission to seamlessly integrate SRT streaming into web browsers relied on a foundation of carefully selected tools and architecture. GStreamer, a versatile open-source multimedia framework, served as the core of our project, enabling efficient processing and streaming of audio and video content. The modularity of GStreamer allowed us to seamlessly integrate various components, optimizing SRT streaming within web browsers.

In terms of coding, we chose the Rust programming language. Renowned for its exceptional performance, memory safety, and cloud deployment suitability, Rust's unique attributes were pivotal in ensuring a robust and efficient solution. Rust's focus on preventing memory-related errors and its concurrency model were instrumental for real-time streaming and interactivity. Furthermore, its inherent cross-platform compatibility aligned seamlessly with our mission. The strategic combination of Rust and GStreamer produced a codebase capable of navigating various operating systems.

Architecture of Seamless Playback

Our application's architecture followed a streamlined workflow familiar in video and audio processing. An input stream, such as an SRT stream, underwent demuxing to reveal metadata and streams. Compressed stream data was decoded, and optional filters were applied to raw frames, such as resizing for optimization. Notably, video streams from MPEG-TS were transmitted directly to WebRTC peers without decoding, leveraging WebRTC's inherent support for H.264 (AVC) video codec. However, audio transcoding from AAC to OPUS was necessary to achieve compatibility with browser peers. This approach caters to use cases like confidence monitoring or content preview, where maintaining unaltered visual fidelity is essential.

Getting Started

For those interested in exploring our solution, two options are available: building from source code or using our Docker images (compatible with Ubuntu systems only). Here, we demonstrate the Docker option.

Generate an SRT test source, for example, using our testsrc Docker container:

docker run --rm -p 1234:1234/udp eyevinntechnology/testsrc

An SRT stream (in listener mode) is then available at srt://127.0.0.1:1234. Then run the srt-whep application:

docker run --rm --network host eyevinntechnology/srt-whep -i 127.0.0.1:1234 -o 127.0.0.1:8888 -p 8000 -s caller

It will connect to the SRT test stream in caller mode as the generated SRT stream is in listener mode.

WHEP endpoint is available at http://localhost:8000/channel. You can then play it for example using our WebRTC Player.

Challenges: Overcoming Hurdles

Our journey to integrate SRT streaming with browser playback encountered several challenges.

Complexity in Tool Compatibility: The intricacies of tool compatibility emerged as we engaged with tools like FFmpeg/FFplay and GStreamer. Despite the apparent simplicity of creating and playing an SRT stream, intricacies arose due to diverse toolsets across platforms. An SRT stream generated using FFmpeg, for instance, might not seamlessly harmonize with GStreamer without precise configuration. Even seemingly minor details, like a missing bitrate parameter, could disrupt streaming unexpectedly.

Browser Support for WebRTC Codecs: Navigating the nuances of browser support for WebRTC codecs presented another challenge. Widely-used video codecs like H.264 and H.265 thrived within tools like ffmpeg. However, aligning these codecs harmoniously with browser prerequisites was a complex endeavor. While the WebRTC standard embraced both H.264 and H.265 video streams, browser compatibility revolved around video profiles. Chrome exhibited versatility, supporting all profiles – baseline, main, and high – catering to diverse developer needs. On the contrary, Safari displayed exclusivity, embracing only the Baseline profile. This diversity necessitated strategic decisions, carefully balancing codec capabilities, WebRTC's potential, and browser preferences for a seamless union of streaming and browsing experiences.

Cross-Platform Compatibility. The pursuit of cross-platform compatibility emerged as a formidable challenge, encompassing devices, operating systems, and browser ecosystems. Seamlessly delivering content across this spectrum required a comprehensive understanding of browser intricacies and interactions with underlying operating systems. Variances in audio and video handling, network conditions, and user behaviors contributed to a complex tapestry.

Realizing cross-platform compatibility demanded a harmonious fusion of technologies – from SRT streaming and WebRTC to GStreamer and Rust. Each component needed to align cohesively to ensure a uniform experience for users, irrespective of their chosen platform.

Summary: The Odyssey of Integration

Our journey to seamlessly integrate SRT streaming with web browsers was rooted in a commitment to effective solutions and adaptability in the face of evolving technology. The convergence of WebRTC for seamless playback, browser-approved codecs, and cross-platform tools encapsulated our approach. This harmonization ensured that processes, testing, and accessibility converged harmoniously, underscoring our dedication to an interactive and universally accessible streaming solution.

Adding support for subtitles in The Eyevinn Channel Engine

Johan Lautakoski — Mon, 15 May 2023 11:03:54 +0000

In this blog post, I'll describe how I added support for subtitles in the Channel Engine so it could play subtitle tracks. I will also assume that the reader is somewhat familiar with the HLS streaming format and Channel Engine or has at least read the documentation in the Channel Engine git repo, or this article link beforehand.

Introduction

The Eyevinn Channel Engine is an Open-Source service that works well with VODs,

But before we get into it, I need to clarify what I mean when I say "subtitle tracks" and "subtitle groups", as I will be using these words throughout this post.

In an HLS multivariant playlist, you can have a media item with the attribute TYPE=SUBTITLES with a reference to a media playlist containing the subtitle segments. This is what I will be referring to as a "subtitle track". Multiple subtitle tracks can exist in the HLS multivariant playlist. These tracks can be grouped/categorized, by the media item's GROUP-ID attribute. Subtitle tracks that have the same GROUP-ID value are what I will refer to as a "subtitle group". In other words, a subtitle group consists of one or more subtitle tracks. GROUP-IDs are an HLS requirement for media items.

Challenges

The task in question will have some implementation challenges.
A few things needed to be taken into account. Namely:

How to handle subtitle tracks between different VODs.
How to handle VODs without subtitles.
How to handle VODs where subtitle segments are longer than video segments.
How to handle subtitle groups.
How to handle subtitle tracks with the same language.

Delimitations

The implemented solution in its current state did not cover every edge case. Meaning that some points mentioned in Challenges have yet to be addressed. However, the implementation works fairly well for the most basic case and can be extended in the future to handle more edge cases.

My solution will when faced with multiple subtitle tracks with the same name, it will pick the first one. Also when a subtitle track that was not defined before the start shows up it will be ignored.

Proposed solutions will be mentioned in the Future Work chapter.

Implementation

The following steps give an overview of how I added support for subtitles in the Channel Engine.

Step 1: Adding subtitle media items to the multivariant playlist based on a set of predefined subtitle languages.

To address the challenge of how the client is to select a certain subtitle group and subtitle track, I implemented a method that works like audio. To read more about the audio implantation, read this article

The plan was to let the client select a track based on what's been predefined. So to have it work as it does for VOD profiles, I needed to extend the ChannelManager class with an extra function.

We are doing this to solve the issue where VODs has different languages.

Media Items are added to the multivariant playlist with attribute values set according to a predefined JSON object, defined in a _getSubtitleTracks() function in the ChannelManager class/object.

Now the resulting multivariant playlist looks something like this...

Note: Notice that GROUP-ID is not a field in the subtitleTrack JSON, and so the GROUP-ID in the multivariant playlist media items is actually permanently set by the his-vodtolive library. this is done to handle VODs without subtitles. See Delimitations.

Step 2 Make it possible to load subtitles in hls-vodtolive

Now that we know what segments the client is looking for, how do we find them? This is where the Eyevinn dependency package hls-vodtolive comes into play.

In short, the hls-vodtolive package creates an HLSVod class/object which given a VOD multivariant playlist as input, will load and store all segments referenced in that playlist into a JSON object organized by profiles. An HLSVod object will also divide the segments into an array of subsets, that we call media sequences. So each subset/media sequence will be used to create a pseudo-live-looking media playlist.

There were three important things that needed to be done here.

Load subtitle segments from source VOD and create a pseudo-live-looking subtitle playlist.
What to do when a VOD does not have subtitles?
The reason we need to do something about a VOD without subtitles is that the client is expecting subtitles and if we do not provide them the client will start buffering until subtitles are provided.
We solved it by looking at the source VOD and then creating dummy subtitle segments that have the same duration as the video segments and for the duration of the entire VOD. The dummy segments contain a URL to an endpoint that needs to be defined in the options when creating an instance of hls-vodtolive. the handler of the endpoint should return empty WebVTT segments.
How to handle subtitle segments that have a longer duration than video segments.
Why are we doing this, the answer is that we are creating a live stream, and when subtitles are used in a live stream they need to have the same duration as the other media.
We solved this by creating smaller segments out of a bigger segment or vice versa. the newer segments have a URL to an endpoint where a WebVTT file can be sliced into smaller files.
The URL has a few query parameters that help the endpoint to know which files/files to slice from and at what time in the VOD

Step 3 add dummy endpoint and slice endpoint to the Channel Engine

The Channel Engine was extended with two endpoints the dummy endpoint and the slice endpoint. As previously mentioned the dummy endpoint only returns an empty WebVtt file which only contains the following lines

WEBVTT
X-TIMESTAMP-MAP=MPEGTS:0,LOCAL:00:00:00.000

The slice endpoint on the other hand is a bit more complex, it needs to be able to fetch and parse a WebVTT file and also be able to extract and return the relevant subtitles lines.

Future Work

Add support for subtitle characteristics Adding support for characteristics would enable you to have the same language multiple times, eg simplistic Swedish and Swedish.

Interpreting HTML5 Video Events

Martin Stark — Mon, 13 Mar 2023 12:05:55 +0000

When building a HTML5 web video player, it tends to need a user interface, quality of service tracking, user interaction tracking, and sometimes ad tracking. To achieve this, the HTML5 Media Event standard needs to be interpreted.

After building multiple web video players, and having worked on more than one iteration of an event interpreter, something didn't feel right. Things weren't kept DRY.

When repetition strikes, it's usually a good idea to do something about it.

Creating a Media Event Filter

Enter @eyevinn/media-event-filter, a filter that interprets HTML5 media events in a way that makes sense.

This filter aims to provide a single source of truth that can be used across player engines and native browser playback.

Whether using one or multiple player engines, like Shaka Player, HLS.js, or dash.js, the filter will output a standardised event sequence. It does so without relying on events output by the player engine, giving applications a single, player agnostic, source of truth for video playback state updates.

Example Implementation

import {
  getMediaEventFilter,
  FilteredMediaEvent,
} from "@eyevinn/media-event-filter";

const videoElement = document.createElement("video");

const mediaEventFilter = getMediaEventFilter({
  videoElement,
  callback: (event: FilteredMediaEvent) => {
    switch (event) {
      case FilteredMediaEvent.LOADED:
        // handle loaded
        break;
      case FilteredMediaEvent.BUFFERING:
        // handle buffering
        break;
      case FilteredMediaEvent.BUFFERED:
      // handle buffered
      // ...
      default:
        break;
    }
  },
});

// Create a player, using your engine of choice
const player = createPlayer(videoElement);

// Load a manifest and play
player.load(url).then(player.play);

// Call when done
mediaEventFilter.teardown();

Live Example

See a barebones React + Shaka + Media Event Filter example over at codepen.

Production Readiness

While making the filter open source is recent, it has been battle tested for years. With millions of playback sessions, multiple third party tracking systems have been validated using the filter.

Event Sequence

The sequence follows the Eyevinn Player Analytics Specification, which maps directly to many popular tracking service provider SDKs.

Sample event sequence comparison, using <video autoplay> with Shaka Player in Firefox, excluding timeupdate:

Native	Filtered	Comment
ratechange	—	Shaka sets playback rate to 0 to control buffering
progress	—
durationchange	—
resize	—
loadedmetadata	—
ratechange	—
progress	—
loadeddata	—
canplay	—
play	—
playing	—
canplaythrough	loaded	video is ready to start playing
—	playing	video is playing, play event missing due to autoplay
progress	—
...	—
progress	—
pause	pause	manual pause
progress	—
seeking	seeking	seeking while paused
ratechange	—	Shaka sets playback rate to 0 to control buffering
play	play	Play requested during a seek
waiting	—
progress	—
progress	—
progress	—
seeked	—
canplay	—
playing	—
canplaythrough	—	video element thinks it can start playing, but playback rate is 0
progress	—
ratechange	seeked	shaka returns playback rate to normal
—	playing	video is playing again after previous pause
progress	—
progress	—
pause	pause
progress	—
play	play
playing	playing
progress	—
progress	—
seeking	seeking	seeking while playing
waiting	—
ratechange	—
progress	—
progress	—
seeked	—
canplay	—
playing	—
progress	—
canplaythrough	—
progress	—
ratechange	seeked	seek finished, video is rolling
progress	—
progress	—
progress	—
progress	—
ratechange	buffering	buffer empty, shaka sets playbackrate to 0
progress	—
progress	—
progress	—
canplaythrough	—	video element thinks it can start playing, but playback rate is 0
progress	—
ratechange	buffered	shaka returns playback rate to normal
progress	—
...	—
progress	—
seeking	seeking
waiting	—
ratechange	—
progress	—
...	—
progress	—
seeked	—
canplay	—
playing	—
canplaythrough	—
progress	—
ratechange	seeked
progress	—
progress	—
progress	—
durationchange	—
progress	—
seeking	seeking	seeking within buffer, no ratechange
waiting	—
seeked	seeked	seek finished
canplay	—
playing	—
canplaythrough	—
pause	pause
ended	ended	end of stream reached
emptied	—

Download it at npm.

Contribute at github.

Exploring video generators in FFMPEG...

Alan Allard — Fri, 24 Feb 2023 11:05:04 +0000

...in which the author creates entirely inappropriate amounts of eye candy by testing the ranges of most of the video generation options within ffmpeg - but also manages, as luck would have it, to identify some valuable testing tools in the process...

If you haven't started following Eyevinn's ffmpeg of the day series on Instagram (of which I am the author), perhaps you should! Anyway, the very first command that I looked at in the series was an example of the cellauto filter in ffmpeg. While trawling through the long, long list of ffmpeg filters available for audio and video, I noticed that video generators have their own quite sizeable section in the list. There are so many of these generators available, but many of them may turn out to be useful for various purposes. Not only are there many separate generators in the list, several of them are extensible with code or plugins in a specific format. In this article, I will try and get a notion of what all of these generators can do, by examining their various parameter ranges.

A caveat, first. I am on macOS, so I will be focusing on the filters that are readily available for me - not least of all the coreimagesrc generator with it's considerable list of built-in effects.

In fact, coreimagesrc seems like a pretty good place to start...

The `coreimagesrc` generator

Several of these generators we will look at are also available as more general filters, with some slight variation in options. Generators are more specific in that they are designed to be first in the filter chain, as inputs. This coreimagesrc has a more general counterpart in the filter coreimage, for example.

The first thing we want to do is identify which particular generators are available on our Mac:

ffmpeg -f lavfi -i coreimagesrc=list_generators=true null

which will give you a long list of generators with all their parameters. Let's grep it (notice the little pirouette of piping everything from stderr to stdout, which is necessary in order to pipe the ffmpeg output to grep successfully). We may as well cut the first part of the line too for a nice tidy list:

ffmpeg -f lavfi -i coreimagesrc=list_generators=true 2>&1 | grep Filter: | cut -c 32-

which gives us the following list (in my case):

 Filter: CIAttributedTextImageGenerator
 Filter: CIAztecCodeGenerator
 Filter: CIBarcodeGenerator
 Filter: CICheckerboardGenerator
 Filter: CICode128BarcodeGenerator
 Filter: CIConstantColorGenerator
 Filter: CILenticularHaloGenerator
 Filter: CIMeshGenerator
 Filter: CIPDF417BarcodeGenerator
 Filter: CIQRCodeGenerator
 Filter: CIRandomGenerator
 Filter: CIRoundedRectangleGenerator
 Filter: CIStarShineGenerator
 Filter: CIStripesGenerator
 Filter: CISunbeamsGenerator
 Filter: CITextImageGenerator

Quite a mixed bag. How about a lenticular halo generator?
With many filters and/or generators in ffmpeg, you can just give the filter name and the default parameters will be used. That's only partially true with the coreimage generators; there are default parameters available, but all of these generators have one or more parameters that have no default, so you will get an error if you try something like:

ffmpeg -f lavfi -i \
"coreimagesrc=s=600x600:\
filter=CILenticularHaloGenerator" \
-t 30 -pix_fmt yuv420p output.mov

which will give:

[coreimagesrc @ 0x7fa3fc205a00] Parsing of filters failed.
[lavfi @ 0x7fa3fb7045c0] Error initializing filter 'coreimagesrc' with args 's=600x600:filter=CIMeshGenerator'
coreimagesrc=s=600x600:filter=CIMeshGenerator: Input/output error

Let's examine the parameters of the lenticular halo generator in the list from earlier (we filtered them out last time):

ffmpeg -f lavfi -i coreimagesrc=list_generators=true 2>&1 | grep -A 8 'Filter: CILenticularHaloGenerator' | cut -c 32-

giving

Option: inputCenter     [CIVector]
Option: inputColor      [CIColor]
Option: inputHaloRadius [NSNumber]      [0 1000][70]
Option: inputHaloWidth  [NSNumber]      [0 300][87]
Option: inputHaloOverlap        [NSNumber]      [0 1][0.77]
Option: inputStriationStrength  [NSNumber]      [0 3][0.5]
Option: inputStriationContrast  [NSNumber]      [0 5][1]
Option: inputTime       [NSNumber]      [0 1][0]

which is very useful in that we can see which parameters have defaults (the second pair of square brackets are the defaults). So at the very least we need a CIVector and a CIColor. Next problem, how do we represent a CIVector and CIColor as parameters to the generator within an ffmpeg command? It took me a while to work this out, but with the help of the very sparse ffmpeg documentation, the ffmpeg source code and a quick look at the CoreImage apple documentation, I came up with the answer, as follows:

ffmpeg -f lavfi -i \
"coreimagesrc=s=600x600:\
filter=CILenticularHaloGenerator\
@inputCenter=200.0 200.0\
@inputColor=0.0 0.0 1.0" \
-t 5 -pix_fmt yuv420p output.mov

Which gives us five seconds of a rather nice sort of focusing iris effect:

Ok, how about a star shine generator:

ffmpeg -f lavfi -i \
"coreimagesrc=s=600x600:\
filter=CIStarShineGenerator\
@inputCenter=300.0 300.0\
@inputRadius=25\
@inputCrossAngle=0\
@inputColor=1.0 0.0 1.0" \
-t 5 -pix_fmt yuv420p starshine.mov

which yields:

The `gradients` generator

There are several more of these coreimagesrc generators to explore, maybe we'll come back to them in a bit. As there are so many more types of ffmpeg generator, let's move on for the moment. Let's examine the gradients generator next. This is a powerful filter that can generate several different types of gradient, with random or designated colours and even rotate them if you wish (merely a convenience of course, as it's pretty trivial to add a rotation filter to any filter graph). An example:

ffmpeg -f lavfi -i \
gradients=duration=5\
:nb_colors=5:x0=320:y0=240\
:type=spiral:speed=0.1 \
-pix_fmt yuv420p gradients.mov

Here, in fact, it becomes apparent that rotation is very much meant to be part of the gradient effect - the lowest speed is not 0 but 1e-05. So in fact it always rotates even at the lowest setting, just very slowly. You could always use the rotate filter in the opposite direction to keep it stationary of course...for that you would need to know what exactly that speed parameter represents. Here's a clue: in the ffmpeg source code, video sources are sensibly prefixed with vsrc_, so here we can take a look in vsrc_gradients.c and see that the speed parameter is used to calculate an angle as follows: float angle = fmodf(s->pts * s->speed, 2.f * M_PI); So basically the angle is in radians and is a function of pts. We'll leave that topic there this time round, just as a little note on how to compensate for the occasional shortfalls in the documentation by taking a quick look at the source code. Anyway, this is what we got from the last command:

Certainly pleasing to the eye, but the back and forth partial rotation is a bit odd. I haven't yet found a combination of gradients parameters to make a spiral gradient rotate around its centre (and I welcome suggestions), but I did notice that setting parameters x1 and y1 to the same as x0 and y0is a handy way to stop all rotation. After that we can simply apply the rotate filter instead, which looks...pretty good, I suppose:

When I'm going through all the options of a filter or generator to get a feel for them, I often want to see some kind of overview of all the options together. In those cases, I typically reach for javascript and build an ffmpeg command in that way, one that results in either a grid showing all the different options or one option after another in a concatenated video. For example, here is a grid of all four gradient types in the gradients generator. In this case, some of the default options are used, so the colours are randomly chosen and will vary each time the video is generated:

Take a look at the output ffmpeg command to understand why you might like to offload things like this to a custom command! :

ffmpeg -y -filter_complex \
"gradients=duration=10:type=spiral[out0];\
gradients=duration=10:type=radial[out1];\
gradients=duration=10:type=linear[out2];\
gradients=duration=10:type=circular[out3];\
[out0]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=spiral:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out0];\
[out1]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=radial:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out1];\
[out2]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=linear:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out2];\
[out3]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=circular:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out3];\
[text_out0][text_out1][text_out2][text_out3]xstack=inputs=4:grid=2x2" \
-t 10 allGradients.mp4

There is clearly a whole load of repetition, so this should be fairly easy to build and parameterise. Essentially this will all just be string building so we won't need to use any particular libraries for most of this script. We will need a way to call ffmpeg though - and ffmpeg will need to be present too, of course. To call a CLI command we can use the package commander.

We can see from a glance at the above command that we need:

The start line: ffmpeg -y -filter_complex
A line for each gradient type with a uniquely-named output at the end: gradients=duration=10:type=spiral[out0];\
A line for each text label with the matching inputs from 2. and uniquely-named outputs:

[out0]drawtext=fontfile='fonts/crystal-radio-kit/crystal radio kit.ttf'\
:text=spiral:fontsize=35:x=(w-text_w)/2:y=(h-text_h)/10:fontcolor=black[text_out0];\

An xstack of the appropriate size that collects all of the uniquely-named drawtext-outputs as inputs to it.

Apart from that, there will be a few syntax issues to deal with as regards adding line ends, quotation marks and then building the final command. When it's built we can call exec() from commander to invoke ffmpeg. I tend to build in line breaks to the final commands so that they are easy to debug in the console.

Without further ado here is the code for that:

// gradientsDemo.js

const { exec } = require("child_process");

// Set up variables used to construct lines
const allGradientsTypes = [
    'spiral',
    'radial',
    'linear',
    'circular',
];

const TEXT_FONT = '\'fonts/crystal-radio-kit/crystal radio kit.ttf\'';
const TEXT_POS = 'x=(w-text_w)/2:y=(h-text_h)/10';
const TEXT_COLOR = 'black';
const FONT_SIZE = 35;
const TIME = 10;
// Setup lines
let filterLinesStart= "ffmpeg -y -filter_complex \\";
let filterLinesGenerator = ""; // generator lines
let filterLinesText = ""; // text lines
let filterLinesXStackInputs = "";
let filterLinesStacker = "";

let filterLinesLast = ` -t ${TIME} -an -pix_fmt yuv420p gradients.mp4`;

// Loop through lines
allGradientsTypes.forEach((gradient, index, gradients) => {
    filterLinesGenerator += `gradients=duration=${TIME}:type=${gradient}[out${index}];`;
    filterLinesText += `[out${index}]drawtext=fontfile=${TEXT_FONT}\
:text=${gradient}:fontsize=${FONT_SIZE}:${TEXT_POS}:fontcolor=${TEXT_COLOR}[text_out${index}];`;
    filterLinesXStackInputs += `[text_out${index}]`;

    if (Object.is(gradients.length - 1, index)) {
        // Finish up lines
        filterLinesGenerator += `\\`;
        filterLinesText += '\\';
        filterLinesStacker = `xstack=inputs=${allGradientsTypes.length}:grid=${2}x${2}`;
        filterLinesXStackInputs += ``;
    } else {
        filterLinesGenerator += `\\\n`;
        filterLinesText += `\\\n`;
    }
});

// Build command...
const command = `${filterLinesStart}
"${filterLinesGenerator}
${filterLinesText}
${filterLinesXStackInputs}${filterLinesStacker}"\\
${filterLinesLast}`;

console.info("Command is: ", command);

exec(command, (error, stdout, stderr) => {
    if (error) {
        console.log(`error: ${error.message}`);
        return;
    }

    console.log(`stdout: ${stdout}`);

    if (stderr) {
        console.log(`stderr: ${stderr}`);
    }
    console.info(`Generated a file with ${allGradientsTypes.length} gradient demos in a ${2} by ${2} grid`);
    return;
});

And as we get completely different colours each time, let's test that code and produce another instance of the video:

Already at this stage we are able to alter some of the text properties (as you have probably noticed you will want to have a path to the font in question on your machine). It's now fairly easy to add other parameters. For example, if the intention is to use these generated videos for analytic purposes, you may want to understand how the colours look for each of the gradients, so why not make the colours settable. By default, gradients sets only 2 random colours, but if we pass in the colour list as an array we can set the colour count parameter nb_colors too.

// gradientsDemo2.js

const { exec } = require("child_process");

// Set up variables used to construct lines
const allGradientsTypes = [
    'spiral',
    'radial',
    'linear',
    'circular',
];

// NEW - define the colours
const colours = [
    'blue',
    'red',
    'yellow'
];

const TEXT_FONT = '\'fonts/crystal-radio-kit/crystal radio kit.ttf\'';
const TEXT_POS = 'x=(w-text_w)/2:y=(h-text_h)/10';
const TEXT_COLOR = 'black';
const FONT_SIZE = 35;
const TIME = 10;
// Setup lines
let filterLinesStart= "ffmpeg -y -filter_complex \\";
let filterLinesGenerator = ""; // generator lines
let filterLinesText = ""; // text lines
let filterLinesXStackInputs = "";
let filterLinesStacker = "";

let filterLinesLast = ` -t ${TIME} -an -pix_fmt yuv420p gradients2.mp4`;

// NEW - Build colours list
let colourParams = "";
colours.forEach((colour, index, colours) => {
    colourParams += `:c${index}=${colour}`;
});

// Loop through lines
allGradientsTypes.forEach((gradient, index, gradients) => {
    filterLinesGenerator += `gradients=duration=${TIME}:type=${gradient}${colourParams}[out${index}];`; // NEW - add colours to command
    filterLinesText += `[out${index}]drawtext=fontfile=${TEXT_FONT}\
:text=${gradient}:fontsize=${FONT_SIZE}:${TEXT_POS}:fontcolor=${TEXT_COLOR}[text_out${index}];`;
    filterLinesXStackInputs += `[text_out${index}]`;

    if (Object.is(gradients.length - 1, index)) {
        // Finish up lines
        filterLinesGenerator += `\\`;
        filterLinesText += '\\';
        filterLinesStacker = `xstack=inputs=${allGradientsTypes.length}:grid=${2}x${2}`;
        filterLinesXStackInputs += ``;
    } else {
        filterLinesGenerator += `\\\n`;
        filterLinesText += `\\\n`;
    }
});

// Build command...
const command = `${filterLinesStart}
"${filterLinesGenerator}
${filterLinesText}
${filterLinesXStackInputs}${filterLinesStacker}"\\
${filterLinesLast}`;

console.info("Command is: ", command);

exec(command, (error, stdout, stderr) => {
    if (error) {
        console.log(`error: ${error.message}`);
        return;
    }

    console.log(`stdout: ${stdout}`);

    if (stderr) {
        console.log(`stderr: ${stderr}`);
    }
    // NEW - Show the colours
    console.info(`Generated a file with ${allGradientsTypes.length} gradient demos in a ${2} by ${2} grid with colours: ${colours}`);
    return;
});

This, to me, makes it a lot easier to understand how each gradient uses the colours it is given...well, I'm sure code like this will be useful again soon. Perhaps we should move on to the next generator - mandelbrot.

The `mandelbrot` generator

This filter will take you right back to the 80s or 90s with it's classic (and slightly cheesy) psychedelic effects. Here's one example of a rapid descent into the world of chaos fractals:

ffmpeg -f lavfi -i \
mandelbrot=end_pts=100 \
-pix_fmt yuv420p -t 30 mandelbrot1.mov

(Here it is the end_pts value that speeds up the journey).

Let's do something similar to last time in javascript to explore the different shading effect presets for the parameter inner:

const { exec } = require("child_process");

// Set up variables used to construct lines
const allInnerPresets = [
    'black',
    'convergence',
    'mincol',
    'period',
];

const TEXT_FONT = '\'fonts/crystal-radio-kit/crystal radio kit.ttf\'';
const TEXT_POS = 'x=(w-text_w)/2:y=(h-text_h)/10';
const TEXT_COLOR = 'black';
const FONT_SIZE = 35;
const TIME = 30;
// Setup lines
let filterLinesStart= "ffmpeg -y -filter_complex \\";
let filterLinesGenerator = ""; // generator lines
let filterLinesText = ""; // text lines
let filterLinesXStackInputs = "";
let filterLinesStacker = "";

let filterLinesLast = ` -t ${TIME} -an -pix_fmt yuv420p mandelbrotDemo.mp4`;

// Loop through lines
allInnerPresets.forEach((innerPreset, index, allInnerPresets) => {
    filterLinesGenerator += `mandelbrot=inner=${innerPreset}:end_pts=100[out${index}];`;
    filterLinesText += `[out${index}]drawtext=fontfile=${TEXT_FONT}\
:text=${innerPreset}:fontsize=${FONT_SIZE}:${TEXT_POS}:fontcolor=${TEXT_COLOR}[text_out${index}];`;
filterLinesXStackInputs += `[text_out${index}]`;

    if (Object.is(allInnerPresets.length - 1, index)) {
        // Finish up lines
        filterLinesGenerator += `\\`;
        filterLinesText += '\\';
        filterLinesStacker = `xstack=inputs=${allInnerPresets.length}:grid=${2}x${2}`;
        filterLinesXStackInputs += ``;
    } else {
        filterLinesGenerator += `\\\n`;
        filterLinesText += `\\\n`;
    }
});

// Build command...
const command = `${filterLinesStart}
"${filterLinesGenerator}
${filterLinesText}
${filterLinesXStackInputs}${filterLinesStacker}"\\
${filterLinesLast}`;

console.info("Command is: ", command);

exec(command, (error, stdout, stderr) => {
    if (error) {
        console.log(`error: ${error.message}`);
        return;
    }

    console.log(`stdout: ${stdout}`);

    if (stderr) {
        console.log(`stderr: ${stderr}`);
    }

    console.info(`Generated a file with ${allInnerPresets.length} mandelbrot demos in a ${2} by ${2} grid`);
    return;
});

This yields even more trippiness:

As a next step, we could use a similar process to handle the outer parameter for example, using a nested loop. This is also something we might return to later. Moving on though, we can use very similar code for the next filter which is mptestsrc.

The `mptestsrc` filter

mptestsrc is yet another option for generating useful test patterns and is apparently based on the Mplayer test filter. It only generates patterns of size 256x256 and has an option to cycle through all of them which is handy (and otherwise something we might do using the concat filter):

ffmpeg -f lavfi -i \
mptestsrc=d=40\
:max_frames=100 \
-pix_fmt yuv420p mptestsrc.mov

That gives us 4 seconds of each test source:

It becomes apparent that the size is in fact different for some of the tests, which could be useful to know. We could try applying a similar approach to the above javascript, but using this list of filters instead:

const allPresets = [
    'dc_luma',
    'dc_chroma',
    'freq_luma',
    'freq_chroma',
    'amp_luma',
    'amp_chroma',
    'cbp',
    'mv',
    'ring1',
    'ring2'
];

which produces:

The mysterious section '15.10'...

We could theoretically do a similar thing with another generator - a generator that is basically another list of test sources. It doesn't seem to have a group name, but at the time of writing it is under section 15.10. This is the list:

const allPresets = [
    'allrgb', 
    'allyuv',
    'color',
    'colorchart',
    'colorspectrum',
    'haldclutsrc',
    'nullsrc',
    'pal75bars',
    'pal100bars',
    'rgbtestsrc',
    'smptebars',
    'smptehdbars',
    'testsrc',
    'testsrc2',
    'yuvtestsrc'
];

However, in this particular case several of the filters are differently sized and best viewed in full resolution, so we can't really present them all at once (well, I'm sure we could really and maybe we will, although not right now). This is an example that is probably best suited to the approach of concatenating the test source outputs (much like the mptestsrc). That, too, is an exercise for a later article. Here are a couple of the available test sources though:

ffmpeg -f lavfi -i \
allrgb \
-update 1 -frames:v 1 testSrc1.png

(this one is static so I present it as an image here).

ffmpeg -f lavfi -i \
testsrc \
-pix_fmt yuv420p -t 6 testSrc2.mov

Various other generators...

So we have several more generators to explore... Just to round things off nicely here are a couple more - there is an implementation of The Game of Life as source video in which you can alter various parameters:

ffmpeg -f lavfi -i \
life=ratio=0.1:death_color=#FF0000:\
life_color=#00FF00:mold_color=yellow:mold=1:s=400x400 \
-t 30 life3.mp4

And there are also some other simpler Fractal generators - the sierpinski and cellauto test sources shown here:

All three of the above have many parameters to test out, so would be ideal to present with different parameters in an xstack demo. Also something to explore in the future.

Generators with plug-in capability

Beyond that, we still have yet to explore two larger generators. These are the openclsrc generator which is a way of using OpenCL code to generate video sources, and the frei0r_src generator which enables using frei0r scripts as video generators. As you might expect, there is a lot to examine there. And that is what we'll do in a later article.

Finally, all of the videos in this article were converted to gifs with code such as:

ffmpeg -t 6 -i testSrc2.mov \
-vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" \
-loop 0 testSrc2.gif

Alan Allard is a developer at Eyevinn Technology, the European leading independent consultancy firm specializing in video technology and media distribution.

If you need assistance in the development and implementation of this, our team of video developers are happy to help out. If you have any questions or comments just drop us a line in the comments section to this post.

DEV Community: Eyevinn Video Dev-Team Blog

Creating a web video application

Setup the React Typescript project

Create a Video Player component

Use the Player component

Make the web application available online

Conclusion

Stream video with open web services

In this guide

Prerequisites

Get an API Access Token and setup project

Setup video processing pipeline

Upload and process video

Transcode video with open web services

In this guide

Prerequisites

Get an API Access Token and setup project

Setup a storage for storing transcoded files

Setup video transcoder

Transcode a video file

Sever-Guided Ad Insertion Made Easy.

How SGAI Works

Compared to SSAI and CSAI

Our SGAI Proxy Tool

Steps to Use the Ad Proxy Tool

Client Side Ad Insertion Crash Course

What is CSAI?

VMAP

VAST

Displaying the Ads

How to quickly setup a continue watching endpoint for your video players

Create a database on Redis Cloud

Create service on Open Source Cloud

Generate subtitles with OpenAI Whisper and Eyevinn OSC

Stream from OBS to Twitch and Preview With SRT-WHEP

Prerequisites

Steps

Conclusion

Seamlessly Streaming SRT Content in Web Browsers

Background: The Nexus of SRT and WebRTC

Our Endeavor: Forging with Tools and Architecture

Architecture of Seamless Playback

Getting Started

Challenges: Overcoming Hurdles

Summary: The Odyssey of Integration

Adding support for subtitles in The Eyevinn Channel Engine

Introduction

Challenges

Delimitations

Implementation

Future Work

Interpreting HTML5 Video Events

Creating a Media Event Filter

Example Implementation

Live Example

Production Readiness

Event Sequence

Exploring video generators in FFMPEG...

The coreimagesrc generator

The gradients generator

The mandelbrot generator

The mptestsrc filter

The mysterious section '15.10'...

Various other generators...

Generators with plug-in capability

The `coreimagesrc` generator

The `gradients` generator

The `mandelbrot` generator

The `mptestsrc` filter