DEV Community: Yonatan Bendahan

Using Electron to create videos (Canvas + FFmpeg)

Yonatan Bendahan — Wed, 14 Feb 2024 15:21:56 +0000

At Gling I faced the challenge of rendering captions with transparent bg from the browser Canvas to a video file. I had the code for painting the captions to the Canvas by keyframe (for playback purposes), but we needed to render and encode those frames into a file the user could use.

Using WebCodecs (Doesn't support transparent bg)

Initially, I tried using the new WebCodecs API to pass each Canvas image to a VideoFrame and encode them using VideoEncoder using a VP9 codec that supports transparency, And finally muxing the video stream into a webm file using webm-muxer.
That worked great (and relatively very fast), but then I saw that even though VP9 codec supports alpha channel (for the transparent background) it is not implemented in WebCodecs API (yet?).

So I had to use FFmpeg to process the raw frames into a codec and format the supports alpha channel.
But how do you transfer Canvas bitmap data to FFmpeg for encoding and muxing?

Part 1 - Creating frames image data

Assuming I need to create a video with a 60-second length, at 30 frames per second, that means I need to create 1,800 frames. For each frame, I should paint the right captions to the canvas, and convert it to raw image data.

// renderer.ts

function createFramesData(
  seconds: number, 
  framerate: number, 
  dimensions: { width: number, height: number },
  onFrame: (data: Uint8ClampedArray) => Promise<unknown>
) {
  const frames = 60 * 30;

  const canvas = new OffscreenCanvas(dimensions.width, dimensions.height);
  const ctx = canvas.getContext('2d', {
        alpha: true,
        willReadFrequently: true,
  }) as OffscreenCanvasRenderingContext2D;

  for (let i = 1; i <= frames; i++) {
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    // The function that paints to the canvas by the frame number
    paintCaptionFrame(ctx, i);
    const rawFrameImage = ctx.getImageData(x, y, width, height).data;
    await onFrame(rawFramImage);
  }
}

Part 2 - Sending raw frames to FFmpeg

Now it is the interesting part. We need to listen to the onFrame callback that sends the raw image data, and pipe it to a FFmpeg process. FFmpeg should do the magic: ingest raw frames, encode them to any codec we choose (one that supports alpha channel), and muxing it into a file.

// renderer.ts

const duration = 60;
const framerate = 30;
const dimensions = {
  width: 1920,
  height: 1080,
}
const processId = await startFFmpeg(framerate, dimensions);
await createFramesData( // The function from above
  duration, 
  framerate, 
  dimensions, 
  (data) => {
    return pipeFramesToFFmpegProcess(processId, data);
  }
);
await closeFFmpegPipe(processId);

To start the FFmpeg process, we need to create an IPC call to the Electron Node.js backend, which will create an FFmpeg process.

To understand Electron IPC mechanism, there is an official Electron guide here: https://www.electronjs.org/docs/latest/tutorial/ipc

In short, we need to have a listener in the Node.js backend, that listens for IPC calls and executes FFmpeg, pipes data to a process and closes the process std input.

In main.ts Node.js Electron backend

// main.ts

import { ChildProcess, execFile } from 'child_process';

const processes = new Map<number, ChildProcess>();

ipcMain.on('exec', (_, executable, args) => {
  const child = execFile(
    executable,
    args.map((arg) => arg.toString()),
    { maxBuffer: 100 * 1024 * 1024 },
  );
  childProcesses.set(child.pid, child);
  return child.pid;
});

ipcMain.on('pipeIn', (_, pid, data: ArrayBuffer) => {
  return new Promise<void>((res, rej) => {
    const child = childProcesses.get(pid);
    if (!child?.stdin) {
      rej(new Error('No child process found with pid ' + pid));
      return;
    }
    child.stdin.write(Buffer.from(data), 'utf8', (error) => {
      if (error) {
        rej(error);
      } else {
        res();
      }
    });
  });
});

ipcMain.on('endPipeIn', (_, pid) => {
    return new Promise<void>((res, rej) => {
    const child = childProcesses.get(pid);
    if (!child?.stdin) {
      rej(new Error('No child process found with pid ' + pid));
      return;
    }
    child.stdin.end(() => {
      res();
      childProcesses.delete(pid);
    });
  });
});

Then expose IPC calls in the renderer using a preload file:

// preload.ts

import { contextBridge, ipcRenderer } from 'electron/renderer';

contextBridge.exposeInMainWorld('api', {
  exec: (executable, args) => ipcRenderer.send('exec', executable, args),
  pipeIn: (pid, data) => ipcRenderer.send('pipeIn', executable, args),
  endPipeIn: (pid) => ipcRenderer.send('endPipeIn', pid),
});

Part 3 - Connecting everything

Now we can execute FFmpeg, and send the raw frames to the FFmpeg process for encoding and muxing.

//renderer.ts

const duration = 60;
const framerate = 30;
const dimensions = {
  width: 1920,
  height: 1080,
}
const processId = await startFFmpeg(framerate, dimensions);
await createFramesData( // The function from above
  duration, 
  framerate, 
  dimensions, 
  (data) => {
    return pipeFramesToFFmpegProcess(processId, data);
  }
);
await closeFFmpegPipe(processId);

function createFramesData(
  seconds: number, 
  framerate: number, 
  dimensions: { width: number, height: number },
  onFrame: (data: Uint8ClampedArray) => Promise<unknown>
) {
  const frames = 60 * 30;

  const canvas = new OffscreenCanvas(dimensions.width, dimensions.height);
  const ctx = canvas.getContext('2d', {
        alpha: true,
        willReadFrequently: true,
  }) as OffscreenCanvasRenderingContext2D;

  for (let i = 1; i <= frames; i++) {
    ctx.clearRect(0, 0, canvas.width, canvas.height);
    // The function that paints to the canvas by the frame number
    paintCaptionFrame(ctx, i);
    const rawFrameImage = ctx.getImageData(x, y, width, height).data;
    await onFrame(rawFramImage);
  }
}

startFFmpeg(
  framerate: number, 
  dimensions: { width: number, height: number });
) {
  return window.api.exec('ffmpeg', [
    '-f',
    'rawvideo',
    '-pix_fmt',
    'rgba',
    '-s',
    `${dimensions.width}x${dimensions.height}`,
    '-r',
    framerate,
    '-i',
    '-',
    '-c:v',
    'libvpx-vp9',
    '-pix_fmt',
    'yuva420p',
    'out.webm',
  ]);
}

pipeFramesToFFmpegProcess(pid: number, data: Uint8ClampedArray) {
  return window.api.pipeIn(pid, data);
}

closeFFmpegPipe(pid: number) {
  return window.api.endPipeIn(pid);
}

And that's all, at the end you will have a webm file with the video containing the frames created by the canvas.

Some things might be missing here (like waiting for the ffmpeg process to complete), I removed them to keep the example as simple as possible.
You don't have to use vp9 codec, there are other codecs there that have alpha channel (but not many)

We hire at Gling

I'm always looking for talented developers to join our small, but excellent, team. If you want to join us and build a product that users love, with a focus on video editing on the web, feel free to send me an email to yonatan@gling.ai. We are fully remote and hire globally! Visit our website: https://gling.ai

An Opinionated Node.js Boilerplate Using TypeScript & Fastify

Yonatan Bendahan — Sun, 27 Sep 2020 07:01:05 +0000

After some time using Node.js for building backend APIs, I realised I have a ready-for-production boilerplate using my favorite build tools and libraries.

You can clone the boilerplate now using git from: https://github.com/yonathan06/fastify-typescript-boilerplate

Included in the boilerplate:

TypeScript (with module alias)
Development environment
Tests (using Jest)
Fastify
File based routing (using fastify-now)
Env vars config
CI with github actions
Docker image
Linting

TypeScript

Adding static types to your JavaScript code might look tedious, and it definitely feels like that when you just begin. But using TypeScript eventually saves a lot of “brain power”, especially when building large projects (try refactoring your models after using TypeScript and you’ll see what I’m talking about)
I used module alias in my typescript config (so imports from ‘…/lib/..’ will resolve to ‘@lib/..’). The problem was that TypeScript doesn’t switch back to regular module path when compiling, so had to create a little script to support that (resolve-module-alias.js) and call it after compilation.

Development

Using nodemon for watching file changes, that triggers TypeScript compiler and runs the server from /build

NOTICE: I am using TypeScript incremental builds, for fast recompiling. It can cause unexpected bugs, as I do not clear the build folder before each compilation. That means, if you remove a file from your src folder it might still present in the build folder. You will need to clear the build folder manually.

Fastify

In my humble opinion, Fastify is the best Node.js framework out there (yes, even better then Express). It comes with useful tools (schema validation, pino logger and more), and it is one of the fastest Node.js frameworks out there.

File Based Routing

A few months ago I built a Next.js app, with API serverless endpoints using Next.js API Routing.

After a while, the API backend got larger and larger, and I had to move to a standalone backend repo, to easily scale the backend API code(does that mean I used microservices? 😉)
The problem was, the Next.js API Routing uses a file based routing. For example /api/user.ts will resolve to my-domain.com/api/user endpoint.

I wanted to keep this feature when using Fastify, but didn’t find anything around (fastify-autoload was close to answer my needs, but wasn’t enough for me)

For that, I built a little plugin for Fastify, called fastify-now, that does just that: you pass it a folder path, and it loads the files there as routes (check the repo for documentation)

Env Vars

Using dotenv to load env files, and env-schema to validate them
I load different env files, depending on the NODE_ENV value. Meaning, if NODE_ENV=”production” it will try to load “production.env” from the root folder (the default is “development”).

Docker

A very simple docker file included, you can use it to build an image.

Notice that while building the docker image, it sets the NODE_ENV to “production”, so a production.env file must be present in the root folder before building the docker image.

Linting, Prettier, Testing & CI

Using Eslint and Prettier for a smooth development experience .

Using Jest with TypeScript for testing (suits for unit testing and e2e testing).

Even included a github action for running tests when pushing or PRing to master branch.

Database

I didn’t include any database connection and packages in this repo, feel free to add your own database

Hope this boilerplate repo will help you speed up your backend development process, without dropping important tools.

Looking for help and guidance with advanced web technologies? Feel free to contact me through my website: https://opsteacherpage.com/t/yonatanb

How to stay motivated while learning programming and new technologies

Yonatan Bendahan — Mon, 24 Aug 2020 16:34:13 +0000

One of programming main struggles is the learning process. whether you just started your programming journey, or you want to learn new technology. In this article, I want to share some of my personal tactics and strategies that help keep me motivated in this sometimes difficult journey.

Learning programming is not about memorizing. try to ditch the mentality of a high school/college student, going through books and lecture notes, remember as much as possible, and score high on exams.

Learning programming is about solving problems, to clear your way through building something.

My first tip is, focus on building, not on memorizing - start by choosing a project.

Choose a project

After you have in mind what you want to learn, choose a project.
Usually, I go for an idea I have for a while and excited about it. Then I define the parts of the project I want to implement (number of screens/number of features/number of components) and choose the language/technology I want to implement it with.

Break the project into milestones

This is an important part of learning and building projects.
Do not try to do the entire project at once, you will be exhausted and demotivated quickly. It is difficult to stay positive when the goal is too far away.
For example, if you want to build a twitter clone as a project, the first milestone can be to build the first page where you can post a tweet, and see a list of tweets (without a server connection). Then you can move forward to implementing other features: signup/sign in, likes, comments, and so on.
Make sure your steps are as small as possible

Priorities the subjects you want to learn

I know that you want to learn a lot of things at once, me too. But it's difficult, it will take a long time until you will feel a sense of progress.
So just like breaking down your project into milestones, try to priorities the languages/technologies you want to learn.
For example, You want to learn HTML & CSS? build only the UI without functionalities. Want to learn React? build the UI without building a server (you can use mock data or a public API), want to learn Node.js? build a server and test it using curl/postwoman/postman.

Focus is the key

Focusing is about you making the right decision on what to do when, and more importantly what not to do.
it will always be choosing to learn/build small parts, step by step, instead of doing all at once.

The actual learning

After you decided on a project, defined your milestones, and choose the languages/technologies you want to learn, it is time to do research.
Usually googling the subject you want to learn will be enough, but you can always ask a friend or online communities for resources. Don't be afraid to share your goals - what you want to build, and with which technologies.
Remember! When you have a well-defined project in mind, the actual Youtube lessons you watch, or articles you read, are less important. You will see you have a "flashlight" that guides your way while learning - the project you want to build.
You will also notice, that while you read or listen to teachers online, your mind will already process the information better, you will have a context of what can be done with the new knowledge you have, and how you can implement what you learn in the project you want to build.

Motivation

How to stay motivated is always an open question, and each one of you might have different techniques for that.
I believe that focusing on building something and moving the learning part to be a by-process of it, helps much with staying motivated, as long as you plan your path the way it fits you.
And who knows, maybe your next learning project will be the beginning of a successful startup :)

Hope it helps you in your programming journey. Happy coding!

Looking for personal/group training? Feel free to contact me through my website: https://yonatanbendahan.com

Photo by Julian Christian Anderson on Unsplash