Yuiko Koyanagi

Posted on Jul 26, 2021

TensorFlow + Next.js + TypeScript: Remove background and add virtual background image with web camera

#tensorflow #machinelearning #nextjs #typescript

Hello guys,

I have developed an application with BodyPix, that removes an original background and add a new virtual background image.

In this article, I will explain how to develop this application.

DEMO→https://travel-app-three.vercel.app/
github→https://github.com/yuikoito/tensorflow-bodypix-sample

Setup Next.js application and install react-webcam

$ yarn create next-app <app-name>
$ cd <app-name>
$ touch tsconfig.json
$ yarn add --dev typescript @types/react

Then, rename index.js and _app.js to index.tsx, _app.tsx.

Install webcam.

$ yarn add react-webcam @types/react-webcam

Now, we are ready to develop :)

Install TensorFlow.js

When using TensorFlow.js with TypeScript, you need to be a little careful.

I wrote in another article as well, but you shouldn't do just yarn add @tensorflow/tfjs when you use TypeScript, otherwise you would get a type error.

Then, install like as following.

$ yarn add @tensorflow-models/body-pix @tensorflow/tfjs-core @tensorflow/tfjs-converter @tensorflow/tfjs-backend-webgl

In this case, we are also using the body-pix model, so install it as well.

Set up to use bodyPix

As stated in the official documentation, it is very simple to use.
All you have to do is 1) import bodyPix, 2) load it, and 3) when the loading is complete, put the image data you want to analyze into an argument in the segmentPerson function.

Now, we're writing it in Next.js, so the code is like this.

// first, import all you need
import { useRef, useState, useEffect } from "react";
import Head from "next/head";
import "@tensorflow/tfjs-core";
import "@tensorflow/tfjs-converter";
import "@tensorflow/tfjs-backend-webgl";
import styles from "../styles/Home.module.scss";
import * as bodyPix from "@tensorflow-models/body-pix";
import Webcam from "react-webcam";

function Home() {
  const canvasRef = useRef<HTMLCanvasElement>(null);
  const webcamRef = useRef<Webcam>(null);
  // Manage the state of bodypixnet with useState
  const [bodypixnet, setBodypixnet] = useState<bodyPix.BodyPix>();

  // Run only when the page is first loaded
  useEffect(() => {
    bodyPix.load().then((net: bodyPix.BodyPix) => {
      setBodypixnet(net);
    });
  }, []);

  const drawimage = async (
    webcam: HTMLVideoElement
  ) => {
    const segmentation = await bodypixnet.segmentPerson(webcam);
    console.log(segmentation);
  };

  const clickHandler = async () => {
    const webcam = webcamRef.current.video as HTMLVideoElement;
    const canvas = canvasRef.current;
    // Make the canvas, webcam, and video size all the same size.
    webcam.width = canvas.width = webcam.videoWidth;
    webcam.height = canvas.height = webcam.videoHeight;
    const context = canvas.getContext("2d");
    context.clearRect(0, 0, canvas.width, canvas.height);
    // If it is clicked before bodypixnet is set, it will cause an error, so just in case.
    if (bodypixnet) {
      drawimage(webcam);
    }
  };
  return (
    <div className={styles.container}>
      <Head>
        <title>Travel App</title>
        <meta name="description" content="Generated by create next app" />
        <link rel="icon" href="/static/logo.jpg" />
      </Head>
      <header className={styles.header}>
        <h1 className={styles.title}>Title</h1>
      </header>
      <main className={styles.main}>
        <div className={styles.videoContainer}>
          <Webcam audio={false} ref={webcamRef} className={styles.video} />
          <canvas ref={canvasRef} className={styles.canvas} />
        </div>
        <div className={styles.right}>
          <h4 className={styles.title}>{t.select}</h4>
          <div className={styles.buttons}>
            <button onClick={clickHandler}>
              Button
            </button>
          </div>
        </div>
      </main>
    </div>
  );
}

export default Home;

Let's make sure that the bodypix is working correctly.

When the button is clicked, all elements of the webcam, canvas, and video data are set in the same size.

In drawimage, we can parse the video data (webcam) by putting it in the argument of bodypixnet.segmentPerson. (The following part)

const segmentation = await bodypixnet.segmentPerson(webcam);

If the segmentation is correctly output to the log, everything is fine.

By the way, if you just want to blur the background or separate colors for people and others, you can easily do so as follows.

const coloredPartImage = bodyPix.toMask(segmentation);
const opacity = 0.7;
const flipHorizontal = false;
const maskBlurAmount = 0;
const canvas = document.getElementById('canvas');
// Draw the mask image on top of the original image onto a canvas.
// The colored part image will be drawn semi-transparent, with an opacity of
// 0.7, allowing for the original image to be visible under.
bodyPix.drawMask(
    canvas, img, coloredPartImage, opacity, maskBlurAmount,
    flipHorizontal);

If need, read https://github.com/tensorflow/tfjs-models/tree/master/body-pix#output-visualization-utility-functions.

With bodyPix.drawMask, you can easily specify the transparency, blur the border, flip the left and right sides, and so on.

Here, we will not use bodyPix.drawMask, but use drawImage() directly on canvas.

Remove the background to make a virtual background

Although I wrote remove background, I am not actually hollowing out the background to create a transparent image.

I've been doing some research, and it seems that if we want to completely remove the background, we may need to use WebGL.

Reference→Using BodyPix segmentation in a WebGL shader

Therefore, I decided to use destination-out of canvas.
(xor is also possible.)

As you can see in the figure below, destination-out remove overlapping areas.

So, if we make a temporary canvas element and put the image obtained by bodyPix.toMask() on it, only the mask area will be removed and we can achieve the virtual background.

You can also see in the log that the contents of bodyPix.toMask() are stored as ImageData.

OK, now it's time to write a code!

  const drawimage = async (
    webcam: HTMLVideoElement,
    context: CanvasRenderingContext2D,
    canvas: HTMLCanvasElement
  ) => {
    // create tempCanvas
    const tempCanvas = document.createElement("canvas");
    tempCanvas.width = webcam.videoWidth;
    tempCanvas.height = webcam.videoHeight;
    const tempCtx = tempCanvas.getContext("2d");
    const segmentation = await bodypixnet.segmentPerson(webcam);
    const mask = bodyPix.toMask(segmentation);
    (async function drawMask() {
      requestAnimationFrame(drawMask);
      // draw mask on tempCanvas
      const segmentation = await bodypixnet.segmentPerson(webcam);
      const mask = bodyPix.toMask(segmentation);
      tempCtx.putImageData(mask, 0, 0);
      // draw original image
      context.drawImage(webcam, 0, 0, canvas.width, canvas.height);
      // use destination-out, then only masked area will be removed
      context.save();
      context.globalCompositeOperation = "destination-out";
      context.drawImage(tempCanvas, 0, 0, canvas.width, canvas.height);
      context.restore();
    })();
  };

Yay! Now that the background is transparent, then we can add the background image to the canvas element.

You can add the background image directly by using canvas.style.backgroundImage, but here I recommend you to add a class name because there is a possibility images may not load instantly.

So, I wrote like this.

  const clickHandler = async (className: string) => {
    const webcam = webcamRef.current.video as HTMLVideoElement;
    const canvas = canvasRef.current;
    webcam.width = canvas.width = webcam.videoWidth;
    webcam.height = canvas.height = webcam.videoHeight;
    const context = canvas.getContext("2d");
    context.clearRect(0, 0, canvas.width, canvas.height);
    canvas.classList.add(className);
    if (bodypixnet) {
      drawimage(webcam, context, canvas);
    }
  };

css is as following.

.turky {
  background-image: url(../assets/turky.jpg);
  background-size: cover;
}

Then, when the turky class is added, /assets/turky.jpg will be placed as the background image.

FYI, if you need to prepare several class names as I did, you need to remove the class name before you add a new one.

    if (prevClassName) {
      canvas.classList.remove(prevClassName);
      setPrevClassName(className);
    } else {
      setPrevClassName(className);
    }
    canvas.classList.add(className);

Then finish!

By the way, the background images are all I took before.
I hope to be able to travel soon!

This is a kind of advertisement, but I copied and pasted in ui-components, which I developed for the button layout.
It's convenient!

That's it!

This article is the tenth week of trying to write at least one article every week.

If you'd like, please take a look at my previous weekly posts!
See you soon!

Contact

Please send me a message if you want to offer a job or ask me something.

yuiko.dev@gmail.com
https://twitter.com/yui_active

Latest comments (6)

Benson Ruan • Jul 26 '23 • Edited

Great to know that someone have exact same idea with me.
One small suggestion is to mirror the webcam when it is facing user
check out my demo :
bensonruan.com/selfie-anywhere-per...

sysmaya • Mar 21 '23 • Edited

Hello.
Do you have an example of how to remove the background of images (not webcam).
I'm looking for an example of removing backgrounds in javascript on the client side. no NodeJS.