DEV Community

loading...

TensorFlow + Next.js + TypeScript: Remove background and add virtual background image with web camera

Yuiko Ito
I'm a web developer.React, Next.js, TypeScript, Nuxt.js, Vue.js
・6 min read

Hello guys,

I have developed an application with BodyPix, that removes an original background and add a new virtual background image.

Image from Gyazo

In this article, I will explain how to develop this application.

DEMO→https://travel-app-three.vercel.app/
github→https://github.com/yuikoito/tensorflow-bodypix-sample

Setup Next.js application and install react-webcam

$ yarn create next-app <app-name>
$ cd <app-name>
$ touch tsconfig.json
$ yarn add --dev typescript @types/react
Enter fullscreen mode Exit fullscreen mode

Then, rename index.js and _app.js to index.tsx, _app.tsx.

Install webcam.

$ yarn add react-webcam @types/react-webcam
Enter fullscreen mode Exit fullscreen mode

Now, we are ready to develop :)

Install TensorFlow.js

When using TensorFlow.js with TypeScript, you need to be a little careful.

I wrote in another article as well, but you shouldn't do just yarn add @tensorflow/tfjs when you use TypeScript, otherwise you would get a type error.

Then, install like as following.

$ yarn add @tensorflow-models/body-pix @tensorflow/tfjs-core @tensorflow/tfjs-converter @tensorflow/tfjs-backend-webgl
Enter fullscreen mode Exit fullscreen mode

In this case, we are also using the body-pix model, so install it as well.

Set up to use bodyPix

As stated in the official documentation, it is very simple to use.
All you have to do is 1) import bodyPix, 2) load it, and 3) when the loading is complete, put the image data you want to analyze into an argument in the segmentPerson function.

Now, we're writing it in Next.js, so the code is like this.

// first, import all you need
import { useRef, useState, useEffect } from "react";
import Head from "next/head";
import "@tensorflow/tfjs-core";
import "@tensorflow/tfjs-converter";
import "@tensorflow/tfjs-backend-webgl";
import styles from "../styles/Home.module.scss";
import * as bodyPix from "@tensorflow-models/body-pix";
import Webcam from "react-webcam";

function Home() {
  const canvasRef = useRef<HTMLCanvasElement>(null);
  const webcamRef = useRef<Webcam>(null);
  // Manage the state of bodypixnet with useState
  const [bodypixnet, setBodypixnet] = useState<bodyPix.BodyPix>();

  // Run only when the page is first loaded
  useEffect(() => {
    bodyPix.load().then((net: bodyPix.BodyPix) => {
      setBodypixnet(net);
    });
  }, []);

  const drawimage = async (
    webcam: HTMLVideoElement
  ) => {
    const segmentation = await bodypixnet.segmentPerson(webcam);
    console.log(segmentation);
  };

  const clickHandler = async () => {
    const webcam = webcamRef.current.video as HTMLVideoElement;
    const canvas = canvasRef.current;
    // Make the canvas, webcam, and video size all the same size.
    webcam.width = canvas.width = webcam.videoWidth;
    webcam.height = canvas.height = webcam.videoHeight;
    const context = canvas.getContext("2d");
    context.clearRect(0, 0, canvas.width, canvas.height);
    // If it is clicked before bodypixnet is set, it will cause an error, so just in case.
    if (bodypixnet) {
      drawimage(webcam);
    }
  };
  return (
    <div className={styles.container}>
      <Head>
        <title>Travel App</title>
        <meta name="description" content="Generated by create next app" />
        <link rel="icon" href="/static/logo.jpg" />
      </Head>
      <header className={styles.header}>
        <h1 className={styles.title}>Title</h1>
      </header>
      <main className={styles.main}>
        <div className={styles.videoContainer}>
          <Webcam audio={false} ref={webcamRef} className={styles.video} />
          <canvas ref={canvasRef} className={styles.canvas} />
        </div>
        <div className={styles.right}>
          <h4 className={styles.title}>{t.select}</h4>
          <div className={styles.buttons}>
            <button onClick={clickHandler}>
              Button
            </button>
          </div>
        </div>
      </main>
    </div>
  );
}

export default Home;
Enter fullscreen mode Exit fullscreen mode

Let's make sure that the bodypix is working correctly.

When the button is clicked, all elements of the webcam, canvas, and video data are set in the same size.

In drawimage, we can parse the video data (webcam) by putting it in the argument of bodypixnet.segmentPerson. (The following part)

const segmentation = await bodypixnet.segmentPerson(webcam);
Enter fullscreen mode Exit fullscreen mode

If the segmentation is correctly output to the log, everything is fine.

image.png

By the way, if you just want to blur the background or separate colors for people and others, you can easily do so as follows.

const coloredPartImage = bodyPix.toMask(segmentation);
const opacity = 0.7;
const flipHorizontal = false;
const maskBlurAmount = 0;
const canvas = document.getElementById('canvas');
// Draw the mask image on top of the original image onto a canvas.
// The colored part image will be drawn semi-transparent, with an opacity of
// 0.7, allowing for the original image to be visible under.
bodyPix.drawMask(
    canvas, img, coloredPartImage, opacity, maskBlurAmount,
    flipHorizontal);
Enter fullscreen mode Exit fullscreen mode

If need, read https://github.com/tensorflow/tfjs-models/tree/master/body-pix#output-visualization-utility-functions.

With bodyPix.drawMask, you can easily specify the transparency, blur the border, flip the left and right sides, and so on.

Here, we will not use bodyPix.drawMask, but use drawImage() directly on canvas.

Remove the background to make a virtual background

Although I wrote remove background, I am not actually hollowing out the background to create a transparent image.

I've been doing some research, and it seems that if we want to completely remove the background, we may need to use WebGL.

Reference→Using BodyPix segmentation in a WebGL shader

Therefore, I decided to use destination-out of canvas.
(xor is also possible.)

As you can see in the figure below, destination-out remove overlapping areas.

image.png

So, if we make a temporary canvas element and put the image obtained by bodyPix.toMask() on it, only the mask area will be removed and we can achieve the virtual background.

You can also see in the log that the contents of bodyPix.toMask() are stored as ImageData.

image.png

OK, now it's time to write a code!

  const drawimage = async (
    webcam: HTMLVideoElement,
    context: CanvasRenderingContext2D,
    canvas: HTMLCanvasElement
  ) => {
    // create tempCanvas
    const tempCanvas = document.createElement("canvas");
    tempCanvas.width = webcam.videoWidth;
    tempCanvas.height = webcam.videoHeight;
    const tempCtx = tempCanvas.getContext("2d");
    const segmentation = await bodypixnet.segmentPerson(webcam);
    const mask = bodyPix.toMask(segmentation);
    (async function drawMask() {
      requestAnimationFrame(drawMask);
      // draw mask on tempCanvas
      const segmentation = await bodypixnet.segmentPerson(webcam);
      const mask = bodyPix.toMask(segmentation);
      tempCtx.putImageData(mask, 0, 0);
      // draw original image
      context.drawImage(webcam, 0, 0, canvas.width, canvas.height);
      // use destination-out, then only masked area will be removed
      context.save();
      context.globalCompositeOperation = "destination-out";
      context.drawImage(tempCanvas, 0, 0, canvas.width, canvas.height);
      context.restore();
    })();
  };
Enter fullscreen mode Exit fullscreen mode

Yay! Now that the background is transparent, then we can add the background image to the canvas element.

You can add the background image directly by using canvas.style.backgroundImage, but here I recommend you to add a class name because there is a possibility images may not load instantly.

So, I wrote like this.

  const clickHandler = async (className: string) => {
    const webcam = webcamRef.current.video as HTMLVideoElement;
    const canvas = canvasRef.current;
    webcam.width = canvas.width = webcam.videoWidth;
    webcam.height = canvas.height = webcam.videoHeight;
    const context = canvas.getContext("2d");
    context.clearRect(0, 0, canvas.width, canvas.height);
    canvas.classList.add(className);
    if (bodypixnet) {
      drawimage(webcam, context, canvas);
    }
  };
Enter fullscreen mode Exit fullscreen mode

css is as following.

.turky {
  background-image: url(../assets/turky.jpg);
  background-size: cover;
}
Enter fullscreen mode Exit fullscreen mode

Then, when the turky class is added, /assets/turky.jpg will be placed as the background image.

FYI, if you need to prepare several class names as I did, you need to remove the class name before you add a new one.

    if (prevClassName) {
      canvas.classList.remove(prevClassName);
      setPrevClassName(className);
    } else {
      setPrevClassName(className);
    }
    canvas.classList.add(className);
Enter fullscreen mode Exit fullscreen mode

Then finish!

By the way, the background images are all I took before.
I hope to be able to travel soon!

This is a kind of advertisement, but I copied and pasted in ui-components, which I developed for the button layout.
It's convenient!

That's it!

This article is the tenth week of trying to write at least one article every week.

If you'd like, please take a look at my previous weekly posts!
See you soon!

Contact

Please send me a message if you want to offer a job or ask me something.

yuiko.dev@gmail.com
https://twitter.com/yui_active

Discussion (4)

Collapse
yuikoito profile image
Yuiko Ito Author

I posted a new article also using TensorFlow.js in Next.js and TypeScript.
Please have a look!

dev.to/yuikoito/tensorflow-next-js...

Collapse
okeeffed profile image
Dennis O'Keeffe

This is super cool.

Collapse
swatirajan7 profile image
Swati Rajan

Great article! Totally trying it out

Collapse
siarhei_siniak_marketing profile image
Siarhei Siniak

funny!