DEV Community: Andreas Schallwig

Do not laugh — A simple AI powered game using TensorFlow.js and Electron

Andreas Schallwig — Tue, 16 Nov 2021 16:43:51 +0000

For quite a while now I’ve been evaluating how AI technology can be used to leverage the user experience of digital applications or even enable completely new UI / UX concepts.

Recently I stumbled upon several articles describing how to train a CNN (Convolutional Neuronal Network) to recognize a person’s emotion expressed by their facial gestures. This sounded like an interesting idea for creating a user interface so in order to test it I came up with a simple game called “Do not laugh”.

The game itself is dead simple. It will start playing a random funny video from YouTube and your only job is: Do not laugh! If the AI catches you laughing it’s game over. So I set out to create this game using Vue.js and Electron which has recently become my frontend frameworks of choice.

Using AI and machine learning technology inside HTML5 / JavaScript based applications is a rather new thing, so I decided to write down my experiences in this article and share some best practices with you. If you’re not interested in all the details you can also just download all the source code from my repository 👇👇

andypotato / do-not-laugh

A simple AI game based on Vue.js and Electron

How to add TensorFlow to an Electron app

If you previously played around a bit with AI or machine learning code chances are you have already seen a library called TensorFlow in action. TensorFlow is Google’s open source AI framework for machine learning (ML) and it is widely used for tasks like image classification — Just what we need for our little game. The only drawbacks are, it is a Python library and it can only use nVidia GPUs for acceleration. Yes, you absolutely want that GPU acceleration when doing anything ML related.

Enter TensorFlow.js and WebGL GPU acceleration

Luckily the TensorFlow team has ported the library to JavaScript and released TensorFlow.js (TFJS) which lets us use it inside an Electron app. And even better — They went the extra mile and added WebGL based GPU acceleration with support for any modern GPU and doesn’t restrict us to nVidia and CUDA enabled hardware any longer👋 👋 👋. Cheers to that🍺!

Making things even more simple with face-api.js

At this point we’re basically all set. We have a powerful library on our hand and there are pre-trained models available for emotion detection. Means we can save ourselves the time and hassle to train our own model.

While I was writing some initial code I found that Vincent Mühler had already created a high-level API called face-api.js built upon TensorFlow.js which wraps all the low-level operations and provides a convenient API for common tasks like face recognition and emotion detection. He also provides lots of example code on his GitHub repository so I was able to build my first emotion recognizer within a matter of minutes.

Putting the application together

Let’s start with the overall game architecture. Using Electron means we have a Chromium renderer process which will become our game UI. It will play back the video file, display your live camera feed and of course run the game logic — You laugh, you loose.

So where does the actual emotion detection take place in this scenario? To answer this question you need to keep in mind two things:

Real-time emotion detection is a resource-intensive task. Even a good GPU will probably just yield you just around 20–30 frames per second. This can cause the UI of your game to become unresponsive.

To keep things running smooth we need a way to move the heavy lifting into a separate process. Luckily Electron can do just that using hidden renderers. Therefore our final game architecture looks like this:

In this scenario we have face-api running inside a hidden renderer (“Detection Worker”), continuously evaluating emotions in the live camera stream. If an emotion is detected the worker will send an IPC message with the detected emotion to the game. Inside the game we can simply treat such messages as events and react accordingly.

Creating a new Vue.js / Electron App

To create your App boilerplate you can follow my instructions here. Start at the section “Getting your environment ready” and follow the instructions until you can successfully run the blank App using npm run electron:serve.

Next install face-api.js:

npm i --save face-api.js

The background detection worker process

First we create the background worker process which will handle all the detection work (aka. “heavy lifting”). Go ahead and create a file worker.html inside the public directory:

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Worker</title>
</head>
<body>
  <video id="cam" autoplay muted playsinline></video>
</body>
</html>

Note the <video> tag here. We will refer to this element in our worker to retrieve the image data from the camera stream.
Next create worker.js inside the src directory. This is a longer file and you can see the full version here. I will break it down and explain the most important parts to you:

import * as faceapi from 'face-api.js';
// init detection options
const minConfidenceFace = 0.5;
const faceapiOptions = new faceapi.SsdMobilenetv1Options({ minConfidenceFace });

Here we include and configure face-api.js. Internally face-api.js uses the SSD MobileNet v1 model to identify the face inside the picture therefore we need to provide a minConfidenceFace configuration parameter which configures the model to identify a face if it is at least 50% confident.

// configure face API
faceapi.env.monkeyPatch({
  Canvas: HTMLCanvasElement,
  Image: HTMLImageElement,
  ImageData: ImageData,
  Video: HTMLVideoElement,
  createCanvasElement: () => document.createElement('canvas'),
  createImageElement: () => document.createElement('img')
});

This part is a workaround to make face-api.js work properly inside an Electron app. In a normal browser environment this would not be required. However we enable nodeIntegration inside the hidden renderer which causes TensorFlow.js to believe we are inside a NodeJS environment. That’s why we need to manually monkey patch the environment back to a browser environment. If you skip over this step you will receive an error Uncaught (in promise) TypeError: Illegal constructor at createCanvasElement [...].

let loadNet = async () => {
  let detectionNet = faceapi.nets.ssdMobilenetv1;
  await detectionNet.load('/data/weights');
  await faceapi.loadFaceExpressionModel('/data/weights');
  return detectionNet;
};

Next we load the pre-trained model weights for the MobileNet V1 network and also the face expression model. As I wanted to make my App work offline I load them from the local URL /data/weights which translates to the /public/data/weights folder inside your project directory. You can download the required files from Vincent Mühler’s GitHub repository.

let cam;
let initCamera = async (width, height) => {
  cam = document.getElementById('cam');
  cam.width = width;
  cam.height = height;
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: false,
    video: {
      facingMode: "user",
      width: width,
      height: height
    }
  });
  cam.srcObject = stream;
  return new Promise((resolve) => {
    cam.onloadedmetadata = () => {
      resolve(cam);
    };
  });
};

The above code is pretty much the standard code for getting the video stream from a camera connected to your computer. We just wrap it inside a promise for convenience.

Now with everything in place we could directly continue with the detection part. I just add some convenience methods for sending the event messages to the game UI renderer:

let onReady = () => {
  notifyRenderer('ready', {});
};
let onExpression = (type) => {
  notifyRenderer('expression', { type: type });
};
let notifyRenderer = (command, payload) => {
  ipcRenderer.send('window-message-from-worker', {
    command: command, payload: payload
  });
}

The onReady event is triggered once the model has initialized “warmed up” and is ready for detections. Afterwards whenever an expression is detected, the onExpression event will forward the detected expression to the main process via IPC.

And now for the actual detection part:

let detectExpressions = async () => {
  // detect expression
  let result = await faceapi.detectSingleFace(cam, faceapiOptions)
    .withFaceExpressions();
  if(!isReady) {
    isReady = true;
    onReady();
  }
  if(typeof result !== 'undefined') {
    let happiness = 0, anger = 0;
    if(result.expressions.hasOwnProperty('happy')) {
      happiness = result.expressions.happy;
    }
    if(result.expressions.hasOwnProperty('angry')) {
      anger = result.expressions.angry;
    }
    if(happiness > 0.7) {
      onExpression('happy');
    } else if(anger > 0.7) {
      onExpression('angry');
    }
  }
  if(isRunning) {
    detectExpressions();
  }
};

This function is basically an infinite loop which will first detect a single face in the camera picture and then try to determine the facial expression (=emotion) on that face.

The result of detectSingleFace().withFaceExpression() will return a result object with an expressions dictionary, containing the probabilities (0–1) of a given expression like “angry” or “happy”. In my example I decided to set the threshold of the probability to 0.7 (70%) for triggering an onExpression event.

That’s it! We can now run the detection worker with this code:

loadNet()
.then(net   => { return initCamera(640, 480); })
.then(video => { detectExpressions(); });

Configure Vue.js to use a hidden background renderer

With your detection worker in place the next step is to configure both Vue.js and Electron to run your hidden renderer. Open up (or create) the file vue.config.js in your app’s root directory and insert / append the following configuration:

module.exports = {
  pages: {
    index: {
      entry: 'src/main.js', //entry for the public page
      template: 'public/index.html', // source template
      filename: 'index.html' // output as dist/*
    },
    worker: {
      entry: 'src/worker.js',
      template: 'public/worker.html',
      filename: 'worker.html'
    }
  },
  devServer: {
    historyApiFallback: {
      rewrites: [
        { from: /\/index/, to: '/index.html' },
        { from: /\/worker/, to: '/worker.html' }
      ]
    }
  }
};

This configuration will add a second entry point for the worker to Vue’s WebPack configuration and also create an alias to make it work during development.

Finally make the following modifications to background.js:

import { app, protocol, BrowserWindow, ipcMain } from 'electron'
import {
  createProtocol,
  installVueDevtools
} from 'vue-cli-plugin-electron-builder/lib';
const isDevelopment = process.env.NODE_ENV !== 'production';
let win;
let workerWin;
// check if the "App" protocol has already been created
let createdAppProtocol = false;
// Scheme must be registered before the app is ready
protocol.registerSchemesAsPrivileged([{
  scheme: 'app', privileges: {
    secure: true,
    standard: true,
    corsEnabled: true,
    supportFetchAPI: true
  }
}])
function createWindow () {
  // create the game UI window
  win = new BrowserWindow({
    width: 1024, height: 790,
    webPreferences: { nodeIntegration: true }
  });
  if (process.env.WEBPACK_DEV_SERVER_URL) {
    win.loadURL(process.env.WEBPACK_DEV_SERVER_URL)
  } else {
    win.loadURL('app://./index.html');
  }
  win.on('closed', () => {
    // closing the main (visible) window should quit the App
    app.quit();
  });
}
function createWorker(devPath, prodPath) {
  // create hidden worker window
  workerWin = new BrowserWindow({
    show: false,
    webPreferences: { nodeIntegration: true }
  });
  if(process.env.WEBPACK_DEV_SERVER_URL) {
    workerWin.loadURL(process.env.WEBPACK_DEV_SERVER_URL + devPath);
  } else {
    workerWin.loadURL(`app://./${prodPath}`)
  }
  workerWin.on('closed', () => { workerWin = null; });
}
function sendWindowMessage(targetWindow, message, payload) {
  if(typeof targetWindow === 'undefined') {
    console.log('Target window does not exist');
    return;
  }
  targetWindow.webContents.send(message, payload);
}
[...]
app.on('ready', async () => {
  if (isDevelopment && !process.env.IS_TEST) {
    // Install Vue Devtools
    try {
      await installVueDevtools()
    } catch (e) {
      console.error('Vue Devtools failed to install:', e.toString())
    }
  }
  if(!createdAppProtocol) {
    createProtocol('app');
    createdAppProtocol = true;
  }
  // create the main application window
  createWindow();
  // create the background worker window
  createWorker('worker', 'worker.html');
  // setup message channels
  ipcMain.on('window-message-from-worker', (event, arg) => {
    sendWindowMessage(win, 'message-from-worker', arg);
  });
})
[...]

Let’s look at the changes and additions I made here. The most obvious one is the second window workerWin which will be our hidden renderer. To make things more manageable I’ve created the function createWorker which kind of duplicates the default createWindow function, just with the specific requirements of the hidden worker.

Next I modified the app protocol to enable CORS and support for the fetch API. This is necessary to allow loading the model weights from the local /public folder.

Finally I added an IPC listener for the window-message-from-worker channel to relay incoming messages from the detection worker to the game via the sendWindowMessage method.

Adding the game user interface

I won’t go too much into detail on how to build the HTML / CSS and focus on how to to receive and process the “emotion” messages you receive from your detection worker. As a reminder, all the code is available on my GitHub repository for you to review.

Let’s look at this part of the source inside src/views/Play.vue:

this.$electron.ipcRenderer.on('message-from-worker', (ev, data) => {
  if(typeof data.command === 'undefined') {
    console.error('IPC message is missing command string');
    return;
  }
  if(data.command == 'expression') {
    if(data.payload.type == 'happy') {
      this.onLaugh();
      return;
    }
    if(data.payload.type == 'angry') {
      this.onAngry();
      return;
    }
  }
});

Here we start listening for incoming IPC messages via the message-from-worker channel. If the message contains an expression command we trigger a game event related to the message. In my game onLaugh would contain the logic when you get caught laughing, onAngry will re-start the game once it’s game over.

That’s it! If you’re interested in the finished project go ahead and download the code from my GitHub page. Feel free to modify it or use it as a base for your own game — and I’d love to see all the cool stuff you come up with!

Thank you very much for reading! Leave me some ❤️ if you liked my article - And if you have any comments, questions or suggestions please start a conversation in the comments

Rock, Paper, Scissors with hand gesture recognition

Andreas Schallwig — Tue, 16 Nov 2021 14:10:14 +0000

Create a fun browser game with JavaScript and TensorFlow.js

TL;DR, if you’re just here for the source code, you can download it from my repository below 👇👇👇

andypotato / rock-paper-scissors

Rock, Paper, Scissors game implemented with TensorFlow.js and FingerPose

Remember the Microsoft Kinect? That bulky sensor bar which was once the world’s fastest-selling consumer electronics device and later famously had to be unbundled from the Xbox One package?

It didn’t get much love from game developers, but the Kinect was actually a pretty decent piece of hardware. Head tracking and body pose estimation worked very well, and it could even detect simple hand gestures like “pinch” and “fist”. At an affordable retail price of around 99 USD, it quickly became a favorite in hacker and maker communities.

Sadly we haven’t seen a real successor to the 2014 “Kinect 2” model which was discontinued in 2018. The 2019 “Kinect Azure” does not fill this gap for multiple reasons:
It is much more expensive. You can find a used Kinect 2 for around 50 USD while the Azure development kits currently sells for prices north of 1,000 USD.

The Kinect 2 is a self-contained device that works out of the box. The Azure requires an additional Nvidia GPU (RTX 2060 or better) to work. Good luck finding one these days.

Luckily, with all the recent advances in machine learning and computer vision, there are now some great alternatives for adding Kinect-like features to your project.

Introducing TensorFlow.js

Developed by the Google Brain Team, TensorFlow is a popular machine learning (ML) library for the Python programming language. TensorFlow.js (TFJS) is its companion library for JavaScript. Quoting the official website:

TensorFlow.js is a library for machine learning in JavaScript. Develop ML models in JavaScript, and use ML directly in the browser or in Node.js.

TensorFlow.js is not just an amazing piece of software, but it also gives you access to a growing library of machine learning models, ready to use with your project.

In this tutorial, I will show you how to use a TFJS based machine learning model to create a “Rock, Paper, Scissors” game with gesture controls. The final game will be running in your web browser, using just HTML and JavaScript.

Let’s make a game!

The focus of this tutorial is on the hand gesture recognition part, not so much on game development. So to speed things up, I have already prepared a simple game UI for you. 👇👇

Still, to get a better idea of the game we’re building, let’s create a simple game design document.

The game design document

When developing a game, usually the best way to start is to describe the gameplay by creating a game design document. There are many ways to do this, for example by drawing a story board.

For a simple game like “Rock, paper, scissors”, we will just verbally describe how the game should work:

Game will load and show a “please wait” message during initialization
Player and opponent are shown next to each other
The first round of the game is started by the press of a key or button
A new round starts with a countdown, asking the player to get ready
Player should show a hand gesture (rock, paper or scissors)
Once a valid gesture is registered, it is compared to the computer gesture (which is still hidden at this point)
The result is evaluated according to the rules of “Rock, paper, scissors”
Winner is awarded one point, score display gets updated
Start next round (go to item #4)

With the UI out of the way, let’s get right to the good stuff.

Building a hand gesture recognizer

When building a Rock, Paper, Scissors game, the key challenge is to recognize the three hand gestures ✊🤚✌ inside a camera picture.

Before we look into the actual implementation of things, let’s first take a step back and think about how a high-level process to detect hand gestures would look like:

We are trying to determine hand gestures from a camera picture. Therefore, the first step is to detect whether a hand is actually inside the camera picture.
If a hand is detected, we estimate the position of all finger joints to trace the hand skeleton.

The hand skeleton detector returns 21 key points (also called “landmarks”): Four joints for each finger plus the wrist. This is our raw data which we will process further.

Working with the key points

The key points represent 2D coordinates, telling us the position of each skeleton point in the picture. This isn’t very useful to describe a hand gesture, as it is hard to compare two hand gestures based on the position of the joints. A hand can appear anywhere in the picture, it could be rotated, and people could be left- or right-handed.

Let’s try to find a better representation by describing a hand gesture using natural language:

Take the “Thumbs Up” gesture 👍 as an example: It can be described as “All four fingers fully curled and pointing to either the left or right. Thumb must not be curled and point upwards”.

Curl and pointing direction are a much more concise way of describing a hand gesture. They are independent of the size and position of the hand in the camera picture, also both can easily be deduced from the raw 2D coordinates.

This brings us to the next steps in our detection process:

Using the key points, we describe the curl and pointing direction for each detected finger.
Finally, we compare this description to a list of known hand gestures and check which one is the best match.

Great, we figured out how to detect hand gestures — At least in theory. Now let’s see how TensorFlow.js can help us to actually implement it:

Picking a suitable machine learning model

As I mentioned in the introduction, TensorFlow.js gives you access to a library of many useful machine learning models which you can immediately use within your own application.

One of the models is called "HandPose" offers “Hand pose detection”. The description reads:

A palm detector and a hand-skeleton finger tracking model. It predicts 21 3D hand key points per detected hand.

Sounds like this model can already cover steps (1) and (2) of our detection process and extract the raw data we need. Awesome! Let’s install it:

Install HandPose

First we need to install the model itself:

npm i --save @tensorflow-models/handpose

Next we install its TensorFlow.js dependencies:

npm i --save @tensorflow/tfjs-core
npm i --save @tensorflow/tfjs-converter
npm i --save @tensorflow/tfjs-backend-webgl

TensorFlow.js can use your computer’s GPU for additional performance. Almost any GPU (Nvidia, AMD, Intel) works as long as it supports WebGL. Yours most likely does, so make sure to install the WebGL backend to get a massive speed boost for free.

Processing the raw data

As I mentioned before, the raw data isn’t very useful for gesture detection. In order to work with the data, we need to transform it into “curl” and “pointing direction”. Luckily there is another library which will do just that:

andypotato / fingerpose

TFJS based finger pose classifier for hand landmarks detected by the MediaPipe Handpose model

(Hi 👋 I am also the author of the Fingerpose library)

Install the Fingerpose library with the following command:

npm i --save fingerpose

Define the gestures

Fingerpose expects you to define a hand gesture by describing direction and curl for each finger. Our game uses three distinct hand gestures, so we need to create one GestureDescription for each gesture.

Describe the rock gesture ✊:

The rock gesture is basically just you making a fist:

You bend your fingers into your palm, curling them under until the tip of each finger touches its corresponding base.
Then you bend the thumb down so that it falls across the top halves of the index and middle fingers.

This code describes a “rock” gesture as:

All fingers fully curled
Thumb must be half curled or not curled at all

In case you wonder about the second condition: It is physically impossible for you to fully curl your thumb (unless you are Houdini). Also, some people, when making a fist will place their thumb next to their index finger, effectively stretching it out. So we tell Fingerpose that both are acceptable and equally valid.

Next let’s look at the “paper” gesture 🖐:

No surprises here. To make a “paper” gesture, you have to stretch out all of your fingers and your thumb.

Lastly let’s have a look at “scissors” ✌️:

The “scissors” gesture closely resembles a “victory” sign. Index and middle fingers are stretched out. Ring and pinky should be half or fully curled. We don’t care about the thumb, so we just omit it.

In case you are wondering about the pointing direction of each finger: Unlike a “Thumbs up” gesture which has a completely different meaning when turned upside down, the gestures of our game do not change their meaning when mirrored or rotated. Therefore, the direction can be omitted to keep the descriptions simple.

Putting it all together

The implementation of the hand gesture recognizer consists of two parts:

One-time Initialization of the model
Process every video frame

Let’s see some code for the initialization process:

The code above will first create a Fingerpose GestureEstimator instance. Simply pass the list of known gestures to its constructor, and it is ready to be used.

Afterwards, the HandPose model will load and initialize. This may take some time as it will also download some files (the model weights) from the tfhub.dev website.

The last step is optional but will greatly improve the user experience. After you load the model, I recommend you “warm up” the model by making one single prediction on a sample image. This is because the first prediction can take quite a bit of time, while subsequent predictions will usually be much faster. If you make the first (slow) prediction part of the initialization process, it will not slow down your gameplay later.

Processing a video frame:

Again, let’s see some code first:

The code explained:

We predict the 21 key points (landmarks) by calling the estimateHands function of the HandPose model, passing the HTML video element as the first argument. The second parameter indicates whether the source video is horizontally flipped.
If key points were predicted, we pass them to Fingerpose together with a minimum required match score (0 to 10, 10 = perfect match). The match score tells Fingerpose to ignore possible matches with a lower score.
Depending on your minimum match score, Fingerpose could return an array with multiple possible matches. This code will pick the gesture with the highest match score (confidence).

The return value of this method will be the name of the gesture with the highest match score, or an empty string in case no gesture was detected.

Increase the detection stability

When you run the code above on a source video, you will notice that the predictions are occasionally unstable. In some frames, HandPose will detect “phantom hands” (false positive) or no hands at all (false negative). This can have an impact on gameplay.

One simple solution is to create a low-pass filter by combining detections from several consecutive frames to a single result. For example, we could wait for three consecutive frames to be classified as “Rock” gesture before we emit the “Rock detected” event:

Prevent blocking the game UI

Running a machine learning model can be quite taxing on your CPU and GPU. While TensorFlow.js is incredibly fast (especially when running with the WebGL backend), it can still cause your game UI to become unresponsive. Especially when you run the model on each frame of a video stream.

Again, there is a simple workaround to prevent the UI from locking up. You can wrap the predictGesture function inside a setTimeout call with a timeout of zero seconds. Check out this thread on StackOverflow to learn more about this workaround.

Below is some example code how to create a non-blocking detection loop:

Wrapping it up

With the code above, we have implemented a fast and stable gesture detector. Check out the full source code to learn how to integrate it in the final game:

andypotato / rock-paper-scissors

Rock, Paper, Scissors game implemented with TensorFlow.js and FingerPose

This is the end of my tutorial. Feel free to use my source code as a base for your own game or application. If you have any comments, questions or suggestions please start a conversation in the comments.

Also check out my other tutorial on how to create a fun game using facial expression detection.

andypotato / do-not-laugh

A simple AI game based on Vue.js and Electron

Thank you very much for reading this tutorial! Leave me some love ❤️ if you liked it — And I’d love to see all the cool stuff you come up with! ️