loading...
Cover image for Facial Detection in JavaScript using Tracking.js

Facial Detection in JavaScript using Tracking.js

aspittel profile image Ali Spittel Updated on ・4 min read

Earlier this week, I saw a really cool article about how to create Snapchat like filters in Python. I was curious whether similar opensource technology existed for the frontend. I found a couple resources:

I was really excited that this data-sciencey technology existed on the front-end instead of just in more traditional data science languages like Python or R.

I thought all of the above projects were very interesting, and they all had relatively similar star-level's on GitHub. I decided to use Tracking.js because the documentation was really good, and there were a bunch of examples which, for me is the easiest way to learn! I do wish there was better documentation on what was going on behind the scenes with this library -- I'm not sure what statistics are going on or how the tool works past implementation.

That being said, it was very easy to implement. I could easily extend the face camera example for the purposes of the app I ended up building!

The Learning Process

Since Tracking.js is a smaller library with less of a community behind it than I normally end up using, my learning was pretty limited to looking at the examples on their website. I did search Codepen, but the few pens on there seemed to be incomplete or very similar to the examples.

The Final Project

Warning: I definitely learned throughout this project that webcam selfies are super awkward! I have no idea how I had a webcam selfie as my profile pic on Facebook back in high school!

I started off by copying the face camera example on the tracking.js site. I ended up getting it working locally, with a few tweaks, and with downloading the tracking library locally.

the example code

Then, I found a couple PNG's online to layer on top to be the filters. I just used Google Image search and then copied them locally. I then implemented a simple algorithm for adding the filter to the face -- I just hard-coded the dimensions based on where the user's face was. It ended up looking like this:


I also tried to implement the dog face that Snapchat has, but the math turned out to be too specific to the person's face for something as quick as this project!

I then generalized the math and added the ability for the user to switch between the filters. The math was relatively easy -- the event listener that fires when the person moves on the screen returns an array of the coordinates of the faces within view. Then I would draw the filter on top using my improved coordinates. The key code looked like this:

    context.drawImage(img, rect.x + (filterX * rect.width),
      rect.y + (filterY * rect.height),
      rect.width * filterWidth,
      rect.height * filterHeight
    )

After that, I polished up the CSS, which was really hard! The way this ended up working was layering a HTML Canvas over a video element, so getting the grid system to align everything was really tricky. I ended up plugging in CSS Grid for the first time to see if that would work. It ended up working, but the way I did it felt hacky. I will have to keep looking at CSS Grid in the future!

The app ended up looking like this:

My final output seems okay, it is by no means perfect, but to get it better I would probably have to create my own library or my own filters. I also really struggled to get more complex filters to work. I would have had to break them up into their parts -- like each ear and the nose -- and then figure out the math to add these to each face. I seemed to have issues linking the different facial features back together if I wanted to track the eyes or mouth separately, rather than the face as a whole.

If I wanted to put more into this project, I would probably have also tried to add some smoothing so that the filter jumps less when a person moved a tiny bit. Overall, the project probably could have been better but I achieved what I wanted to in order to learn the library.

Next Steps

Tracking.js is really cool and well documented! I think its a great library for doing something simple like this app. If I were to do this for a job or something larger, I would probably have to improve a lot of the functionality in the app. I also found the webcam API to be shockingly difficult to use -- I struggled to find examples of styling and picture taking online. I would also like to look into it more in the future. Overall, I had fun with this! It isn't perfect, but it was a good quick project.

Code
App

Part of my On Learning New Things Series

Posted on Dec 6 '17 by:

aspittel profile

Ali Spittel

@aspittel

Passionate about education, Python, JavaScript, and code art.

Discussion

markdown guide
 

Come on! How do know what I want to learn next? You are usually a step ahead of me ;)

Great post, thanks

 
 
 
 

Fantastic work! You could try implementing lerping to the position of the overlay to combat the jagged movements!

 

Ah! That's super exciting!

 
 
 

Hi, I want to know how we can integrate tracking.js into angular 2 application?
I want to use tracking.js to detect the faces from the webcam and do face recognition.

 
 

How to add take photo command? so that user can take photo while using it?

 

dog And CatAndSkeletonAndLionAndTigerThat's it
Can you please add those filters inThat website
Thank you

 

how to add take photo command while using this?

 
 
 

OMG! My 2yo LOOOOOVVVVVEEEESSSSS bunnies! I'm going to show her this and she is going to flip out love this!!! Thanks Ali!!!! This is awesome :)

 

This is so cool! πŸ’― Did you have any issues with the library's speed at all?