DEV Community

loading...
Cover image for Facial Detection in JavaScript using Tracking.js

Facial Detection in JavaScript using Tracking.js

aspittel profile image Ali Spittel ・Updated on ・4 min read

Earlier this week, I saw a really cool article about how to create Snapchat like filters in Python. I was curious whether similar opensource technology existed for the frontend. I found a couple resources:

I was really excited that this data-sciencey technology existed on the front-end instead of just in more traditional data science languages like Python or R.

I thought all of the above projects were very interesting, and they all had relatively similar star-level's on GitHub. I decided to use Tracking.js because the documentation was really good, and there were a bunch of examples which, for me is the easiest way to learn! I do wish there was better documentation on what was going on behind the scenes with this library -- I'm not sure what statistics are going on or how the tool works past implementation.

That being said, it was very easy to implement. I could easily extend the face camera example for the purposes of the app I ended up building!

The Learning Process

Since Tracking.js is a smaller library with less of a community behind it than I normally end up using, my learning was pretty limited to looking at the examples on their website. I did search Codepen, but the few pens on there seemed to be incomplete or very similar to the examples.

The Final Project

Warning: I definitely learned throughout this project that webcam selfies are super awkward! I have no idea how I had a webcam selfie as my profile pic on Facebook back in high school!

I started off by copying the face camera example on the tracking.js site. I ended up getting it working locally, with a few tweaks, and with downloading the tracking library locally.

the example code

Then, I found a couple PNG's online to layer on top to be the filters. I just used Google Image search and then copied them locally. I then implemented a simple algorithm for adding the filter to the face -- I just hard-coded the dimensions based on where the user's face was. It ended up looking like this:


I also tried to implement the dog face that Snapchat has, but the math turned out to be too specific to the person's face for something as quick as this project!

I then generalized the math and added the ability for the user to switch between the filters. The math was relatively easy -- the event listener that fires when the person moves on the screen returns an array of the coordinates of the faces within view. Then I would draw the filter on top using my improved coordinates. The key code looked like this:

    context.drawImage(img, rect.x + (filterX * rect.width),
      rect.y + (filterY * rect.height),
      rect.width * filterWidth,
      rect.height * filterHeight
    )
Enter fullscreen mode Exit fullscreen mode

After that, I polished up the CSS, which was really hard! The way this ended up working was layering a HTML Canvas over a video element, so getting the grid system to align everything was really tricky. I ended up plugging in CSS Grid for the first time to see if that would work. It ended up working, but the way I did it felt hacky. I will have to keep looking at CSS Grid in the future!

The app ended up looking like this:

My final output seems okay, it is by no means perfect, but to get it better I would probably have to create my own library or my own filters. I also really struggled to get more complex filters to work. I would have had to break them up into their parts -- like each ear and the nose -- and then figure out the math to add these to each face. I seemed to have issues linking the different facial features back together if I wanted to track the eyes or mouth separately, rather than the face as a whole.

If I wanted to put more into this project, I would probably have also tried to add some smoothing so that the filter jumps less when a person moved a tiny bit. Overall, the project probably could have been better but I achieved what I wanted to in order to learn the library.

Next Steps

Tracking.js is really cool and well documented! I think its a great library for doing something simple like this app. If I were to do this for a job or something larger, I would probably have to improve a lot of the functionality in the app. I also found the webcam API to be shockingly difficult to use -- I struggled to find examples of styling and picture taking online. I would also like to look into it more in the future. Overall, I had fun with this! It isn't perfect, but it was a good quick project.

Code
App

Part of my On Learning New Things Series

Discussion (18)

pic
Editor guide
Collapse
z0al profile image
z0al

Come on! How do know what I want to learn next? You are usually a step ahead of me ;)

Great post, thanks

Collapse
aspittel profile image
Collapse
ioedeveloper profile image
David Disu

You are awesome!!!

Collapse
kwabenberko profile image
Collapse
frontend_fairy profile image
Freyr H

Fantastic work! You could try implementing lerping to the position of the overlay to combat the jagged movements!

Collapse
semosem profile image
Semosem

Amazing

Collapse
goatsaretasty profile image
Adesola

This is super duper cool

Collapse
kidsclimatecha1 profile image
snow

dog And CatAndSkeletonAndLionAndTigerThat's it
Can you please add those filters inThat website
Thank you

Collapse
michael profile image
Michael Lee πŸ•

@juantalon hahahaha!!! Awesome :)

Collapse
aspittel profile image
Ali Spittel Author

Ah! That's super exciting!

Collapse
annarankin profile image
Anna Rankin

This is so cool! πŸ’― Did you have any issues with the library's speed at all?

Collapse
enzoftware profile image
Enzo Lizama Paredes

Just amazing <3

Collapse
saiky93 profile image
saiky93

Hi, I want to know how we can integrate tracking.js into angular 2 application?
I want to use tracking.js to detect the faces from the webcam and do face recognition.

Collapse
ananthavijay profile image
Ananthavijay

how to add take photo command while using this?

Collapse
ananthavijay profile image
Ananthavijay

How to add take photo command? so that user can take photo while using it?

Collapse
michael profile image
Michael Lee πŸ•

OMG! My 2yo LOOOOOVVVVVEEEESSSSS bunnies! I'm going to show her this and she is going to flip out love this!!! Thanks Ali!!!! This is awesome :)

Collapse
alanguir profile image
Alan Languirand

This is great!