DEV Community

loading...
Cover image for Introducing Handsfree.js - Integrate hand, face, and pose gestures to your frontend πŸ–πŸ‘€πŸ–

Introducing Handsfree.js - Integrate hand, face, and pose gestures to your frontend πŸ–πŸ‘€πŸ–

midiblocks profile image Oz Ramos ・9 min read

Handsfree.js

Hello and thank you for checking out this introductory post, I'm so excited to share Handsfree.js with you! Handsfree.js is a client side library that helps you add hand, face, and pose estimation to your front end projects in a snap βœ¨πŸ‘Œ

Since this is an introductory post, I'll start by sharing some of the things I've made with it so that you can get an idea of what's possible. Once I've hopefully hyped you up a bit I'll then show you how to get started!

Handsfree.js can help you do quite a bit, and I'm using it to completely and totally handsfree-ify the web and therefore entire world around us. You can see how I plan to do that in my Master Plan or you can see me actually doing it on Twitter @Midiblocks.

OK! Let me show you what you can do with Handsfree.js ✨


Examples

Use it to trigger events

Just yesterday I released the Pincher Plugin which emits 24+ pinching events with 3 states - start, held, released - for pinching with your index, middle, ring, and pinky fingers. It's modelled after the Mouse Events and you can listen to them similarly with document.addEventListener(): https://handsfree.js.org/ref/plugin/pinchers.html

Pincher Plugin

Use it to scroll pages handsfree

Here's a Browser Extension I'm working on that helps you scroll websites handsfree. It uses the MediaPipe Hands model to track your hands. This GIF was actually really easy to make with the built in pinchScroll plugin, which enables this customizable functionality in a single line of code: https://handsfree.js.org/ref/plugin/pinchScroll.html

Handsfree Browser Extension

Use it to create new kinds of Assistive Technologies

This is one of my favorites, and it uses the "Face Pointer" plugin to allow you to move a pointer with your face, scroll pages, and click on things. It's powered by the Jeeliz Weboji model and a few face plugins.

Face Pointers

Use it to control desktop games

Here's me playing "Into the Breach" on my desktop with Face Pointers and Hand Pointers. These are super easy to make too, all I did was use the Face and Hand Pointer plugins and then stream them over to my desktop with Robot.js to trigger native mouse events:

Face Pointers Into the Breah

Hand Pointers Into the Breach

Handsfree Solitaire

Use it to make your own games

But why just play games when you can make them too!? Here are a few games I've made, which I plan on grouping together into a "Mario Party" like game where you roll dice to move on a board and then play these minigames with your friends at the end of each round.

Here is "DuckFace Hunt", "Flappy Pose", and "Handsfree Jenga":

DuckFace Hunt

Flappy Pose

Handsfree Jenga

Use it to control robots and drones

Of course, you're not limited to controlling things on the web or even desktop. With WebSockets you control anything connected to your computer, like this Universal Robot which I tried to puppeteer with my own head:

Handsfree robot

Handsfree UR5

Use it for art, music, and other experiences

There's so much more that you can do! Here are some other experiments like my upcoming "Diffusionist" app designed to help you make trippy art to the beat of music (check out my brand new Instagram for the audio version). I'm also making a WebXR DevTools Chrome Extension so that you can work on WebXR apps handsfree without XR equipment:

Diffusionist

Handsfree A-Frame

Handsfree WebXR


Getting Started

Great! So now that I've shown you a little of what you can do, let me show you how. Don't worry if this is overwhelming at first, it's more of an overview. I'll have lots of shorter and more focused tutorials coming soon πŸ™

If you clone my repo (and please give it a star πŸ€—) you can find a boilerplate in /boilerplate/cdn.html. I'll have many more soon 😊

Initializing and starting Handsfree

The easiest way to get started is with a CDN. If you'd like, you can create an HTML file and copy/paste this in without the need for a server:

<head>
  <!-- Import helper classes and styles -->
  <link
    rel="stylesheet"
    href="https://unpkg.com/handsfree@8.1.1/build/lib/assets/handsfree.css" />
</head>
<body>
  <!-- Import Handsfree.js in body (as it adds body classes) -->
  <script src="https://unpkg.com/handsfree@8.1.1/build/lib/handsfree.js"></script>

  <script>
    // Use the hand with defaults (and show the webcam with wireframes)
    handsfree = new Handsfree({
      showDebug: true,
      hands: true
    })

    // Start webcam and tracking (personally, I always like to ask first)
    handsfree.start()
  </script>
</body>
Enter fullscreen mode Exit fullscreen mode

You can also import with NPM. By default this will still load the models from a CDN as they are quite large (some are over 10Mb), but I have instructions for ejecting the models into your assets folder here: https://handsfree.js.org/#hosting-the-models-yourself

npm i handsfree
Enter fullscreen mode Exit fullscreen mode
handsfree = new Handsfree({
  showDebug: true,

  // Use the hand model with custom config
  hands: {
    // Always make sure to enable them
    enabled: true,

    // Let's track up to 4 hands. It's best to be kind and ask permission first tho!
    maxNumHands: 4,
  }
})

// Start webcam and tracking (personally, I always like to ask first)
handsfree.start()
Enter fullscreen mode Exit fullscreen mode

basic demo

For the full list of config options you can pass into Handsfree, see: https://handsfree.js.org/ref/prop/config.html#the-full-list

Working with the data

Of course that'll just show the wireframes over your hands, but it won't actually do anything yet. There are two main ways to work with Handsfree.js, and my preferred way is by creating plugins using handsfree.use(newPluginName, callback). I call them plugins because they "plug into" the main webcam loop that's started when you run handsfree.start().

Plugins run their callback on every webcam frame and receive all the data from all the running computer vision models. Here's a very simple plugin that simply console logs data. I'll call it "logger":

// Let's use our hands again
handsfree = new Handsfree({showDebug: true, hands: true})
handsfree.start()

// Let's create a plugin called "logger" to console.log the data
handsfree.use('logger', (data) => {
  // I like to always bail if there's no data,
  // which might happen if you swap out hands for the face later on
  if (!data.hands) return

  // Log the data  
  console.log(data.hands)

  // Do something if we are pinching with left [0] pinky [3]
  if (data.hands.pinchState[0][3] === 'held') {
    console.log('pinching with left pinky')
  }
})
Enter fullscreen mode Exit fullscreen mode

Once you create a plugin, it becomes available at handsfree.plugin.pluginName and comes with a few methods and properties. Most importantly, they get a .disable() and .enable() method:

handsfree.plugin.logger.enable()
handsfree.plugin.logger.disable()

// This is what the callback gets mapped to,
// and is what gets called on every frame that this plugin is enabled
handsfree.plugin.logger.onFrame
Enter fullscreen mode Exit fullscreen mode

If you need more advanced functionality then you can pass an object with specific hooks that'll run during various phases of the plugin. For example:

handsfree.use('advancedLogger', {
  // True by default
  enabled: true,

  // A list of strings for tagging this plugin.
  // Later you can bulk disable/enable these with: handsfree.enablePlugins(['tag1', 'tag2'])
  tags: [],

  // This special property can be adjusted later (or even before!) in various ways
  config: {},

  // Called immediately after the plugin is added, even if disabled
  // The `this` context is the plugin itself: handsfree.plugin.advancedLogger
  // If you need to create DOM elements or other setup, this is the method to do it in
  onUse () {},

  // Called when you .enable() this plugin
  onEnabled () {},
  // Called when you .disable() this plugin
  onEnabled () {}
})
Enter fullscreen mode Exit fullscreen mode

Using data without plugins

Sometimes you may only want to track just one frame, or use an image, canvas, or video element instead of a webcam, or you might be in a part of your app where you don't easily have access to your handsfree (like in node module). In these cases, you can just listen to events on the document.

Because these are events, the data you want is always in ev.detail.data

// This will get called on every frame
document.addEventListener('handsfree-data', ev => console.log(ev.detail.data))

// Listen to when the thumb and index (0) are pinched on any hand
document.addEventListener('handsfree-finger-pinched-0')

// Listen to when the right (1) thumb and pinky (3) are pinched
document.addEventListener('handsfree-finger-pinched-1-3')
Enter fullscreen mode Exit fullscreen mode

Also, know that you can always access the data directly on your handsfree instance:

console.log(handsfree.data.hands)
Enter fullscreen mode Exit fullscreen mode

Updating models and plugins

The real magic of Handsfree.js is in its ability to instantly swap out models and plugins. This is useful if different routes in your app have different handsfree user experiences. This is where the very powerful handsfree.update(config) comes into play. I use this everywhere on Handsfree.js.org to allow you to try out different demos without restarting the webcam.

handsfree.use takes in the same Config Object as when you instantiate Handsfree, but it does a few extra things:

  • It stacks changes, so if you only pass in handsfree.update({facemesh: true}) while you have hands turned on then you'll end up with both
  • It automatically handles loading in any required models and dependencies
  • Gives you the ability to also configure plugins, or turn them off completely

Here's an example

// Start with hands
const handsfree = new Handsfree({hands: true})
handsfree.start()

// Add facemesh
handsfree.update({facemesh: true})

// Replace both with pose
handsfree.update({
  hands: false,
  facemesh: false,
  pose: true
})

// Use Weboji and enable the Face Pointer plugins
handsfree.update({
  hands: false, facemesh: false, pose: false,
  weboji: true,

  plugin: {
    // Enable some plugins
    faceClick: true
    faceScroll: true,
    // Update the special .config properties of the plugins (this is so magical!)
    facePointer: {
      speed: {
        x: 2,
        y: 2
      }
    },
  }
})
Enter fullscreen mode Exit fullscreen mode

You can also bulk disable and enable plugins, even by tag:

// Disable and enable them all
handsfree.disablePlugins()
handsfree.enablePlugins()

// Disable all the "core" tagged plugins
handsfree.disablePlugins('core')

// Enable handsfree "browser" tagged plugins
handsfree.enablePlugins('browser')
Enter fullscreen mode Exit fullscreen mode

Each of the current plugins has a set of configs that you can update in real time. Just access their .config by name:

// Change the Face Pointer speed
handsfree.plugin.facePointer.config.speed.x = 2
handsfree.plugin.facePointer.config.speed.y = 2

// Set the threshold for how much you have to smile to click (0 - 1)
handsfree.plugin.faceClick.config.morphs[0] = .25
Enter fullscreen mode Exit fullscreen mode

Classes

Anytime something happens within Handsfree.js, a class is added to the document.body. This makes styling your app or showing and hiding elements based on the state super easy. Here are where you can find them:


Thanks for reading πŸ™

So that about covers the basics! I know that might have been overwhelming, but one of my New Year Resolutions is to write more and so I'll be releasing smaller, more focused tutorials about once a week covering one specific topic at a time.

I start my 2nd residency at the STUDIO for Creative Inquiry this month where I'll be exploring new ways to apply Handsfree.js. It's my full time thing so please leave me a comment below or stop by my Discord so that I can help you integrate Handsfree.js in new and creative ways.

If you'd like to learn a little bit about my story and what inspired me on this path 3 years ago, do check out my GitHub Sponsors. Thank you so much for reading this introduction, I can't wait to see what you do with Handsfree.js πŸ–πŸ‘€πŸ–

Discussion (38)

pic
Editor guide
Collapse
shaileshcodes profile image
Shailesh Vasandani

Amazing project! Super excited to see how this can be used for accessibility β€” I see it opening up entirely new avenues of interacting with the web, especially if paired with something like VR. The possibilities are endless!

Thanks so much for sharing!

Collapse
midiblocks profile image
Oz Ramos Author

Thank you! Accessibility is actually what initially inspired this project. There are many nights when I have a hard time falling asleep because of all the possibilities lol and by releasing this as a library and browser extension I'm hoping to provide the tools to help others explore them!

Collapse
braydentw profile image
Brayden W ⚑️

Looks awesome! and a bit familiar... πŸ˜‰

stark gif

🀯

Collapse
midiblocks profile image
Oz Ramos Author

100% what I think about every time I sit down to code haha

Collapse
braydentw profile image
Brayden W ⚑️

πŸ˜‚πŸ˜‚

Collapse
aidenybai profile image
Aiden Bai

I tried out handsfree around dec of 2019. Easily one of the coolest js libraries out there!

Collapse
midiblocks profile image
Oz Ramos Author

Oh thank you so much! I ended up taking a break for almost a year πŸ˜… but the break has allowed me to recharge and take this in a new and exciting direction

Now I'm solely focused on the journey and not the destination, and I've been having sooo much fun making it! Thanks for the comment :)

Collapse
tyler320sc profile image
Tyler Carroll

I’m a noob who will be building a portfolio in the next few months and I would LOVE to have a project in there made with handsfree.js! However, idk what would be plausible for someone with limited experience to make. Any ideas? Amazing lib btw!!!

Collapse
midiblocks profile image
Oz Ramos Author

Hi that would be amazing and also great question! The thing is there are so many applications so it's hard to suggest something πŸ˜… how I go about it is to just making something as normal (where you can use with mouse and keyboard) and then make it handsfree...my next batch of plugins will emit mouse and keyboard events, so you wouldn't even really need to do much with Handsfree.js other than to listen to the events in your app as you would any other mouse or keyboard event

I know how vague that response was πŸ˜… I'll end my next series of posts with an "Ideas" section to help you pick a direction to try and I'll include more boilerplate code and samples.

Thanks for the question and have fun with your portfolio!

Collapse
thomasbnt profile image
Thomas Bnt

Badass 🀩🀩🀩

Collapse
midiblocks profile image
Oz Ramos Author

Thanks! I have lists upon lists for follow up tutorials, can't wait!

Collapse
thomasbnt profile image
Thomas Bnt

Greaat ! You can notify me if you want when you posted a new tutorial πŸ˜ƒ

Thread Thread
midiblocks profile image
Oz Ramos Author

That's super encouraging, thank you! I'll make a few first and then ping ya πŸ™

Collapse
akbartus profile image
akbartus

Oz thank you for great solution for tracking. I wonder is it also suitable for mobile based experiences. I already tried MediaPipe hands before and it was not mobile friendly meaning it was supporting well only latest models of phones with bigger Ram. On older devices there was lag. Is handsfree good also for mobile web (Chrome, Mozilla)?

Collapse
iliark82 profile image
iliark82

Sooo Cool!

Collapse
sirseanofloxley profile image
Sean Allin Newell

WOWOW!! Big major stonks coming your way.

Collapse
chiubaca profile image
Alex Chiu

This is amazing!!!

Collapse
tqbit profile image
tq-bit

That's easily one of the coolest project I've ever seen.

Collapse
naowal profile image
Naowal Siripatana

Awesome, It​ can.expand to.many thing.

Collapse
nikhilmwarrier profile image
Nikhil M Warrier

Badass! Really cool! I am pretty darn sure it will be the jQuery of the AR/VR web...

Collapse
jaketracey profile image
Jake Tracey

This is incredible! Going to do some proof of concepts for users with limited mobility using Face pointer!

Collapse
midiblocks profile image
Oz Ramos Author

Thank you and please do! Here is the documentation for my current implementation of the Face Pointer along with the different configs. At the bottom is the current code for it if you'd like to see how it's done: handsfree.js.org/ref/plugin/facePo...

I'll be re-implementing the automatic face pointer calibrator from the last version soon. By default it works best when the webcam is directly in front of you but the calibrator allows it to be used from different positions (ex: when the webcam is attached to the arm of a wheelchair).

Collapse
benjaminguma profile image
benjaminguma

Ohh woww nice

Collapse
gorfadvijay profile image
Vijay Gorfad

I tried out handsfree today. Easily one of the coolest js libraries out there!

Collapse
midiblocks profile image
Oz Ramos Author

Thank you for trying it out πŸ™Œ

Collapse
martineboh profile image
Martin Eboh

Simply...Brilliance...best ever js library since 1998!

Collapse
midiblocks profile image
Oz Ramos Author

haha thank you! the docs are designed to look like a Windows 98 app so I love this comment so much πŸ’œ (I'm using 98.css)

Collapse
jenlooper profile image
Jen Looper

wow, this is incredible. Looking to do some hand shadows with this, great that you can have two hands showing!

Collapse
flyingduck92 profile image
Sekti W

Awesome man.

Collapse
lallenfrancisl profile image
Allen Francis

Wooo...the demos are next level

Collapse
j94 profile image
j-94

The number of tabs im going to accidentally open when in use this on p**hub.com....

Collapse
houdinii profile image
Brian B.

Oh, man, you just destroyed this week's productivity. This is the absolute coolest project I've seen in the past year. I can't wait to play!

Collapse
slaven3kopic profile image
Slaven Kopic

Very cool!!

Collapse
mayankkalbhor profile image
Mayank Kalbhor

Brilliant !! This is next generation thing.

Collapse
midiblocks profile image
Oz Ramos Author

Aaah thank you! I'm so excited to be sharing this!

Collapse
ahmedghazi profile image
Ahmed

Funny, i used something similar to track emotions and build a smile gate on my website. It is called clmtrackr.js.
You can see the result here ahmedghazi.com

Collapse
escornwell profile image
Eric Cornwell

Very cool. Possible post edit: in the code sample for advancedLogger should the last function be "onDisabled" or something like that?

Collapse
beginnercoder17 profile image
BeginnerCoder17

WOW! clap, clap, clap