Liquid error: internal
Handsfree.js is a small wrapper library around web based computer vision libraries that I'm working on for the purpose of helping you interact with web pages handsfree. Currently it only handles head tracking through Jeeliz Weboji, but I also plan to add eye tracking, hand tracking, voice, and full body pose estimation over my next 100 days of code.
I also plan on adding integrations for popular libraries like P5.js, Babylon.js, A-Frame, and others!
But in this post, I'll just focus on how to get a simple Handsfree.js project going which involves three steps:
- Requiring dependencies
- Creating a
handsfreeinstance - Adding a plugin (a callback that runs on every video inference frame)
At the end, you'll have a red face controlled pointer like in the following:
Liquid error: internal
Adding a handsfree pointer
Because everything is already bundled for you, all you need to do is include the Handsfree.js JavaScript and Stylesheet in your DOM:
<!-- Require dependencies -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
<script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>
This adds the Handsfree class to your page, along with basic styles for the pointer. The next step is to create an instance of Handsfree:
const config = {}
const handsfree = new Handsfree(config)
You'll need one instance for each camera that you plan to use (if you have multiple cameras), but each instance can track multiple users. See the documentation on the config object for the defaults and other settings you can use.
And that's it! To start and stop tracking, use handsfree.start() and handsfree.stop().
Adding functionality, and using the pointer values
On each frame, your handsfree instance will have several properties that contain useful information:
// The x coordinate of the pointer on the screen
handsfee.head.pointer.x
// The y coordinate of the pointer on the screen (from 0 at the top)
handsfee.head.pointer.y
// The pointer element
handsfree.head.pointer.$el
// The pointer state ("", "mouseDown", "mouseDrag", "mouseUp")
handsfree.head.pointer.state
// The head position [x, y, scale]
handsfree.head.translation
// The head rotation [pitch, yaw, roll]
handsfree.head.rotation
// Head Morphs (how raised the eyebrows are, how smiley your smiling to each side, etc)
// @see https://github.com/handsfreejs/handsfree/wiki/Head
handsfree.head.morphs
The Handsfree class has a global loop for all instances which you can hook into with the use method:
// Create a simple plugin that displays pointer values on every frame
Handsfree.use('consoleLogger', (instance) => {
console.log(instance.head.morphs)
})
// Same plugin, but with destructuring
Handsfree.use('consoleLogger', ({head}) => {
console.log(head.morphs)
})
These are called plugins, where "consoleLogger" is the name of the plugin and instance is the handsfree instance running the plugin (handsfree = new Handsfree()). Adding multiple plugins with the same name overwrites the previous plugin, and to disable a plugin you can call handsfree.stop().
Handsfree.js ships with a few plugins - "head.click" and "head.vertScroll" - which add clicking functionality (with a smile gesture) and scrolling as in this tweet:
Liquid error: internal
A complete example
<!DOCTYPE html>
<head>
<!-- Require dependencies -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
<script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>
<style>
button {font-size: 24px; padding: 20px}
</style>
</head>
<body>
<!-- Let's always ask the user to start -->
<button onclick="handsfree.start()">Start Webcam</button>
<button onclick="handsfree.stop()">Stop Webcam</button>
<script>
// Create a new instance. Use one instance for each camera
const handsfree = new Handsfree({})
// Create a simple plugin that displays pointer values on every frame
Handsfree.use('consoleLogger', ({head}) => {
console.log(head.rotation)
})
</script>
</body>
Next Steps
I hope this gave you a good introduction into Handsfree.js. In this post I covered how to setup a basic pointer, and in future posts we'll be going over:
- How to use head morphs (eyebrows, smiles, kissy faces, and more)
- Get the users head pose (yaw, pitch, roll)
- Multiplayer support
- Complex plugins
- Integrations with popular frameworks
- and more
Follow me on Twitter at @HeyOzRamos if you'd like to follow along on my 100DaysofCode. Thanks for reading!
Oz
Updates
- 11/9 - Added references to a few new properties:
handsfree.head.rotation,handsfree.head.morphs. New docs at: https://github.com/handsfreejs/handsfree/wiki - 11/23 - Updated to reflect new v6 API
Oldest comments (1)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.