DEV Community: Oz Ramos

How I made Flappy Pose, Flappy Bird but where you flap with your arms

Oz Ramos — Tue, 03 Mar 2020 21:18:26 +0000

Oz Ramos

@checkboxoz

Day 5: I made a Flappy Bird but where you have to flap your arms! Try it on @Glitch: flappy-pose.glitch.me

Click the fish at top right to view the source or remix it! Made with @p5xjs and @ml5js #100DaysofCode

04:38 AM - 29 Feb 2020

56 238

Now that my muscles aren't sore from a day full of flapping, I wanted to take a moment to show you how I made the flapping portion of the game. For brevity, I'll only be covering the flapping portion, but if you'd like to see how I made the actual game itself I recommend watching this video:

Initializing PoseNet through ml5.js

ml5.js is a super friendly library for using Machine Learning on the web. It comes packaged with a few computer vision models, and the one we're interested in for flapping is PoseNet.

So the first thing I did was add ml5.js and p5.js to the page:

<script src="https://unpkg.com/ml5@0.4.3/dist/ml5.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/p5@0.10.2/lib/p5.js"></script>

The next step is to initialize PoseNet. This is done by passing a video element, a callback to run once it's done, and optimization options:

  // createCapture is a P5.js method that creates a webcam stream
  video = createCapture(VIDEO)
  video.hide()

  // Initialize posenet...easy peasy!
  posenet = ml5.poseNet(video, () => {console.log('posenet ready!'}, { maxPoseDetections: 1})

The next step is to watch for flaps. In my approach, I consider a flap to start when the wrists go above shoulder level and end when the wrists fall back below shoulder level.

To determine how this happens, we first need to listen for posenet poses. This is done with posenet.on('pose', callback). The callback here receives a pose object with 17 keypoints and their positions. The ones we're interested in are keypoints 9 and 10 (wrists) and 5 and 6 (shoulders).

To get flapping to work, we simply compare the keypoints y values and run a method when the flap occurs:

  posenet.on('pose', poses => {
    pose = poses[0].pose

    // Detect up flap
    // (Note: I add 40 to the y so we don't have to flap as high)
    if (
      pose.keypoints[9].position.y < pose.keypoints[5].position.y + 40 &&
      pose.keypoints[10].position.y < pose.keypoints[6].position.y + 40
    ) {
      hasFlappedUp = true
    }

    // Detect down flap
    if (
      hasFlappedUp &&
      pose.keypoints[9].position.y > pose.keypoints[5].position.y + 40 &&
      pose.keypoints[10].position.y > pose.keypoints[6].position.y + 40
    ) {
      hasFlappedUp = false

      // FLAP 🐦
      bird.flap()
    }
  })

That's all there is to it!

You can look through the source code on Glitch, or try the demo out yourself.

Thanks for reading and happy flapping!

Follow me on Twitter @CheckboxOz

Instagram SparkAR "clone yourself" segmentation effect with Handsfree.js and p5.js

Oz Ramos — Wed, 22 Jan 2020 18:53:42 +0000

Demo: https://editor.p5js.org/heyozramos/sketches/LVsjLSPt7

A while back I came across a really fun tweet by @cassidoo on Twitter

Cassidy Williams

@cassidoo

When your code works on the first try

18:48 PM - 23 Jul 2019

15970 56156

At the end of the clip, there's a scene where a million copies of her are singing together. It's a really cool effect and in this first tutorial in a series on Segmentation Effects, I'll show you how to recreate it with JavaScript!

Our tools

We'll be using:

p5.js in global mode. p5.js is a beginner-friendly creative coding library
Google's Bodypix. Bodypix is a segmentation and pose estimation computer vision library powered by TensorFlow. We'll initialize BodyPix through Handsfree.js
Handsfree.js is my library for interacting with web pages handsfree

<head>
    <script src="https://cdn.jsdelivr.net/npm/p5@0.10.2/lib/p5.js"></script>
    <script src="https://unpkg.com/handsfree@6.1.7/dist/handsfree.js"></script>
    <link rel="stylesheet" type="text/css" href="https://unpkg.com/handsfree@6.1.7/dist/handsfree.css">
</head>

Instantiating Handsfree.js

Handsfree.js is a library that wraps popular computer vision models like Jeeliz Weboji (head tracking), Google's BodyPix (segmentation and pose estimation), and soon, eye tracking through WebGazer and Handtracking through Handtrack.js. I originally started it as an assistive tech tool to help people browse the web handsfree through face gestures, but it also doubles as a controller for games, Algoraves, and other creative coding projects!

Oz Ramos

@heyozramos

Day 1 of #100DaysofCode and #100DaysofMLCode

My goal with Handsfree.js will be to wrap JavaScipt based computer vision libraries under a single API for the purpose of interacting with web pages (with head, body, hand tracking)

Here's a preview (link at end is broken, will fix!)

00:36 AM - 05 Nov 2019

12 42

So to begin, we need to instantiate Handsfree.js. We instantiate once for every camera that we want to use (if you have a multi-cam setup or using a mobile device with front/back webcams). Instantiation looks like this:

/**
 * This method gets called once automatically by p5.js
 */
function setup () {
  // Instantiate Handsfree. "H"andsfree is the class, "h"andsfree is the instance
  handsfree = new Handsfree({
    autostart: true,
    models: {
      head: {enabled: false},
      bodypix: {
        enabled: true,

        // Uncomment this if it runs slow for you
        // modelConfig: {
        //   architecture: 'MobileNetV1',
        //   outputStride: 8,
        //   multiplier: 0.75,
        //   quantBytes: 2
        // }
      }
    }
  })

  // Disable default Handsfree.js plugins
  Handsfree.disableAll()
}

Creating a buffer canvas

The next thing we'll need to do is create a "buffer" canvas. This canvas will contain the cutout, which we'll then paste into the p5 canvas a bunch of times:

function setup () {
  // ...

  // Create a canvas that'll contain our segmentation
  $buffer = document.createElement('canvas')
  $buffer.style.display = 'none'
  bufferCtx = $buffer.getContext('2d')
  document.body.appendChild($buffer)

  // This part's important, match the buffer dimensions to the webcam feed
  $buffer.width = handsfree.debugger.video.width
  $buffer.height = handsfree.debugger.video.height
  widthHeightRatio = $buffer.width / $buffer.height

  // Create a slider to adjust the number of clones
  slider = createSlider(3, 12, 4, 1)

  // Finally, create the p5 canvas
  renderer = createCanvas(600, 600)
  $canvas = renderer.canvas
  canvasCtx = $canvas.getContext('2d')
}

Creating a mask

(GIF source: https://blog.tensorflow.org/2019/11/updated-bodypix-2.html)

We'll be using BodyPix as a kind of "smart green screen" that's able to remove all the background pixels. We'll do this in a method called draw that is run automatically by p5 every few milliseconds:

function draw() {
  background(220);

  // Just for convenience
  const model = handsfree.model
  const body = handsfree.body

  // Only paint when we have data
  if (body.data) {
    // Create a mask, with all non segmented pixels as magenta
    mask = model.bodypix.sdk.toMask(
      body.data,
      {r: 0, g: 0, b: 0, a: 0},      // foreground
      {r: 255, g: 0, b: 255, a: 255} // background
    )

    // Paint the mask into a buffer canvas
    model.bodypix.sdk.drawMask(
      $buffer,
      handsfree.debugger.video,
      mask,
      1, 0, 0
    )

    // Make all magenta pixels in the buffer mask transparent
    let imageData = bufferCtx.getImageData(0, 0, $buffer.width, $buffer.height)
    for (let i = 0; i < imageData.data.length; i += 4) {
      if (imageData.data[i] === 255 && imageData.data[i + 1] === 0 && imageData.data[i + 2] === 255) {
        imageData.data[i + 3] = 0
      }
    }
    bufferCtx.putImageData(imageData, 0, 0)

    // Dimensions of each mask
    w = $buffer.width / slider.value()
    h = w * widthHeightRatio

    // Paste the mask a bunch of times
    for (row = 0; row < slider.value() * 3; row++) {
      for (col = 0; col < slider.value(); col++) {
        // Stagger every other row
        x = col * w - (w / 2) * (row % 2)
        y = row * h / 3 - h / 2 * (1 + row / 10)

        canvasCtx.drawImage($buffer, x, y, w, h)
      }
    }
  }
}

On each webcam frame, Handsfree.js captures segmentation data into handsfree.body.data which we'll use to create the mask.

    // Create a mask, with all background pixels as magenta (255, 0, 255)
    mask = model.bodypix.sdk.toMask(
      body.data,
      {r: 0, g: 0, b: 0, a: 0},      // foreground
      {r: 255, g: 0, b: 255, a: 255} // background
    )

    // Paint the mask into a buffer canvas
    model.bodypix.sdk.drawMask(
      $buffer,
      handsfree.debugger.video,
      mask,
      1, 0, 0
    )

Next, we'll go through each pixel and make any magenta pixels transparent:

    // Make all magenta pixels in the buffer mask transparent
    let imageData = bufferCtx.getImageData(0, 0, $buffer.width, $buffer.height)
    for (let i = 0; i < imageData.data.length; i += 4) {
      if (imageData.data[i] === 255 && imageData.data[i + 1] === 0 && imageData.data[i + 2] === 255) {
        imageData.data[i + 3] = 0
      }
    }
    bufferCtx.putImageData(imageData, 0, 0)

Finally, we paste the buffer into the p5 canvas a bunch of times:

    // Dimensions of each mask
    w = $buffer.width / slider.value()
    h = w * widthHeightRatio

    // Paste the mask a bunch of times
    for (row = 0; row < slider.value() * 3; row++) {
      for (col = 0; col < slider.value(); col++) {
        // Stagger every other row
        x = col * w - (w / 2) * (row % 2)
        y = row * h / 3 - h / 2 * (1 + row / 10)

        canvasCtx.drawImage($buffer, x, y, w, h)
      }
    }

And that's all there is to it!

Let me know in the comments if anything is unclear. I hope you enjoy playing around with this demo, it's the first in a set of tutorials on creating Instagram SparkAR-like filters for the web with p5.js.

Demo: https://editor.p5js.org/heyozramos/sketches/LVsjLSPt7

How to start and stop videos from a distance with your hand ✋

Oz Ramos — Sat, 21 Dec 2019 04:32:34 +0000

In this tutorial you'll learn how to use BodyPix, through Handsfree.js, to start and stop a YouTube video by raising up your hand ✋

The Setup

Let's start by adding our dependencies:

<head>
  <!-- Handsfree dependencies -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@6.1.3/dist/handsfree.css" />
  <script src="https://unpkg.com/handsfree@6.1.3/dist/handsfree.js"></script>

  <!-- YouTube dependencies -->
  <script defer src="https://www.youtube.com/iframe_api"></script>
</head>
<body>
  <div id="player"></div>
</body>

We'll then use the YouTube iframe API to instantiate a video:

/**
  * This method is automatically called by the YouTube API
  * @see https://developers.google.com/youtube/iframe_api_reference
  */
let player
function onYouTubeIframeAPIReady () {
  player = new YT.Player('player', {
    // I picked this video because it's ad free
    videoId: 'AFKktZURpJ8'
  })
}

Setting up BodyPix through Handsfree.js

The next step is to instantiate Handsfree with the bodypix model enabled, and then create a plugin to watch for when the hand is raised. Handsfree.js plugins are simply callbacks that run after a model does inference.

// Create one instance for each camera you plan to use. By default it uses the main webcam (the selfie camera on mobile devices)
const handsfree = new Handsfree({
  models: {
    // The head model is enabled by default, so lets disable it
    head: {enabled: false},
    bodypix: {enabled: true}
  }
})


/**
  * Create a plugin that stops the video when the hand is up
  */
Handsfree.use('hand.stopVideo', {
  // Whether we have entered "hands have been up" state
  isBeingHeld: false,

  /**
   * This method is called once per webcam frame by Handsfree.js
   * @param {Object} The handsfree instance. Here we destructure it to just get instance.body
   */
  onFrame ({body}) {
    // only continue if one of the wrists are visible
    if (!body.pose || !body.pose.keypoints
      || (body.leftWrist.score < 0.6 && body.rightWrist.score < 0.6)) return

    // Check if one wrist is level with that shoulder
    if (body.leftWrist.y <= body.leftShoulder.y || body.rightWrist.y <= body.rightShoulder.y) {
      if (!this.isBeingHeld) {
        this.toggleVideo()
        this.isBeingHeld = true
      }
    } else {
      this.isBeingHeld = false
    }
  },

  /**
    * Toggle the video
    */
  toggleVideo () {
    const state = player.getPlayerState()

    if (state === 1) {
      player.pauseVideo()
    } else {
      player.playVideo()
    }
  }
})

That's all there is too it!

This tutorial quickly covered how to instantiate a YouTube video using the YouTube Iframe API. We then learned how to instantiate Handsfree with the bodypix model, and check for when the hand is raised...but there's so much more you can do!

Here are some other links you might find useful:

The bodypix helpers that Handsfree.js provides
Handsfree.js GitHub
My Twitter @HeyOzRamos
The Demo
Demo Source

Thanks for reading and happy coding 👋

// Detect dark theme var iframe = document.getElementById('tweet-1208246401005547520-943'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1208246401005547520&theme=dark" }

Face Painting with P5.js

Oz Ramos — Mon, 02 Dec 2019 03:00:48 +0000

Liquid error: internal

In this tutorial you'll learn everything you need to start painting happy little trees with your face 🌳 This technique uses a "face pointer", which lets you control a pointer with your head and face gestures for clicking and more!

We'll be using the new Handsfree.js API to quickly setup our face pointer and P5.js to do the painting. Behind the scenes, Handsfree.js is powered by the Weboji head tracker.

So let's begin!

Setting up our environment

So the first thing we'll want to do is handle dependencies:

<!-- Handsfree.js -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.3/dist/handsfreejs/handsfree.css" />
<script src="https://unpkg.com/handsfree@6.0.3/dist/handsfreejs/handsfree.js"></script>

<!-- Demo dependencies: P5 and lodash -->
<script src="https://cdn.jsdelivr.net/npm/p5@0.10.2/lib/p5.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.15/lodash.min.js"></script>

<!-- Our P5 Sketch will go in here -->
<div id="canvas-wrap"></div>

This gives us a global Handsfree class object. The next thing that we do is create an instance of Handsfree. We need one instance for every webcam that we plan to use, but each instance can track multiple users (see config options):

const config = {};
handsfree = new Handsfree(config);

If at this point we were to run handsfree.start() then we would see a red face controlled cursor, along with the debug video feed.

Adding Functionality

To add functionality you add callbacks (I call them plugins) to the Handsfree class object with Handsfree.use("pluginName", opts).

Here pluginName can be anything and is there so that we can disable/enable plugins by name with Handsfree.disable('pluginName') or access them under the hood with Handsfree.plugins['pluginName'].

opts can either be a callback function to run on every webcam frame, or it can be an object with the following core properties and methods:

Handsfree.use("pluginName", {
  // Whether to start using this plugin immediately or not
  enabled: true,

  // Called once when the plugin is first used
  // - Use this to initialize stuff
  onUse({ head }) {},

  // Called on every frame loop
  // - Use this as your "game loop"
  // - This is the same as only passing a callback
  onFrame({ head }) {}
});

These callbacks pass in the handsfree instance, which we usually destructure to get the handsfree.head object...these two are equivalent:

Handsfree.use("test1", instance => {
  console.log(instance.head.rotation);
});

Handsfree.use("test2", ({ head }) => {
  console.log(head.rotation);
});

Knowing all that, let's define our "P5.facePaint" plugin:

Setup P5.js and remember get a reference to our canvas
Capture face gestures on every frame
Paint and/or change colors

Handsfree.use("p5.facePaint", {
  // Current pointer position
  x: 0,
  y: 0,
  // Last frames pointer position
  lastX: 0,
  lastY: 0,

  // Contains our P5 instance
  p5: null,

  /**
   * Called exactly once when the plugin is first used
   */
  onUse() {
    // Store a reference of our p5 sketch
    this.p5 = new p5(p => {
      const $canvasWrap = document.querySelector("#canvas-wrap");

      // Setup P5 canvas
      p.setup = () => {
        // Create a P5 canvas with the dimensions of our container
        const $canvas = p.createCanvas(
          $canvasWrap.clientWidth,
          $canvasWrap.clientHeight
        );
        $canvas.parent($canvasWrap);
        p.strokeWeight(6);
      };

      // Match canvas size to window
      p.windowResized = () => {
        p.resizeCanvas($canvasWrap.clientWidth, $canvasWrap.clientHeight);
      };
    });
  },

  /**
   * Called on every webcam frame
   */
  onFrame({ head }) {
    // Setup point coordinates
    this.lastX = this.x;
    this.lastY = this.y;
    // @todo: pointer origin should be at center, not corner (fix via CSS?)
    this.x = head.pointer.x + 10;
    this.y = head.pointer.y + 10;

    this.p5.stroke(this.p5.color(strokeColor));

    // Draw lines
    if (head.state.smirk || head.state.smile) {
      this.p5.line(this.x, this.y, this.lastX, this.lastY);
    }

    // Change colors with eyebrows
    if (head.state.browLeftUp) this.updateColor(1);
    else if (head.state.browRightUp) this.updateColor(-1);
  },

  /**
   * Throttled to 4 times a second
   * - Please see source code for this tutorial for the rest of this method
   * @see https://glitch.com/edit/#!/handsfree-face-painting?path=app.js
   */
  updateColor: _.throttle(function(step) {}, 250, { trailing: false })
});

That's all there is to it!

This tutorial quickly went over how to setup a P5 Sketch to work with Handsfree.js. It's still not perfect and I plan to create an official P5.js - Handsfree.js integration soon, but it should be enough to get you going!

Make sure to check out the source for the rest of the code as I omitted some non-Handsfree.js stuff for brevity.

Thanks and have fun 👋

How to get a JS.ORG subdomain served through a gh-pages branch

Oz Ramos — Thu, 14 Nov 2019 21:41:15 +0000

JS.org is a free service started by Stefan Keim in 2015 that provides custom subdomains for your GitHub Pages. What I especially love about the service is that it makes it easy for users of your projects to remember the URL...in my case I have a library called Handsfree.js hosted on handsfree.js.org!

The easiest way to do this is to follow their setup guide, but in this topic I'll take things further by explaining how to create an NPM script that commits your build directory to a gh-pages branch on the same repo.

Step 1 - Naming your repo

In order for this to work, your desired subdomain must match your username or project name exactly (see exceptions). If you have a project located at github.com/username/project then the subdomain could be username.js.org or project.js.org.

Step 2: Add a CNAME

The next step is to add a file called CNAME (no extension) to the root of your project's build directory. A CNAME maps a GitHub Pages domains to a custom domain.

Inside the CNAME, add a single line with the .js.org subdomain you'd like:

    project.js.org

Here's a sample project directory:

    .gitignore
    dist/            <---- Build directory
      |- js/
      |- index.html
      |- CNAME       <---- Contains "project.js.org"
    src/
    package.json

Step 3: Commit build directory to `gh-pages`

The next step is to commit your build directory to a gh-pages branch. This is a special branch that when committed to will expose its contents at username.github.io/project. Because we have a CNAME in there, it'll tell GitHub to use that domain instead.

Step 4: Make a Pull Request to the js.org repo

The final step is to make a pull request to the js.org repo. Simply add a new line to that file, where the key is your project name and the value is the subdomain you added in the CNAME. In our case it would be:

    {
      ...
      "project": "project.js.org"
      ...
    }

Bonus Step - npm script

If you've followed the steps so far, you'll see your project live at the project.js.org in around 24 hours!

However, because we are serving out of the /dist folder you'll quickly discover that you have to cd into your /dist folder each time, add your CNAME, and commit it to the gh-pages branch. That's really annoying and so the following steps will help automate that.

First, add the shell package to your projects devDependencies with npm install -D shelljs. This will help us automate the terminal commands we would normally use to deploy from the dist folder.

Next, open your project's package.json file and add the following script:

    {
      "scripts: {
        ...
        "deploy": "node deploy.js"
        ...
      }
    }

Then add a file named deploy.js to your projects root. This script will:

Navigate to your build directory
Create a CNAME record for you with your JS.org subdomain
Commit the contents of your build directory to the gh-branch of your

    /**
     * Helper script for building and deploying
     * Deploys out of /dist/
     */

    // Change these to match your Repo URL and the js.org domain
    const gitRepo = "https://github.com/username/project"
    const domainName = "project.js.org"
    const distPath = "dist"

    // build
    const pkg = require('./package.json')
    const shell = require('shelljs')

    // navigate into the build output directory
    shell.cd(distPath)

    // Create the CNAME and commit the directory
    shell.exec(`echo ${domainName} > CNAME`)
    shell.exec('git init')
    shell.exec('git add -A')
    shell.exec(`git commit -m "deploy docs for ${pkg.version}"`)

    // Push the project to the gh-pages branch
    shell.exec(`git remote add origin ${gitRepo}`)
    shell.exec('git push origin master:gh-pages -f')

    shell.cd('-')

Now, whenever you build your project you can just run npm run deploy to deploy your build directory to your new js.org domain, even if the build directory itself .gitignored!

Gotchas

There are a few gotchas with using JS.org:

You don't own the subdomain
If a more popular project comes along with the same name, it's possible that they can take it

Thanks for reading!

In this topic we learned how get a js.org subdomain, which pulls its contents from our build directory. We also explored an automation script to commit our build directory to the project's gh-branch.

Let me know if you get stuck at any stage, this was a bit tricky for me to figure out but in my opinion is definitely worth it!

Puppeteering Emojis with face morphs through Handsfree.js

Oz Ramos — Wed, 13 Nov 2019 04:46:55 +0000

Liquid error: internal

Face morphs are powerful properties of many computer vision models which represent how much a certain facial feature is activated. They can be used to puppeteer 3D models and 2D cartoons, or to trigger events like sounding an alert when a driver becomes drowsy or to snap a photo when you make a perfect smile 😊

Usually these numbers range from either [0, 1] or [-1, 1], and with Weboji through Handsfree.js I've abstracted those to a few new properties I call "morph activations" like instance.head.state.eyesClosed and instance.head.state.pursed . We'll explore these properties by making a simple "emoji puppeteering" app.

Setting up

As usual, we'll include our dependencies. These new properties are available on 5.0.5:

    <!-- Handsfree dependencies -->
    <link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css" />
    <script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>

Next, we'll add a start/stop button and an element to hold our Emoji:

    <button onclick="handsfree.start()">Start Webcam</button>
    <button onclick="handsfree.stop()">Stop Webcam</button>

    <div id="emoji">😀</div>

Finally, we'll grab a reference to the emoji element and instantiate Handsfree:

    const $emoji = document.querySelector('#emoji')
    window.handsfree = new Handsfree()

Adding our Emoji plugin

Now we'll add a plugin named "emoji". If you recall from our getting started tutorial, a plugin is simply a labeled callback that runs on every webcam frame: Handsfree.use('nameOfPlugin', ({head}) => {}). Let's start the emoji plugin:

    Handsfree.use('emojify', (pointer, instance) => {
      let emoji = '😐'
      let isFlipped = false

      // Instance.head.state contains a list of activated morphs
      // We just map it to a shorthand here so we don't have to type it all out each time
      let state = head.state

      // Let's start with some easy ones
      if (state.mouthOpen) emoji = '😃'
      if (state.browsUp) emoji = '🙄'
      if (state.smile) emoji = '🙂'

      // Some emojis can be made by combining activations
      if (state.eyesClosed && state.pursed) emoji = '😙'
      if (state.mouthOpen && state.eyesClosed && state.browsUp) emoji = '😂'
      if (!state.mouthClosed && state.pursed && state.browsUp) emoji = '😲'

      // Here we flip the emoji if the user smirks the other way
      // A smirk happens if and only if a user smiles to one side
      if (state.smirk && state.browsUp) {
        if (state.smileLeft) isFlipped = true
        emoji = '😏'
      }

      // Aplly transforms
      $emoji.style.transform = `perspective(1000px)
            rotateX(${-head.rotation[0]}rad)
            rotateY(${head.rotation[1]}rad)
            rotateZ(${-head.rotation[2]}rad)
            scale(${isFlipped ? -1 : 1}, 1)`

      // Show the emoji
      $emoji.innerText = emoji
    })

As you can see, it's quite easy to mix and match activations! For a complete list of head activations, check out the wiki page for head properties.

Check out my demo to see what other emojis I've mapped. Order can sometimes matter!

Configuring activation thresholds

To configure the threshold for those, that is, how pursed your lips need to be to activate head.state.pursed, check out the config section.

For instance, let's say that you're building an accessibility tool for someone who can't quite smile to the right all the way. In this case, you might lower the activation threshold:

    handsfree = new Handsfree({
        head: {
            threshold: {
                // The default iss 0.7
                smileRight: 0.2
            }
        }
    })

Going further

This post explained how to use the new morph activation properties of Handsfree.js, but you can take this way further than that! Remember, handsfree.head.morphs contains a set of morph values from 0, 1 which you can use to morph 3D models in real time and so much more.

Thanks for reading!

We have one more head property to learn, position, which tells us where in 3D space the head is. This can be useful for things like AR apps, as a "zooming" gesture, and other things we'll explore soon.

Here are some other links to keep you going:

Try the demo
or view its source
Getting Started with Handsfree.js
Controlling YouTube 360 videos handsfree
Handsfree.js GitHub
Weboji's GitHub
Wiki
Follow me here and on Twitter @HeyOzRamos to follow along in this projects 100DaysofCode!

Have fun coding 👋

Updates

11/23/19 - Made updates to reflect new v6 API

A look back at week 1 of my #100DaysofCode

Oz Ramos — Mon, 11 Nov 2019 21:00:00 +0000

I joined #100DaysofCode because it provides structure and reinforces commitments, something that's important if you're like me and always starting (but not finishing) side projects 😅

My goal with this round is to work on Handsfree.js - a wrapper library around computer vision models for interacting with pages handsfree. My mission is to help you leverage computer vision without needing to know how each model works, and my vision is a fully handsfree web!

The following is a look back at what I did over my first week of the challenge

Day 1 - 4

Oz Ramos

@heyozramos

Made a game called "Smile Tiles" to test the accuracy of Handsfree.js over time: handsfree.js.org/#/smile-tiles

- Click Start Webcam on top right
- Smile/smirk to click
- Black tiles increase score/time
- White tiles resets everything

Day 4 of #100DaysofMLCode and #100DaysofCode

23:31 PM - 07 Nov 2019

3 6

Most of my focus was on getting a simple "face pointer" working. The idea is that by knowing the your head pose, we can project an imaginary laser beam from between your eyes and onto the screen. Then through different face gestures we can treat the pointer like a mouse by using the native MouseEvent() API.

This means that any mouse based listeners you have on your existing page will work out of the box without any extra coding! To test that the idea works, I created a simple game where you have to click the black tiles in a certain amount of time.

I hope to make it more accurate over time 😅

Day 5 - 6

Oz Ramos

@heyozramos

Added a more exciting 360 video so that you can really check out the handsfree controls: handsfree-youtube-360.glitch.me

Also wrote a step-by-step tutorial on how to set this demo up with Handsfree.js: dev.to/heyozramos/con…

Day 6 of #100DaysofMLCode and #100DaysofCode

03:57 AM - 10 Nov 2019

3 0

Because I'm already using handsfree.head.rotation to calculate where to place the cursor, we can use that data to control different things on the page. To test this, I made a simple demo that controls a YouTube 360 video with your head.

Day 7

Oz Ramos

@heyozramos

Started on a face to Emoji tutorial, this time to explore face morphs and CSS 3D bindings. It still goes haywire sometimes tho so I'll share link with tutorial tomorrow 🤡

Day 7 of #100DaysofMLCode and #100DaysofCode

05:31 AM - 11 Nov 2019

7 10

One of the final things to test with handsfree.head are the morph values, which are floats between 0 and 1 that measure how much a part of your face is being activated. To test this, I started working on a demo which matches an emoji to your face!

On this day, I also created a "Pitch Deck" which I'll be using to help me get grants and funding for the project over the next few months. You can take a peek through the slides on the metrics I'm using to hopefully justify funding

Oz Ramos

@heyozramos

Here is my strategy for taking Handsfree.js to 1K stars on GitHub over the next 6 months: slideshare.net/OzRamos1/hands…

I don't think I'll actually get there 😅 but I did this as an exercise to get myself into the mindset I need to give it a try!

21:10 PM - 10 Nov 2019

0 1

Next Week

I hope to finish the Emoji demo by tomorrow, which will include several new events to Handsfree.js, like isSmiling, isWinking, etc. I really love the process of iterating on the library by first creating a demo to see what needs to be done and then adding features.

The final handsfree.head feature set will be based on the head's position in 3D space, which I'll demo by creating a simple 3D face painting app (smile to draw, move head to move brush). I also hope to start working on a plugin for P5.js and start applying to grants.

Thanks!

Thanks for reading, here are some other topics I created this week:

Have fun coding!

Controlling YouTube 360 videos handsfree with Handsfree.js

Oz Ramos — Sun, 10 Nov 2019 02:08:24 +0000

Liquid error: internal

If you've ever tried watching a 360 video on YouTube on your desktop with a mouse or trackpad, you'll immediately have been frustrated by how awkward it is to control the camera. The reason is that they're really meant to be viewed through a VR headset with the aid of your phone's gyroscopic sensors.

In this tutorial we'll explore a handsfree alternative that lets you control the POV with your head, using the rotation properties exposed by the Weboji model through Handsfree.js. If you haven't already, check out my introduction to Handsfree.js to help you get started quickly.

Setting up the YouTube IFrame API

Fortunately, YouTube doesn't require an API key to get this started. Simply add the Handsfree.js dependencies along with the YouTube API library:

    <!-- Handsfree dependencies -->
    <link rel="stylesheet" href="https://unpkg.com/handsfree@6.1.4/dist/handsfreejs/handsfree.css" />
    <script src="https://unpkg.com/handsfree@6.1.4/dist/handsfreejs/handsfree.js"></script>

    <!-- YouTube dependencies, let's defer it so that our other code runs first -->
    <script defer src="https://www.youtube.com/iframe_api"></script>

The YouTube API will look for a onYouTubeIframeAPIReady method once it's loaded, so let's add our YouTube video logic in there. We'll also add a div#player element to transform into a video:

    <!-- We'll transform this element -->
    <div id="player"></div>

    <script>
    let YTPlayer

    function onYouTubeIframeAPIReady () {
      // Instantiate a new Player, selecting '#player' as the element to turn into a video
      YTPlayer = new YT.Player('player', {
        // I picked this video ID because it has no ads
        // @see https://www.youtube.com/watch?v=Crv1hbk9HuM&list=PLxf-CDjxvNVoxF27Pj3VV5pIy4SsqPNcI&index=5&t=0s
        videoId: 'Crv1hbk9HuM',
        // Lets autoplay it
        playerVars: { autoplay: 1 }    
      })
    }
    </script>

And that's it, if you run the above you should see a 360 video being played automatically!

Adding Handsfree controls

The next step is to add head controls with Handsfree.js. First we'll need to instantiate an instance of Handsfree. Then we'll create a plugin called youtube360 that simply maps the YouTube's camera with the users head pose:

    // Create one instance and autostart tracking
    const handsfree = new Handsfree()

    // Create a plugin called "youtube360" that runs on every webcam frame for all instances
    Handsfree.use('youtube360', ({head}) => {
      // Exit if the YouTube API is still loading
      if (!YTPlayer || !document.contains(YTPlayer.a)) return

      // Map the cameras POV with the users
      // - Because Handsfree returns radians, we'll also convert them
      // - We then multiply by some number so that you don't have to literally tilt your head 45 degrees 😅
      YTPlayer.getSphericalProperties && YTPlayer.setSphericalProperties({
        pitch: ((-head.rotation[0] * 180) / Math.PI) * 8 + 90,
        yaw: ((-head.rotation[1] * 180) / Math.PI) * 10,
        roll: ((head.rotation[2] * 180) / Math.PI) * 2
      })
    })

Finally, let's add a start button. It's good etiquette to always ask the user to start the webcam!

    <button onclick="handsfree.start()"></button>

And that's all there is to it!

You'll notice that the camera is quite jittery, visit this tutorials Glitch to see how I tween the values to make it more smooth. One day I plan to automatically tween the values for you!

A recap of what we've learned so far

In this tutorial, you learned how to determine the users head pose, which we used to control the POV settings of the playing YouTube video.

handsfree.use('demo', ({head}) => {
  // Returns [pitch, yaw, roll]
  console.log(head.rotation)
})

Combined with what we learned in the last tutorial, you should now be able to determine where on the screen the user is pointed at with the instance.pointer object as well as how the head is pointing at it with instance.head.rotation.

Thanks for reading

We still have a few properties to learn, including determining the users position in 3D space as well as determining their faces morph values (like, how smiley they are or how high/low their eyebrows are). Until then, here are some links to help you take things further:

Getting Started Tutorial
Handsfree.js GitHub
Wiki
Follow me here and on Twitter @HeyOzRamos to follow along in this projects 100DaysofCode!

Thanks for reading and have fun coding 👋

Updates

11/23 - Updated to reflect new v6 API

Handsfree.js - A web based Face Pointer

Oz Ramos — Fri, 08 Nov 2019 04:39:48 +0000

Liquid error: internal

Handsfree.js is a small wrapper library around web based computer vision libraries that I'm working on for the purpose of helping you interact with web pages handsfree. Currently it only handles head tracking through Jeeliz Weboji, but I also plan to add eye tracking, hand tracking, voice, and full body pose estimation over my next 100 days of code.

I also plan on adding integrations for popular libraries like P5.js, Babylon.js, A-Frame, and others!

But in this post, I'll just focus on how to get a simple Handsfree.js project going which involves three steps:

Requiring dependencies
Creating a handsfree instance
Adding a plugin (a callback that runs on every video inference frame)

At the end, you'll have a red face controlled pointer like in the following:

Liquid error: internal

Adding a handsfree pointer

Because everything is already bundled for you, all you need to do is include the Handsfree.js JavaScript and Stylesheet in your DOM:

<!-- Require dependencies -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
<script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>

This adds the Handsfree class to your page, along with basic styles for the pointer. The next step is to create an instance of Handsfree:

const config = {}
const handsfree = new Handsfree(config)

You'll need one instance for each camera that you plan to use (if you have multiple cameras), but each instance can track multiple users. See the documentation on the config object for the defaults and other settings you can use.

And that's it! To start and stop tracking, use handsfree.start() and handsfree.stop().

Adding functionality, and using the pointer values

On each frame, your handsfree instance will have several properties that contain useful information:

// The x coordinate of the pointer on the screen
handsfee.head.pointer.x

// The y coordinate of the pointer on the screen (from 0 at the top)
handsfee.head.pointer.y

// The pointer element
handsfree.head.pointer.$el

// The pointer state ("", "mouseDown", "mouseDrag", "mouseUp")
handsfree.head.pointer.state

// The head position [x, y, scale]
handsfree.head.translation

// The head rotation [pitch, yaw, roll]
handsfree.head.rotation

// Head Morphs (how raised the eyebrows are, how smiley your smiling to each side, etc)
// @see https://github.com/handsfreejs/handsfree/wiki/Head
handsfree.head.morphs

The Handsfree class has a global loop for all instances which you can hook into with the use method:

// Create a simple plugin that displays pointer values on every frame
Handsfree.use('consoleLogger', (instance) => {
  console.log(instance.head.morphs)
})

// Same plugin, but with destructuring
Handsfree.use('consoleLogger', ({head}) => {
  console.log(head.morphs)
})

These are called plugins, where "consoleLogger" is the name of the plugin and instance is the handsfree instance running the plugin (handsfree = new Handsfree()). Adding multiple plugins with the same name overwrites the previous plugin, and to disable a plugin you can call handsfree.stop().

Handsfree.js ships with a few plugins - "head.click" and "head.vertScroll" - which add clicking functionality (with a smile gesture) and scrolling as in this tweet:

Liquid error: internal

A complete example

<!DOCTYPE html>
<head>
  <!-- Require dependencies -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
  <script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>

  <style>
    button {font-size: 24px; padding: 20px}
  </style>
</head>
<body>
  <!-- Let's always ask the user to start -->
  <button onclick="handsfree.start()">Start Webcam</button>
  <button onclick="handsfree.stop()">Stop Webcam</button>

  <script>
    // Create a new instance. Use one instance for each camera
    const handsfree = new Handsfree({})

    // Create a simple plugin that displays pointer values on every frame
    Handsfree.use('consoleLogger', ({head}) => {
      console.log(head.rotation)
    })
  </script>
</body>

Run the above code on Glitch.

Next Steps

I hope this gave you a good introduction into Handsfree.js. In this post I covered how to setup a basic pointer, and in future posts we'll be going over:

How to use head morphs (eyebrows, smiles, kissy faces, and more)
Get the users head pose (yaw, pitch, roll)
Multiplayer support
Complex plugins
Integrations with popular frameworks
and more

Follow me on Twitter at @HeyOzRamos if you'd like to follow along on my 100DaysofCode. Thanks for reading!
Oz

Updates

11/9 - Added references to a few new properties: handsfree.head.rotation, handsfree.head.morphs. New docs at: https://github.com/handsfreejs/handsfree/wiki
11/23 - Updated to reflect new v6 API

Creating interactive CSS 3D device mockups

Oz Ramos — Mon, 05 Aug 2019 22:42:21 +0000

So the other day I came across this awesome webapp called TriDiv, which helps you create CSS 3D objects with drag and drop 🤯 Since I'm working on redesigning my portfolio, it inspired me to create interactive mockups.

Creating the 3D mockups

TriDiv lets you build up models using basic shapes like cubes, cylinders, prisms, and pyramids each with their own configurables like number of sides and rounded corners. Because each side is made up of normal <div>'s you can assign specific styles to each.

To quickly test out my idea I created a simple laptop made out of 3 cubiods, with 2 of them containing rounded corners. I then gave one of them a class name of "screen" and gave the object a pink default background (though you can use your own image if you're just doing a one-off):

Adding hover animations

To add hover animations, I chose to go with the vanilla version Tilt.js (there's also a jQuery version) for the hover effect. I then added a funky effect where the screen pops out at you with standard CSS on the pink .screen object from above.

Going further

On click, replace the image with an iFrame of the actual item
Add a mobile and tablet device

PhoneSaber Starter Kit

Oz Ramos — Mon, 06 May 2019 21:37:22 +0000

Visit this link on a desktop (or use the embed above): https://phonesaber-starter.glitch.me
Click Start Server, noting the server ID
Visit the same link on a device with Gyro (or use the embed above)
Type the server ID
Click Connect to Server
Move your device around to see the 3D lightsaber update on the screen!

Background

So I've been developing a bunch of tools to help people with disabilities use the web hands-free through computer vision. The idea is that people can use head tracking to control a mouse on the screen, and by using JavaScript you can build tools that anyone can use on any device without downloads.

duckhunt-handsfree (smile to shoot)

However, head tracking only works if you're able to point a camera at your face which isn't always possible or practical. So I started thinking about using gyroscopes as a way to move a mouse.

The first idea I had was to use the phone as a sort of "marble maze platform", where tilting the phone moves a cursor on the screen like a marble, like those old school marble maze toys:

But when I had this idea it was May 4th, and since I'm a fan of Star Wars I wanted to experiment with using Gyro's to control a lightsaber on the screen as a fun way to practice using the API.

The API definitely works really well so I'm very eager to start experimenting with it for accessibility purposes. However, I wanted to share my demo which I've released as a starter kit on Glitch.

Instead of finishing my app, I turned it into a starter kit that's ready for you to make exciting games and experiences!

How it Works

Dependencies

Spectre.css, I've been using this for years as my goto prototyping stylesheet
Babylon.js - The 3D engine I used. aFrame.io is another fantastic engine and is WebVR ready (and with an easier learning curve)
Gyronorm.js - This library hasn't been updated in a while, but this is my first try with the gyro API so I went with this to simplify things
Firebase - How I communicate between the client/server
Kyle Katran's Lightsaber by Al on Sketchfab

Key Files

/src/saber.js - This is where you'll probably start your game code, and contains a callback
/src/scene.js - Where I set up the lightsaber, 3D scene
/src/client-server.js - Sets up the client/server, you can mostly ignore this

Ideas

Add sound effects
Deflect lasers
Use PoseNet to so that it moves around too!

API Key

I added a free Firebase API key for you to try with, but I recommend creating your own. It's super quick to do and all you have to do is replace the values at the bottom of index.html with your own (do a document search for "Initialize Firebase")

Thanks!

Let me know if you need any help setting up etc. I don't write many how-to's so I never know how much info to put 😅

DEV Community: Oz Ramos

How I made Flappy Pose, Flappy Bird but where you flap with your arms

Initializing PoseNet through ml5.js

That's all there is to it!

Instagram SparkAR "clone yourself" segmentation effect with Handsfree.js and p5.js

Our tools

Instantiating Handsfree.js

Creating a buffer canvas

Creating a mask

And that's all there is to it!

How to start and stop videos from a distance with your hand ✋

The Setup

Setting up BodyPix through Handsfree.js

That's all there is too it!

Face Painting with P5.js

Setting up our environment

Adding Functionality

That's all there is to it!

How to get a JS.ORG subdomain served through a gh-pages branch

Step 1 - Naming your repo

Step 2: Add a CNAME

Step 3: Commit build directory to gh-pages

Step 4: Make a Pull Request to the js.org repo

Bonus Step - npm script

Gotchas

Thanks for reading!

Puppeteering Emojis with face morphs through Handsfree.js

Setting up

Adding our Emoji plugin

Configuring activation thresholds

Going further

Thanks for reading!

Updates

A look back at week 1 of my #100DaysofCode

Day 1 - 4

Day 5 - 6

Day 7

Next Week

Thanks!

Controlling YouTube 360 videos handsfree with Handsfree.js

Setting up the YouTube IFrame API

Adding Handsfree controls

And that's all there is to it!

A recap of what we've learned so far

Thanks for reading

Updates

Handsfree.js - A web based Face Pointer

Adding a handsfree pointer

Adding functionality, and using the pointer values

A complete example

Next Steps

Updates

Creating interactive CSS 3D device mockups

Creating the 3D mockups

Adding hover animations

Going further

PhoneSaber Starter Kit

Background

How it Works

Dependencies

Key Files

Ideas

API Key

Thanks!

Step 3: Commit build directory to `gh-pages`