Face Painting with P5.js

Liquid error: internal

In this tutorial you'll learn everything you need to start painting happy little trees with your face 🌳 This technique uses a "face pointer", which lets you control a pointer with your head and face gestures for clicking and more!

We'll be using the new Handsfree.js API to quickly setup our face pointer and P5.js to do the painting. Behind the scenes, Handsfree.js is powered by the Weboji head tracker.

So let's begin!

Setting up our environment

So the first thing we'll want to do is handle dependencies:

<!-- Handsfree.js -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.3/dist/handsfreejs/handsfree.css" />
<script src="https://unpkg.com/handsfree@6.0.3/dist/handsfreejs/handsfree.js"></script>

<!-- Demo dependencies: P5 and lodash -->
<script src="https://cdn.jsdelivr.net/npm/p5@0.10.2/lib/p5.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/lodash.js/4.17.15/lodash.min.js"></script>

<!-- Our P5 Sketch will go in here -->
<div id="canvas-wrap"></div>

This gives us a global Handsfree class object. The next thing that we do is create an instance of Handsfree. We need one instance for every webcam that we plan to use, but each instance can track multiple users (see config options):

const config = {};
handsfree = new Handsfree(config);

If at this point we were to run handsfree.start() then we would see a red face controlled cursor, along with the debug video feed.

Adding Functionality

To add functionality you add callbacks (I call them plugins) to the Handsfree class object with Handsfree.use("pluginName", opts).

Here pluginName can be anything and is there so that we can disable/enable plugins by name with Handsfree.disable('pluginName') or access them under the hood with Handsfree.plugins['pluginName'].

opts can either be a callback function to run on every webcam frame, or it can be an object with the following core properties and methods:

Handsfree.use("pluginName", {
  // Whether to start using this plugin immediately or not
  enabled: true,

  // Called once when the plugin is first used
  // - Use this to initialize stuff
  onUse({ head }) {},

  // Called on every frame loop
  // - Use this as your "game loop"
  // - This is the same as only passing a callback
  onFrame({ head }) {}
});

These callbacks pass in the handsfree instance, which we usually destructure to get the handsfree.head object...these two are equivalent:

Handsfree.use("test1", instance => {
  console.log(instance.head.rotation);
});

Handsfree.use("test2", ({ head }) => {
  console.log(head.rotation);
});

Knowing all that, let's define our "P5.facePaint" plugin:

• Setup P5.js and remember get a reference to our canvas
• Capture face gestures on every frame
• Paint and/or change colors

Handsfree.use("p5.facePaint", {
  // Current pointer position
  x: 0,
  y: 0,
  // Last frames pointer position
  lastX: 0,
  lastY: 0,

  // Contains our P5 instance
  p5: null,

  /**
   * Called exactly once when the plugin is first used
   */
  onUse() {
    // Store a reference of our p5 sketch
    this.p5 = new p5(p => {
      const $canvasWrap = document.querySelector("#canvas-wrap");

      // Setup P5 canvas
      p.setup = () => {
        // Create a P5 canvas with the dimensions of our container
        const $canvas = p.createCanvas(
          $canvasWrap.clientWidth,
          $canvasWrap.clientHeight
        );
        $canvas.parent($canvasWrap);
        p.strokeWeight(6);
      };

      // Match canvas size to window
      p.windowResized = () => {
        p.resizeCanvas($canvasWrap.clientWidth, $canvasWrap.clientHeight);
      };
    });
  },

  /**
   * Called on every webcam frame
   */
  onFrame({ head }) {
    // Setup point coordinates
    this.lastX = this.x;
    this.lastY = this.y;
    // @todo: pointer origin should be at center, not corner (fix via CSS?)
    this.x = head.pointer.x + 10;
    this.y = head.pointer.y + 10;

    this.p5.stroke(this.p5.color(strokeColor));

    // Draw lines
    if (head.state.smirk || head.state.smile) {
      this.p5.line(this.x, this.y, this.lastX, this.lastY);
    }

    // Change colors with eyebrows
    if (head.state.browLeftUp) this.updateColor(1);
    else if (head.state.browRightUp) this.updateColor(-1);
  },

  /**
   * Throttled to 4 times a second
   * - Please see source code for this tutorial for the rest of this method
   * @see https://glitch.com/edit/#!/handsfree-face-painting?path=app.js
   */
  updateColor: _.throttle(function(step) {}, 250, { trailing: false })
});

That's all there is to it!

This tutorial quickly went over how to setup a P5 Sketch to work with Handsfree.js. It's still not perfect and I plan to create an official P5.js - Handsfree.js integration soon, but it should be enough to get you going!

Make sure to check out the source for the rest of the code as I omitted some non-Handsfree.js stuff for brevity.

• Source
• Demo
• Handsfree.js
• Twitter @HeyOzRamos

Thanks and have fun 👋