How to start and stop videos from a distance with your hand ✋

#tutorial #computervision #javascript #handsfreejs

In this tutorial you'll learn how to use BodyPix, through Handsfree.js, to start and stop a YouTube video by raising up your hand ✋

The Setup

Let's start by adding our dependencies:

<head>
  <!-- Handsfree dependencies -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@6.1.3/dist/handsfree.css" />
  <script src="https://unpkg.com/handsfree@6.1.3/dist/handsfree.js"></script>

  <!-- YouTube dependencies -->
  <script defer src="https://www.youtube.com/iframe_api"></script>
</head>
<body>
  <div id="player"></div>
</body>

We'll then use the YouTube iframe API to instantiate a video:

/**
  * This method is automatically called by the YouTube API
  * @see https://developers.google.com/youtube/iframe_api_reference
  */
let player
function onYouTubeIframeAPIReady () {
  player = new YT.Player('player', {
    // I picked this video because it's ad free
    videoId: 'AFKktZURpJ8'
  })
}

Setting up BodyPix through Handsfree.js

The next step is to instantiate Handsfree with the bodypix model enabled, and then create a plugin to watch for when the hand is raised. Handsfree.js plugins are simply callbacks that run after a model does inference.

// Create one instance for each camera you plan to use. By default it uses the main webcam (the selfie camera on mobile devices)
const handsfree = new Handsfree({
  models: {
    // The head model is enabled by default, so lets disable it
    head: {enabled: false},
    bodypix: {enabled: true}
  }
})


/**
  * Create a plugin that stops the video when the hand is up
  */
Handsfree.use('hand.stopVideo', {
  // Whether we have entered "hands have been up" state
  isBeingHeld: false,

  /**
   * This method is called once per webcam frame by Handsfree.js
   * @param {Object} The handsfree instance. Here we destructure it to just get instance.body
   */
  onFrame ({body}) {
    // only continue if one of the wrists are visible
    if (!body.pose || !body.pose.keypoints
      || (body.leftWrist.score < 0.6 && body.rightWrist.score < 0.6)) return

    // Check if one wrist is level with that shoulder
    if (body.leftWrist.y <= body.leftShoulder.y || body.rightWrist.y <= body.rightShoulder.y) {
      if (!this.isBeingHeld) {
        this.toggleVideo()
        this.isBeingHeld = true
      }
    } else {
      this.isBeingHeld = false
    }
  },

  /**
    * Toggle the video
    */
  toggleVideo () {
    const state = player.getPlayerState()

    if (state === 1) {
      player.pauseVideo()
    } else {
      player.playVideo()
    }
  }
})

That's all there is too it!

This tutorial quickly covered how to instantiate a YouTube video using the YouTube Iframe API. We then learned how to instantiate Handsfree with the bodypix model, and check for when the hand is raised...but there's so much more you can do!

Here are some other links you might find useful:

The bodypix helpers that Handsfree.js provides
Handsfree.js GitHub
My Twitter @HeyOzRamos
The Demo
Demo Source

Thanks for reading and happy coding 👋

Oz Ramos

@heyozramos

What if you could start and stop videos by raising your hand ✋

In this tutorial I demonstrate how to use #TensorFlow BodyPix with Handsfree.js to start and stop a YouTube video!

- Tutorial: dev.to/heyozramos/how…
- @Glitch Demo: …free-toggle-video-with-hand.glitch.me

04:42 AM - 21 Dec 2019

0 0