DEV Community: Shamima Hossain

Exploring TensorflowJS: A Hangman Game made with pretrained Handpose model

Shamima Hossain — Tue, 09 Nov 2021 18:18:07 +0000

Tensorflow has been a powerful tool for Machine learning tasks for quite a while. However did you know that traditional ML tasks can also be performed on the browser as well? The learning curve for Tensorflow python library is notoriously steep for beginners in ML let alone for people trying ML from other backgrounds. In this article I will show you the wonderful world of Machine Learning in the browser and how easy it is to get started with your already known tools for the web particularly Javascript.

In this article we will:

quickly go over what ML is
How the pretrained HandPose model works to recognise gestures
create a Hangman game played using Sign Language

Have a look at the end project here

A quick primer on Machine Learning (skip if you like)

Machine Learning is basically the opposite of what you have been doing in software development. Instead of giving the computer a defined set of steps, you would give the computer lots and lots of data and the machine would figure out the pattern from that data and solve a particular task kind of how our brain works in real life. Now, there are famously two ways you can approach an ML task:

Supervised Learning - you would give the machine a bunch of data and also the outcomes of that particular set of observations and train it. Then when you would present an unknown observation the machine would predict an outcome based on that training data. This is the high level idea and you can go a lot deeper into the rabbit hole which I encourage you to do so.
Unsupervised Learning - In this type of training you would only show the observations and would not show the outcomes or labels. The machine would itself figure out the patterns in the dataset and usually group together/cluster data points with similar properties. When you would present a new and unknown data, the trained model would then group it in a cluster with similar properties and will output the cluster it may belong to. This kind of ML task is a bit harder to grasp. Usually you would be using supervised learning for most of the times.

In this article, we would do neither though ! Just kidding, we could technically take a dataset, train it and analyse the model's behaviour on new data points. But often you don't need to reinvent the wheel. As an engineer a useful skill to master is learning how to adapt already existing solutions for your particular problem. By an already existing solution I mean that the collecting, training and evaluating the model is already done and you can get started with the actual fun part, building something cool with your new ML superpowers!

The ASL Hangman Game

We could of course play a hang man with traditional input/output from the keyboard but where's the fun in that. Let's learn how to build a hangman game and recognize letter inputs from our hand gestures. Now please don't fret as it isn't as hard of a task as it sounds. We will be using the pre-trained Handpose model made by the TensorflowJS. You can have a look at it here https://github.com/tensorflow/tfjs-models/tree/master/handpose The great thing is that in order to run the models all you have to do is insert some script tags and voila the desired model would be loaded and ready to run for you. So we will first inspect the Handpose model and explore its features. Go ahead and feel free to run the barebone demo on Codepen

   [{…}]
   0:
   1. annotations: {thumb: Array(4), indexFinger: Array(4), middleFinger: Array(4), ringFinger: Array(4), pinky: Array(4), …}
         2. boundingBox: {topLeft: Array(2), bottomRight: Array(2)}
         3. handInViewConfidence: 0.9999997615814209
         4. landmarks: (21) [Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3), Array(3)]
         5. [[Prototype]]: Object

      3. length: 1

      4. [[Prototype]]: Array(0)

If you inspect the array of objects called predictions in the browser, you would find a number of useful elements and it also contains a probability with which a hand is detected. For our purpose the key annotations is of particular interest. Lets take a closer look at it

   [{…}]
   0:
   annotations:
   indexFinger: Array(4)
   0: (3) [389.5245886969721, 244.7159004390616, -0.30365633964538574]
   1: (3) [381.65693557959514, 181.97510097266763, -3.5919628143310547]
   2: (3) [374.36188515696244, 132.26145430768776, -8.026983261108398]
   3: Array(3)
   length: 4
   [[Prototype]]: Array(0)
   middleFinger: (4) [Array(3), Array(3), Array(3), Array(3)]
   palmBase: [Array(3)]
   pinky: (4) [Array(3), Array(3), Array(3), Array(3)]
   ringFinger: (4) [Array(3), Array(3), Array(3), Array(3)]
   thumb: (4) [Array(3), Array(3), Array(3), Array(3)]
   [[Prototype]]: Object

you would see that it contains five elements each corresponding to the five fingers and upon expanding the keys of each finger we see that there are four nested arrays of x, y and z co-ordinates, corresponding to the four separations of the finger. Now with a little more experimentation we would further discover that the 0th element is the base of the finger and the 3rd element is the tip of the finger and the middle arrays are for the two separations in each of your fingers.

Detecting gestures

We will use simple mathematics in order to recognize certain gestures of the sign language. Take a look at the ASL chart for reference

We will only concentrate on the vowel words for simplicity. So let's start with the A . You can see that in this gesture only the thumb is upright and the rest of the four fingers are folded. So if we want to define this gesture, the co-ordinates of the tip of the thumb will be lower than that of the other four fingers. We are saying lower because the co-ordinate system in the browser canvas is upside down opposite to what we are used to seeing in maths. Hmm..all good now how do we get the co-ordinates of those fingers? Ahh..Handpose model to the rescue. Remember how the model returns co-ordinates of each of landmarks of each five fingers, so let's use those.

To recognize the A gesture we could do the calculations like so

 const thumbTip = predictions[0].annotations.thumb[3][1];
 const indexTip = predictions[0].annotations.indexFinger[3][1];
 const middleFTip = predictions[0].annotations.middleFinger[3][1];
 const ringFTip = predictions[0].annotations.ringFinger[3][1];
 const pinkyTip = predictions[0].annotations.pinky[3][1];
// We need to check if the tip of the thumb is higher than the other for fingers so we could write the condition to check for the other four fingers like this
 const otherFourFingersFolded =
      indexTip > indexBase1 &&
      middleFTip > middleFBase1 &&
      ringFTip > ringFBase1 &&
      pinkyTip > pinkyBase1;
//the entire condition to check for A gesture goes like so
if(thumbTip < (indexTip && middleFTip && ringFTip && pinkyTip) &&
      otherFourFingersFolded
    ) {
      console.log("a");
    }

Similarly, we will combine simple maths with if-else statements to check for other gestures and the whole thing looks like this.

const thumbTip = predictions[0].annotations.thumb[3][1];
const indexTip = predictions[0].annotations.indexFinger[3][1];
const middleFTip = predictions[0].annotations.middleFinger[3][1];
const ringFTip = predictions[0].annotations.ringFinger[3][1];
const pinkyTip = predictions[0].annotations.pinky[3][1];
const indexBase1 = predictions[0].annotations.indexFinger[1][1];
const middleFBase1 = predictions[0].annotations.middleFinger[1][1];
const ringFBase1 = predictions[0].annotations.ringFinger[1][1];
const pinkyBase1 = predictions[0].annotations.pinky[1][1];
const diffThumbIndex = thumbTip - indexTip;
const diffIndexMiddle = indexTip - middleFTip;
const otherFourFingersFolded =
      indexTip > indexBase1 &&
      middleFTip > middleFBase1 &&
      ringFTip > ringFBase1 &&
      pinkyTip > pinkyBase1;
if (diffThumbIndex >= 20 && diffIndexMiddle <= 0) {
    console.log("o");
} else if (pinkyTip < (middleFTip && ringFTip && indexTip)) {
    console.log("i");
} else if (
    thumbTip < (indexTip && middleFTip && ringFTip && pinkyTip) &&
    otherFourFingersFolde
) {
    console.log("a");
} else if (
    thumbTip > (indexTip && middleFTip && ringFTip && pinkyTip) &&
    !(diffThumbIndex >= 20 && diffIndexMiddle <= 0)
) {
    console.log("e");
} else if (diffThumbIndex > 100 && diffIndexMiddle <= 20) {
    console.log("u");
}

Now for the fun part, let's incorporate this new superpower inside a hangman game and give the traditional game a nice twist.

I have created a barebones hangman script for you, feel free to customise and add your personal touch to it but the basic concept is that you would fill the missing letters by showing the gesture to your webcam and the model would decode that gesture to a possible letter. The code structure is very simple and without any external UI/JS framework dependencies. I like separating similar functionalities in separate files since we have two different functionalities U have separated hangman.js and the functionalities containing that of the handpose model into index.js . The output of these will be displayed in the index.html file. The entire project is made using vanilla JS template from Codesandbox with Parcel as the bundler.

It is worth spending sometime looking at how we setup the ML model to run in index.js

import * as tf from "@tensorflow/tfjs";
import * as handpose from "@tensorflow-models/handpose";
let video;
let model;
const init = async () => {
  video = await loadVideo();
  await tf.setBackend("webgl");
  model = await handpose.load();
  main();
};
const loadVideo = async () => {
  const video = await setupCamera();
  video.play();
  return video;
};
const setupCamera = async () => {
  if (!navigator.mediaDevices || !navigator.mediaDevices.getUserMedia) {
    throw new Error(
      "Browser API navigator.mediaDevices.getUserMedia not available"
    );
  }
  video = document.querySelector("video");
  video.width = window.innerWidth;
  video.height = window.innerHeight;
  const stream = await navigator.mediaDevices.getUserMedia({
    audio: false,
    video: {
      facingMode: "user",
      width: window.innerWidth,
      height: window.innerHeight
    }
  });
  video.srcObject = stream;
  return new Promise(
    (resolve) => (video.onloadedmetadata = () => resolve(video))
  );
};
init();
async function main() {
  const predictions = await model.estimateHands(
    document.querySelector("video")
  );
  if (predictions.length > 0) {
    const thumbTip = predictions[0].annotations.thumb[3][1];
    const indexTip = predictions[0].annotations.indexFinger[3][1];
    const middleFTip = predictions[0].annotations.middleFinger[3][1];
    const ringFTip = predictions[0].annotations.ringFinger[3][1];
    const pinkyTip = predictions[0].annotations.pinky[3][1];
    const indexBase1 = predictions[0].annotations.indexFinger[1][1];
    const middleFBase1 = predictions[0].annotations.middleFinger[1][1];
    const ringFBase1 = predictions[0].annotations.ringFinger[1][1];
    const pinkyBase1 = predictions[0].annotations.pinky[1][1];
    const diffThumbIndex = thumbTip - indexTip;
    const diffIndexMiddle = indexTip - middleFTip;
    const otherFourFingersFolded =
      indexTip > indexBase1 &&
      middleFTip > middleFBase1 &&
      ringFTip > ringFBase1 &&
      pinkyTip > pinkyBase1;
    const inputLetter = document.getElementById("letter");

    if (diffThumbIndex >= 20 && diffIndexMiddle <= 0) {
      inputLetter.value = "o";
    } else if (pinkyTip < (middleFTip && ringFTip && indexTip)) {
      inputLetter.value = "i";
    } else if (
      thumbTip < (indexTip && middleFTip && ringFTip && pinkyTip) &&
      otherFourFingersFolded
    ) {
      inputLetter.value = "a";
    } else if (
      thumbTip > (indexTip && middleFTip && ringFTip && pinkyTip) &&
      !(diffThumbIndex >= 20 && diffIndexMiddle <= 0)
    ) {
      inputLetter.value = "e";
    } else if (diffThumbIndex > 100 && diffIndexMiddle <= 20) {
      inputLetter.value = "u";
    }
  }
  requestAnimationFrame(main);
}

After importing the necessary libraries, the script waits for the video object in the index.html file to get input data upon initializing by the init method. The model would then run on the data from the webcam feed and save it in a placeholder called predictions . Once you get your predictions you would put in your logic as we did with the finger co-ordinates. The hangman game would get the inputs from this part of your project and play the game accordingly. you can view the full working project here

That's it folks. In this article you learnt the basic concepts of Machine Learning and played along with how you can implement fun stuff in the browser with already existing models.

Column-like layout in CSS without Bootstrap or Grids (Part-2)

Shamima Hossain — Wed, 01 Jan 2020 16:15:38 +0000

Another great way to to design our text into columns is going to back to the old ways of building websites. If you remember correctly, primitive websites were all just tables tweaked in different ways.

Well that's great because we can use just that to quickly place our texts in different columns . As always I would like to illustrate it with an example.

Let's first create an html skeleton of our texts

<section id="exhibit">

 <div class="col">
 Lorem ipsum dolor sit amet, consectetur adipiscing elit.
 Vivamus porta massa eget pellentesque aliquam. Ut malesuada 
 sapien aliquam, 
 pharetra metus quis, finibus dui. 
 Curabitur fermentum ac leo non consectetur. 
 </div>


 <div class="col">
 Lorem ipsum dolor sit amet, consectetur adipiscing elit.
 Vivamus porta massa eget pellentesque aliquam. Ut malesuada 
 sapien aliquam, 
 pharetra metus quis, finibus dui. 
 Curabitur fermentum ac leo non consectetur. 
 </div>


 <div class="col">
 Lorem ipsum dolor sit amet, consectetur adipiscing elit.
 Vivamus porta massa eget pellentesque aliquam. Ut malesuada 
 sapien aliquam, 
 pharetra metus quis, finibus dui. 
 Curabitur fermentum ac leo non consectetur. 
 </div>

Then the CSS styling would be like this

#exhibit{
    height: 500px;
    display: table;
    width: 100%;
   }
.col{
    padding-left: 2em;
    padding-right: 2em;
    font-size: 1.2em;
    font-family: sans-serif;
    overflow: hidden;
    display: table-cell;
    background: grey;
   }

So we would like to lay these 3 divs as three separate columns. If you read the code, you would see the divs are wrapped in a section.

We set this parent/ wrapper to display: table . Then if we set its children to display: table-cell we would see them lined up in 3 columns.

The final output would look something like this

You can play around more with it if you like. Do let me know what you think of this in the comments below.
Thanks for reading.

Column-like layout in CSS without Bootstrap or Grids (Part-1)

Shamima Hossain — Fri, 16 Aug 2019 18:52:56 +0000

Ever wondered how to create columns in CSS without using grids or those pesky col-md-sth in a labyrinth of div tags ?

There's mainly two ways you could do so. I shall explain with examples and scenarios as this is how I understand the best.

Scenario 1: say you would like to write your article in newspaper style. As you know, big chunks of continuous texts in newspapers are written in columns instead of paragraphs. How could you possibly achieve such a look with minimal effort? Turns out our old but mighty CSS has a property to do so.The property

column-count:

let's take an example
I want to write a piece of text entirely in newspaper-ish style. The body structured with html would go like this

<section>

<h3>Things that don't make sense</h3>

<div class="col">


Lorem ipsum dolor sit amet, consectetur adipiscing elit, 
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. 
Sollicitudin tempor id eu nisl nunc mi ipsum faucibus vitae. 
Integer vitae justo eget magna fermentum iaculis eu non.
Rhoncus urna neque viverra justo nec ultrices dui. Risus viverra
adipiscing at in tellus integer feugiat scelerisque.
In hendrerit gravida
rutrum quisque non tellus. Consequat semper viverra nam
libero  justo laoreet sit amet.


</div>

</section>

And now with one line of CSS we will break this chunk of text in columns. This would be

.col{
  column-count: 3;
}

And the final output would be

The best thing is if you click on the edit on Codepen and try resizing the window size, you'll see it adjusts according to the browser size. Woohoo!

But wait there's more, you can also customize your columns to have gaps, column-rules and what not. I'll leave the link to the w3schools page for you to play around with it more

https://www.w3schools.com/css/css3_multiple_columns.asp

While I'm aware that this might not be as elegant as the generic Grids, this took my attention for its sheer simplicity.
Often you might only need a simple column-ish layout and using Grids are somewhat of an overkill for those times.

Let me know what you guys like and dislike about this CSS property in comments below. Also,stay tuned for part-2.

What is an API for complete idiots like me?

Shamima Hossain — Fri, 19 Jul 2019 04:18:08 +0000

Why it is important and how to get started working with them? Please provide non technical analogies as I already have tried but don't understand them.

My first interview experience

Shamima Hossain — Fri, 12 Jul 2019 15:35:45 +0000

After almost like months I finally built up courage to apply even though I am too far from being industry fit. I saw a post about a startup who were the only one willing to hire people with almost basic problem solving skills and know about basic coding.Here were my key mistakes:

I had unrealistically high expectations from myself
I got nervous and forgot my basics e.g could not tell the difference between HTML tags and attributes
I proposed an unrealistic salary expectation for an intern
I came off as not confident enough and unprepared even though I literally had no sleep the night before preparing

Lessons learned:

No matter how much you know practice controlling your nerves
Research as much as you can about industry before going for the interview
Have realistic expectations regarding salary
Learn to tackle questions about your lacking and find a way to convince them that you can overcome them.

The interviewers were very helpful and understanding but I could see the disappointment in their body language near the end. I guess this opportunity wasn't meant for me in the end but I am glad had the experience and hope to land my dream job soon.