Run PoseNet with Nodejs

What is Posenet?

https://medium.com/tensorflow/real-time-human-pose-estimation-in-the-browser-with-tensorflow-js-7dd0bc881cd5
https://github.com/tensorflow/tfjs-models/tree/master/posenet

As you can see, basically we don't need to use nodejs for PoseNet since we need to use canvas to feed data to tensorflowjs lol

Also, we can use tensorflow with python instead of js, but I did try because there are nodejs and tfjs-node(https://github.com/tensorflow/tfjs-node).

steps

1. install packages/libs
2. install npm packages
3. write code & run it

Step1

As I mentioned, need to use canvas, so need to install libs.
By the way, this is for mac, but you can find out information for Linux and Windows on the internet. In addition, now we can use Homebrew on Linux(I tested it on Ubuntu)

$ brew install pkg-config cairo pango libpng jpeg giflib

Step2

I'm using yarn since npm didn't allow me to install tfjs. But, you can use/try npm instead of yarn.

$ yarn add @tensorflow-models/posenet @tensorflow/tfjs @tensorflow/tfjs-node botkit canvas rollup

Step3

The code is messy since just test tfjs-node with posenet.

const tf = require('@tensorflow/tfjs-node');
const posenet = require('@tensorflow-models/posenet');
const {
    createCanvas, Image
} = require('canvas')
const imageScaleFactor = 0.5;
const outputStride = 16;
const flipHorizontal = false;

const tryModel = async() => {
    console.log('start');
    const net = await posenet.load(0.75);
    const img = new Image();
    img.src = './test.jpg';
    const canvas = createCanvas(img.width, img.height);
    const ctx = canvas.getContext('2d');
    ctx.drawImage(img, 0, 0);
    const input = tf.browser.fromPixels(canvas);
    const pose = await net.estimateSinglePose(input, imageScaleFactor, flipHorizontal, outputStride);
    // console.log(pose);
    for(const keypoint of pose.keypoints) {
        console.log(`${keypoint.part}: (${keypoint.position.x},${keypoint.position.y})`);
    }
    console.log('end');
}



tryModel();

Result

Got 17 keypoints, yay!

Comments

[0xkoji]

i just used npm install @tensorflow-models/posenet to use the model.

nodejs v10.15.0
npm v6.9.0

[Geekster🇨🇲]

I am always receiving this error: UnhandledPromiseRejectionWarning: Error: Image given has not completed loading

[0xkoji]

with my sample code?

[Geekster🇨🇲]

Yes, after installing all the dependencies I ended up with this error code : FATAL ERROR: v8::ToLocalChecked Empty MaybeLocal.

[0xkoji]

can you tell me your os version, nodejs version and your package.json's dependencies

[Geekster🇨🇲]

linux subsystem on windows 10, "dependencies": {
"@tensorflow-models/posenet": "^2.1.3",
"body-parser": "^1.19.0",
"express": "^4.17.1",
"morgan": "^1.9.1",
"typescript": "^3.6.3"
},
"devDependencies": {
"@tensorflow/tfjs": "^1.2.9",
"@tensorflow/tfjs-node": "^1.2.9",
"botkit": "^4.5.0",
"canvas": "^2.6.0",
"rollup": "^1.21.2"
}

[0xkoji]

tensorflor-models/posenet has been updated so you need to update code.

just checked my package.json

"dependencies": {
    "@tensorflow-models/posenet": "^1.0.3",
    "@tensorflow/tfjs": "^1.1.2",
    "@tensorflow/tfjs-node": "^1.1.2",
    "botkit": "^4.0.2",
    "canvas": "^2.5.0",
    "rollup": "^1.13.1"
  }

[yalomdd]

I ended up uninstalling everything and reinstalling. But now when I run the code I get " (node:87485) UnhandledPromiseRejectionWarning: TypeError: Cannot create property 'architecture' on number '0.75'" which seems to be an error within posenet. maybe not everything is completely installed correctly? what do you think thank you for the help

[Dan Oved]

It's because posenet api has been updated in version 2.0. See:
github.com/tensorflow/tfjs-models/...

Example new loading code:

const net = await posenet.load({
  architecture: 'MobileNetV1',
  outputStride: 16,
  inputResolution: 513,
  multiplier: 0.75
});

[Venkataramanan Chockalingam]

Can you please add video inference support for posenet?

[0xkoji]

sorry i don't have time now.
i recommend you to check tfjs repo and dan's repos dev.to/oveddan since he is the peron who has implemented posenet to tfjs.
my code is outdated.

[0xkoji]

@gaelguedia
can you tell me your os version, nodejs version, npm/yarn and your package.json's dependencies

[Geekster🇨🇲]

The above error was on windows 10, js 10. The issue seems to be related to canvas dependency. It's not easy to have it running on windows. It's easier on ubuntu especially easier to install cairo dependencies for Jpg/jpeg support