DEV Community

Cover image for Machine Learning In Node.js With TensorFlow.js
IBM Developer profile image James Thomas
James Thomas for IBM Developer

Posted on • Edited on • Originally published at jamesthom.as

Machine Learning In Node.js With TensorFlow.js

#node #machinelearning #tensorflow #javascript

TensorFlow.js is a new version of the popular open-source library which brings deep learning to JavaScript. Developers can now define, train, and run machine learning models using the high-level library API.

Pre-trained models mean developers can now easily perform complex tasks like visual recognition, generating music or detecting human poses with just a few lines of JavaScript.

Having started as a front-end library for web browsers, recent updates added experimental support for Node.js. This allows TensorFlow.js to be used in backend JavaScript applications without having to use Python.

Reading about the library, I wanted to test it out with a simple task... 🧐

Use TensorFlow.js to perform visual recognition on images using JavaScript from Node.js

Unfortunately, most of the documentation and example code provided uses the library in a browser. Project utilities provided to simplify loading and using pre-trained models have not yet been extended with Node.js support. Getting this working did end up with me spending a lot of time reading the Typescript source files for the library. 👎

However, after a few days' hacking, I managed to get this completed! Hurrah! 🤩

Before we dive into the code, let's start with an overview of the different TensorFlow libraries.

TensorFlow

TensorFlow is an open-source software library for machine learning applications. TensorFlow can be used to implement neural networks and other deep learning algorithms.

Released by Google in November 2015, TensorFlow was originally a Python library. It used either CPU or GPU-based computation for training and evaluating machine learning models. The library was initially designed to run on high-performance servers with expensive GPUs.

Recent updates have extended the software to run in resource-constrained environments like mobile devices and web browsers.

TensorFlow Lite

Tensorflow Lite, a lightweight version of the library for mobile and embedded devices, was released in May 2017. This was accompanied by a new series of pre-trained deep learning models for vision recognition tasks, called MobileNet. MobileNet models were designed to work efficiently in resource-constrained environments like mobile devices.

TensorFlow.js

Following Tensorflow Lite, TensorFlow.js was announced in March 2018. This version of the library was designed to run in the browser, building on an earlier project called deeplearn.js. WebGL provides GPU access to the library. Developers use a JavaScript API to train, load and run models.

TensorFlow.js was recently extended to run on Node.js, using an extension library called tfjs-node.

The Node.js extension is an alpha release and still under active development.

Importing Existing Models Into TensorFlow.js

Existing TensorFlow and Keras models can be executed using the TensorFlow.js library. Models need converting to a new format using this tool before execution. Pre-trained and converted models for image classification, pose detection and k-nearest neighbours are available on Github.

Using TensorFlow.js in Node.js

Installing TensorFlow Libraries

TensorFlow.js can be installed from the NPM registry. 

npm install @tensorflow/tfjs @tensorflow/tfjs-node
// or...
npm install @tensorflow/tfjs @tensorflow/tfjs-node-gpu
Enter fullscreen mode Exit fullscreen mode

Both Node.js extensions use native dependencies which will be compiled on demand.

Loading TensorFlow Libraries

TensorFlow's JavaScript API is exposed from the core library. Extension modules to enable Node.js support do not expose additional APIs.

const tf = require('@tensorflow/tfjs')
// Load the binding (CPU computation)
require('@tensorflow/tfjs-node')
// Or load the binding (GPU computation)
require('@tensorflow/tfjs-node-gpu')
Enter fullscreen mode Exit fullscreen mode

Loading TensorFlow Models

TensorFlow.js provides an NPM library (tfjs-models) to ease loading pre-trained & converted models for image classification, pose detection and k-nearest neighbours.

The MobileNet model used for image classification is a deep neural network trained to identify 1000 different classes.

In the project's README, the following example code is used to load the model.

import * as mobilenet from '@tensorflow-models/mobilenet';

// Load the model.
const model = await mobilenet.load();
Enter fullscreen mode Exit fullscreen mode

One of the first challenges I encountered was that this does not work on Node.js.

Error: browserHTTPRequest is not supported outside the web browser.
Enter fullscreen mode Exit fullscreen mode

Looking at the source code, the mobilenet library is a wrapper around the underlying tf.Model class. When the load() method is called, it automatically downloads the correct model files from an external HTTP address and instantiates the TensorFlow model.

The Node.js extension does not yet support HTTP requests to dynamically retrieve models. Instead, models must be manually loaded from the filesystem.

After reading the source code for the library, I managed to create a work-around...

Loading Models From a Filesystem

Rather than calling the module's load method, if the MobileNet class is created manually, the auto-generated path variable which contains the HTTP address of the model can be overwritten with a local filesystem path. Having done this, calling the load method on the class instance will trigger the filesystem loader class, rather than trying to use the browser-based HTTP loader.

const path = "mobilenet/model.json"
const mn = new mobilenet.MobileNet(1, 1);
mn.path = `file://${path}`
await mn.load()
Enter fullscreen mode Exit fullscreen mode

Awesome, it works!

But how where do the models files come from?

MobileNet Models

Models for TensorFlow.js consist of two file types, a model configuration file stored in JSON and model weights in a binary format. Model weights are often sharded into multiple files for better caching by browsers.

Looking at the automatic loading code for MobileNet models, models configuration and weight shards are retrieved from a public storage bucket at this address.

https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v${version}_${alpha}_${size}/
Enter fullscreen mode Exit fullscreen mode

The template parameters in the URL refer to the model versions listed here. Classification accuracy results for each version are also shown on that page.

According to the source code, only MobileNet v1 models can be loaded using the tensorflow-models/mobilenet library.

The HTTP retrieval code loads the model.json file from this location and then recursively fetches all referenced model weights shards. These files are in the format groupX-shard1of1.

Downloading Models Manually

Saving all model files to a filesystem can be achieved by retrieving the model configuration file, parsing out the referenced weight files and downloading each weight file manually.

I want to use the MobileNet V1 Module with 1.0 alpha value and image size of 224 pixels. This gives me the following URL for the model configuration file.

https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/model.json
Enter fullscreen mode Exit fullscreen mode

Once this file has been downloaded locally, I can use the jq tool to parse all the weight file names.

$ cat model.json | jq -r ".weightsManifest[].paths[0]"
group1-shard1of1
group2-shard1of1
group3-shard1of1
...
Enter fullscreen mode Exit fullscreen mode

Using the sed tool, I can prefix these names with the HTTP URL to generate URLs for each weight file.

$ cat model.json | jq -r ".weightsManifest[].paths[0]" | sed 's/^/https:\/\/storage.googleapis.com\/tfjs-models\/tfjs\/mobilenet_v1_1.0_224\//'
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group1-shard1of1
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group2-shard1of1
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group3-shard1of1
...
Enter fullscreen mode Exit fullscreen mode

Using the parallel and curl commands, I can then download all of these files to my local directory.

cat model.json | jq -r ".weightsManifest[].paths[0]" | sed 's/^/https:\/\/storage.googleapis.com\/tfjs-models\/tfjs\/mobilenet_v1_1.0_224\//' |  parallel curl -O
Enter fullscreen mode Exit fullscreen mode

Classifying Images

This example code is provided by TensorFlow.js to demonstrate returning classifications for an image.

const img = document.getElementById('img');

// Classify the image.
const predictions = await model.classify(img);
Enter fullscreen mode Exit fullscreen mode

This does not work on Node.js due to the lack of a DOM.

The classify method accepts numerous DOM elements (canvas, video, image) and will automatically retrieve and convert image bytes from these elements into a tf.Tensor3D class which is used as the input to the model. Alternatively, the tf.Tensor3D input can be passed directly.

Rather than trying to use an external package to simulate a DOM element in Node.js, I found it easier to construct the tf.Tensor3D manually.

Generating Tensor3D from an Image

Reading the source code for the method used to turn DOM elements into Tensor3D classes, the following input parameters are used to generate the Tensor3D class.

const values = new Int32Array(image.height * image.width * numChannels);
// fill pixels with pixel channel bytes from image
const outShape = [image.height, image.width, numChannels];
const input = tf.tensor3d(values, outShape, 'int32');
Enter fullscreen mode Exit fullscreen mode

pixels is a 2D array of type (Int32Array) which contains a sequential list of channel values for each pixel. numChannels is the number of channel values per pixel.

Creating Input Values For JPEGs

The jpeg-js library is a pure javascript JPEG encoder and decoder for Node.js. Using this library the RGB values for each pixel can be extracted.

const pixels = jpeg.decode(buffer, true);
Enter fullscreen mode Exit fullscreen mode

This will return a Uint8Array with four channel values (RGBA) for each pixel (width * height). The MobileNet model only uses the three colour channels (RGB) for classification, ignoring the alpha channel. This code converts the four channel array into the correct three channel version.

const numChannels = 3;
const numPixels = image.width * image.height;
const values = new Int32Array(numPixels * numChannels);

for (let i = 0; i < numPixels; i++) {
  for (let channel = 0; channel < numChannels; ++channel) {
    values[i * numChannels + channel] = pixels[i * 4 + channel];
  }
}
Enter fullscreen mode Exit fullscreen mode

MobileNet Models Input Requirements

The MobileNet model being used classifies images of width and height 224 pixels. Input tensors must contain float values, between -1 and 1, for each of the three channels pixel values.

Input values for images of different dimensions needs to be re-sized before classification. Additionally, pixels values from the JPEG decoder are in the range 0 - 255, rather than -1 to 1. These values also need converting prior to classification.

TensorFlow.js has library methods to make this process easier but, fortunately for us, the tfjs-models/mobilenet library automatically handles this issue! 👍

Developers can pass in Tensor3D inputs of type int32 and different dimensions to the classify method and it converts the input to the correct format prior to classification. Which means there's nothing to do... Super 🕺🕺🕺.

Obtaining Predictions

MobileNet models in Tensorflow are trained to recognise entities from the top 1000 classes in the ImageNet dataset. The models output the probabilities that each of those entities is in the image being classified.

The full list of trained classes for the model being used can be found in this file.

The tfjs-models/mobilenet library exposes a classify method on the MobileNet class to return the top X classes with highest probabilities from an image input.

const predictions = await mn_model.classify(input, 10);
Enter fullscreen mode Exit fullscreen mode

predictions is an array of X classes and probabilities in the following format.

{
  className: 'panda',
  probability: 0.9993536472320557
}
Enter fullscreen mode Exit fullscreen mode

Example

Having worked how to use the TensorFlow.js library and MobileNet models on Node.js, this script will classify an image given as a command-line argument.

source code

  • Save this script file and package descriptor to local files.

testing it out

  • Download the model files to a mobilenet directory using the instructions above.
  • Install the project dependencies using NPM 
npm install
Enter fullscreen mode Exit fullscreen mode
  • Download a sample JPEG file to classify 
wget http://bit.ly/2JYSal9 -O panda.jpg
Enter fullscreen mode Exit fullscreen mode

Panda

  • Run the script with the model file and input image as arguments. 
node script.js mobilenet/model.json panda.jpg
Enter fullscreen mode Exit fullscreen mode

If everything worked, the following output should be printed to the console.

classification results: [ {
    className: 'giant panda, panda, panda bear, coon bear',
    probability: 0.9993536472320557 
} ]
Enter fullscreen mode Exit fullscreen mode

The image is correctly classified as containing a Panda with 99.93% probability! 🐼🐼🐼

Conclusion

TensorFlow.js brings the power of deep learning to JavaScript developers. Using pre-trained models with the TensorFlow.js library makes it simple to extend JavaScript applications with complex machine learning tasks with minimal effort and code.

Having been released as a browser-based library, TensorFlow.js has now been extended to work on Node.js, although not all of the tools and utilities support the new runtime. With a few days' hacking, I was able to use the library with the MobileNet models for visual recognition on images from a local file.

Getting this working in the Node.js runtime means I now move on to my next idea... making this run inside a serverless function! Come back soon to read about my next adventure with TensorFlow.js. 👋

Top comments (16)

Collapse
 
_diddlysquat profile image
Jonas Heitzer • • Edited

Hi James,
thanks or that really cool article and the example. Sadly i can't get it to work :(.

I am on a MacBook and it throws the following Error:
2018-08-10 15:17:12.564735: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA
(node:5745) Warning: N-API is an experimental feature and could change at any time.
(node:5745) UnhandledPromiseRejectionWarning: Error: SOI not found

Do you have any experience of running it on a Mac? Might Docker be a solution?

Collapse
 
jthomas profile image
James Thomas •

Hello Jonas. I've been running it on a Mac! Looking around, it looks like this is an issue with the image you are trying to classify.
stackoverflow.com/questions/493916...

Are you using the panda test image provided in the blog post or something else?

Collapse
 
_diddlysquat profile image
Jonas Heitzer •

Hi James! Thank for getting back to me. I used the panda you provided, but it seems the image was corrupted. I tried it with several other pandas and it works fine now.

Thanks again!

Collapse
 
aralroca profile image
Aral Roca • • Edited

Good article! 😊 Nevertheless, in the tfjs documentation I see that there is the method tf.fromPixels that creates a tf.Tensor from an image. Maybe is possible to replace it? Instead of jpeg-js library?

I'm not familiar with tensorflow.js, but I think that for matrix operations like this is performing better (with GPU) than using two nested fors. (Not sure...)

Thank you!! BTW; Very good article!

Collapse
 
jthomas profile image
James Thomas •

Hello Aral, tf.fromPixels can only be used in the browser I'm afraid. It extract pixels from DOM elements like video, image or canvas tags, rather than native buffers.

Collapse
 
aralroca profile image
Aral Roca •

Thank you for that clarification! I would like to try tfjs in node.js!

Collapse
 
raghul22 profile image
Raghul • • Edited

Hi James,

i am a newbie to machine learning and its really cool article. This article really helped me in understanding some of the concepts. I have tried your script, it didn't work for the mentioned panda image and got to know that image is corrupted. When i tried with another "cat" image, I am getting this error: Error: browserHTTPRequest is not supported outside the web browser without a fetch polyfill.

I am not sure, why this error is coming, as we are referring the model locally. Please help.

>node script.js mobilenet/model.json panda.jpg 
cpu backend was already registered. Reusing existing backend
2018-10-22 16:47:24.306471: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA
(node:94614) Warning: N-API is an experimental feature and could change at any time.
(node:94614) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: **SOI not found**
(node:94614) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

>node script.js mobilenet/model.json cat.jpg 
cpu backend was already registered. Reusing existing backend
2018-10-22 16:47:17.300905: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.2 AVX AVX2 FMA
(node:94615) Warning: N-API is an experimental feature and could change at any time.
(node:94615) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): Error: **browserHTTPRequest is not supported outside the web browser without a fetch polyfill.**
(node:94615) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

Thanks!

Collapse
 
itachiuchiha profile image
Itachi Uchiha •

Omg. This is amazing.

Collapse
 
tomdong profile image
Jiangfeng.Dong •

Thanks for the great article, I'm following the exact steps as you mentioned, but face some issues when loading the mobilenet model, I guess something might be changed. could you please share the version of the components used in your project?

BTW,
I'm using
"@tensorflow-models/mobilenet": "1.0.0",
"@tensorflow/tfjs": "1.0.1",
"@tensorflow/tfjs-node": "1.0.1",

Thanks a lot!

Collapse
 
rajesh36923908 profile image
rajesh •

Hi
Very useful and interesting Article. Node.js is truly powerful and useful for developers. You have shared lots of information that makes the tasks easy.

thanks
elena

Collapse
 
elenajessi1 profile image
Elena jessi •

Hi
Interesting information and knowledge. Especially the attachment that you have shared, will be much helpful for me to understand the main theme of focus dimensions and use it. there are lot of C++ libraries you could wrap up under JavaScript quite easily. Node.js can be a nice gateway to such a library.

thanks
elena

Collapse
 
veselinastaneva profile image
Vesi Staneva •

Great way to show that Machine Learning and Node.js go well together!👏👏👏

My team just completed an open-sourced Content Moderation Service built Node.js, TensorFlowJS, and ReactJS that we have been working over the past weeks. We have now released the first part of a series of three tutorials - How to create an NSFW Image Classification REST API and we would love to hear your feedback. Any comments & suggestions are more than welcome. Thanks in advance!
(Fork it on GitHub or click🌟star to support us and stay connected🙌)

Collapse
 
japitase profile image
Japita Systems Engineering •

node app.js model_data/model.json panda.jpg
cpu backend was already registered. Reusing existing backend
2019-11-08 13:31:40.445755: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
(node:15585) UnhandledPromiseRejectionWarning: Error: browserHTTPRequest is not supported outside the web browser without a fetch polyfill.
at new BrowserHTTPRequest (/home/giuseppe/Software/tfjs/mobilenet/node_modules/@tensorflow/tfjs-core/dist/io/browser_http.js:46:19)
at Object.browserHTTPRequest (/home/giuseppe/Software/tfjs/mobilenet/node_modules/@tens

Collapse
 
jochemstoel profile image
Jochem Stoel •

Excellent work, James. I'm gonna follow you around.

View full discussion (16 comments)

Read next

kafeel_ahmad profile image

15 Essential JavaScript Array Functions

Kafeel Ahmad (kaf shekh) -

rajnandan1 profile image

Built a cli for browser

Raj Nandan Sharma -

rahulgithubweb profile image

LeetCode Challenge 68: Text Justification - JavaScript Solution 🚀

Rahul Kumar Barnwal -

abhijit_panchal_2406 profile image

Odoo Full Stack Developer Interview Experience

Abhijit Panchal -