For the past two months, I've been doing some research in the field of computer vision on the web.
Image processing makes an extensive usage of maths and algorithms to extract important image features. Computer vision uses the power of image processing along with other techniques (decision trees, Bayes classifiers, deep neural networks...) in order to recognize objects or categorize images.
Computer Vision tries to do what a human brain does when it recognizes shapes, objects or situations in an image while Image processing is mainly focused on processing raw images, making them optimal for other tasks (noise reduction for example) and extracting key features.
Here are some of the libraries that I found specially interesting in the field of image processing and computer vision, all of them open source.
WebGL accelerated computer vision library. It uses a data flow paradigm to create and run graphs on the GPU. This is a very compact library: weights just 32.5K minimized.
In addition to the most common algorithms (grayscaling, color segmentation…) it implements some other more sophisticated algorithms like Canny Edges, Sobel operator and lines detections, but it also lacks of important feature extraction algorithms like FAST or ORB.
github repository: https://github.com/PeculiarVentures/GammaCV
OpenCV (Open Source Computer Vision Library) is a library of programming functions mainly aimed at real-time computer vision.
If execution in the browser is not an important requirement, Opencv4nodejs is probably the most interesting option given the performance and maturity of OpenCV.
Github repository: https://github.com/justadudewhohacks/opencv4nodejs/
You can't expect OpenCV.js to perform all the things you can do with OpenCV using C or Python or even Opencv4nodejs. Documentation is not that good either.
An additional problem to take into account is the size of the library itself, 2MB, which doesn’t make it very appropriate to all networks / devices.
To be clear, OpenCV.js is a really interesting Webassembly implementation but, in my opinion, probably you can find better alternatives depending on the kind of task you are trying to achieve.
MarvinJ provides a set of algorithms and filters (Gaussian, emboss, grayScale, thresholding…) that could be wide enough for your purposes, but as it happens in the case of GammaCV, it lacks of feature extraction algorithms. I could only find the Prewitt edge filter, which is not one exactly one of the most used.
It is very easy to implement color tracking, face detection (not recognition) or eye tracking from video or webcam. Tracking.js also provides a simple framework to implement your own object tracking algorithm. Of course it comes with some filters and feature extraction tools like FAST, BRIEF.
jsfeat has a rich and varied feature set to implement image processing in any browser. It can perform tasks such as: edge detection, image processing (grayscale, blur, etc.), corner detection, object detection, optical flow detection, etc...
This library is very lightweight (23 kB) and really fast, with very good performance on desktop computers or even mobile devices. In its website you can find lots of real time demos and examples using your webcam (webRTC required) so you can check the resulting framerate in all of them.
JSFeat documentation is very good. Of course this library include basic filters and algorithms (grayscale, derivatives, box-blur, resample, gaussian blur, equalize histogram) but also more advanced operations like:
Fast Corners feature detector
Lucas-Kanade optical flow
HAAR object detector
BBF object detector
which can be considered as advanced feature extractors.
In a next article I will show some little experiment using this library.
A machine learning model, built on Tensorflow.js, which allows for real-time human pose estimation in the browser.
PoseNet can be used to estimate either a single pose or multiple poses, meaning there is a version of the algorithm that can detect only one person in an image/video and one version that can detect multiple persons in an image/video.
As you can see there are some interesting options if you don’t want to start coding your image processing system from scratch. if you are planning to learn about computer vision I encourage you to start experimenting with them.
And that is all!. Thanks for reading this is my first article. Hope you found it useful. I look forward to hear any feedback or suggestion.