Android real-time pose detection app using TensorFlow Lite.

#programming #android #career

I hope everyone is doing fine in the current situation.
The quarantine was not something that I have ever imagined before.
In Romania, we are on lockdown for almost one month now and we’ll continue one more.
There are lots of articles about productivity work from home tips but let’s face it, it's hard, it’s stressful, it’s new. When was the last time you spent an entire month inside your home?

I’m sure that each of us is doing their best, no need to overreact about this and I bet that no one is more productive now just because they are wfh, no, this is isolation, not the normal situation.
We are all on the same boat, all over the world.
Thank God we have The Internet and everyone is still connected.

When I feel down and I’m done with Netflix, HBO Go or other resources I try to redirect myself to self-improvement.
I was thinking to try fitness since nowadays I do lack it, but I know myself and I know that I can get injured really easily.
So why not combine Technology and Fitness?

Goal

To keep me busy and to learn a bit more about technology. Developing an Android app using TensorFlow Lite.

What is TensorFlow Lite

TensorFlow Lite is a production-ready, cross-platform framework for deploying ML on mobile devices and embedded systems.

It enables on-device machine learning inference with low latency and a small binary size.

TensorFlow Lite consists of two main components:

The TensorFlow Lite interpreter, which runs specially optimized models on many different hardware types, including mobile phones, embedded Linux devices, and microcontrollers.
The TensorFlow Lite converter, which converts TensorFlow models into an efficient form for use by the interpreter, and can introduce optimizations to improve binary size and performance.

TensorFlow Lite is designed to make it easy to perform machine learning on devices, "at the edge" of the network, instead of sending data back and forth from a server.

But what is it useful for?

There are different models ready to be integrated into your app from the website.

Functionalities

The app I develop will monitor my posture and remind me to stand up after a certain period of time and do some yoga exercises.

The problem

How do I determine if the movement is a yoga one?

Development workflow

Pick a model

The first model I've used is the single-pose detection model. Pose estimation is the process of utilizing computer vision techniques to estimate various elements of human posture within an image or segment of a video.

PoseNet provides a general-purpose pose estimation model. This model may be used for simple applications containing a single human body for the purpose of joint detection. As an example, the model can estimate the position of a person’s elbow and/or knee in an image. The pose estimation model does not identify who is in an image; only the positions of key body parts.

What makes this model fascinating is its ability to track joint movements in real-time with a relatively high level of accuracy and flow.

TensorFlow Lite is sharing an Android sample application that utilizes the device’s camera to detect and display key body parts of a single person in real-time. Check out the source code!

The second model I've used is an Image classification model.
Image classification is a common use of machine learning to identify what an image represents. For example, we might want to know what type of animal appears in a photograph.

The task of predicting what an image represents is called image classification.
An image classification model is trained to recognize various classes of images.

Train and Convert the model

So back to my problem, my model should determine yoga movements so I need to train it and at the end to convert the model to the TensorFlow Lite format since I need it to run on Android.

Data Collection

I’ll be training a classification model, hence my dataset will contain different kinds of images that the model has to identify (yoga movements).

I found that Python provides us with a handy third-party package called google_images_download, which has easy-to-use syntax and can automate the search over the internet of yoga images.

Before I train my model, it’s essential that I get rid of these images from the dataset that are garbage.

Train the model

I'm using AutoML Vision to train and convert the model. A quick start you can find it here.

My dataset looks like:

Doing this training in a traditional way would have taken way more time and effort on my end and would have involved installing and setting up dozens of third party packages. I think it is a good idea to use existing tools like AutoML.

After completing this process I have exported the trained model files: a TF Lite file, a label file and a metadata file.

Test the model

I uploaded a test picture to see if it is classified right

Deploy & use & have fun!

I added my model to my existing app, so now I can run it on the android phone.

I know that everyone is thinking about how to protect themselves and their families, food, and shelter but in the end, programming is how we get paid and we have to remember the privilege that we have by being able to work from home.
Let's take this quarantine to show our passion, to connect with the people we love and to shift the mindset to something better.
Don't get angry, don't let emotions drive you in and focus on what you can control.

Thanks for reading!