CVisor : Using Computer Vision to stop COVID-19

Shoray Singhal — Sun, 29 May 2022 13:41:13 +0000

Introduction of the Program

Every year, more than 30k sophomore students apply for the Microsoft Intern Engage program and only a fraction get selected for the mentorship.
Selected mentees are alloted mentors and have to choose one of given project tracks and have to submit a functional project within 4 weeks. Those with high scoring projects are given Internship offers at Microsoft.

My project idea

For my project I choose the face recognition track. I decided to build a crossplatform webapp where people can analyse various video media for compliance with COVID-19 norms. With COVID cases on the rise once again this app can really come in handy.
Establishments can use this app to feed in their live cctv data for realtime analysis.
They even can upload recordings for analysis.
The app has support for mask detection and even works for crowds.
And since it runs on the browser its pretty fast too.

INIT

I am the type who developer who first develops then designs later. Taking the same approach here I first decided to complete the functionality of the app and then focus on the UI/UX.
In the end after the design process was done the app home page looked like this...

Tools used

Vanilla HTML+CSS+JS
Bootstrap
Python
Tensorflow.js
Teachable Machine
ML5.js
Face-api.js
Git/GitHub
Netlify

Since the webapp was going to do realtime computations I concluded that low-latency and high-speeds were paramount. That's why I went with a client-side approach. To maximise rendering and loading times I wrote everything in Vanilla instead of using somekind of framework like React or Angular.
To hasten the design phase I used Bootstrap.
Since this is a webapp CI/CD is very important for this purpose I first hosted my code on GitHub and connected that repo with netlify that takes care of CI/CD by itself. I just need to push the changes to the project and a new version of the app is deployed automatically saving me a lot of hassle.

Models

While researching for the project I came to know about Tensorflow.js which which allows deployment of pretrained ML models on the web with remarkable speeds.
At first I use face-api's SSDMobilenetv1 model to isolate faces in the given data. I then isolate those faces and then pass them to a
a teachable-machine model trained specially to classify masks. You can find all model links in the project readme

Results

Some Problems I faces

Every project is filled with unexpected projects and this project was no other. Namely I faced great difficulty in integrating different models with each other.
The large size of some models also meant that that testing was difficult.
Adding responsiveness to the app was also a tiresome problem. I had difficulty implementing the video canvas on the stream&video pages.
In the end all these problems were solved by the help of my mentor and StackOverflow bring forward a project that I learned a lot from and am very proud of.

Imporvements Required

While the models used are very light weight, they still take a lot of time to compute. Optimizing these models can result in an improvement.
The app performs poorly on old mobile phones with less computational power. -The models also sometimes fail in picking up some faces in the crowd.
UI/UX improvements.

Check out the webapp HERE

Check out the repo HERE

Conclusion

The Engage program was a very educative experience for me and during this I got to talk to a lot of talented and experienced people.
Thanks Microsoft for the opportunity and now I am very eagerly waiting for the results

DEV Community: Shoray Singhal