Sign Language Translator to Malayalam

# We haven't finished our project. We couldn't collect required dataset due to Pandemic breakout. So we are making our own at home.

Malayalam sign language is different because it have no signs for letters. It as only signs for words and most signs are expressed using upper half of the body.

As most of the signs consist of movements, we are using video dataset.

Videos are splited into image frames. 7 to 9 frames per video and used Openpose for finding the image points.

My Final Project

Link to Code
{% github https://github.com/Amchuz/Action-Recognition-in-Colab %}

I collected videos of 10 Types of actions: ['വീട്', 'ജോലി', 'സഹായിക്കൂ', 'വെള്ളം', 'വൈകുന്നേരം', 'അമ്മ', 'ആശുപത്രി', 'മഴ', 'കുട', 'ബസ് സ്റ്റോപ്പ്']. The total video lengths are about 21 mins, containing about 10000 video frames recorded at 10 frames per second.

The workflow of the algorithm is:

-Get the joints' positions by OpenPose.
-Track each person. Euclidean distance between the joints of two
  skeletons is used for matching two skeletons.
-Fill in a person's missing joints by these joints' relative pos in
  previous frame. 
-Add noise to the (x, y) joint positions to try to augment data.
-Use a window size of 0.5s (5 frames) to extract features.
-Extract features of 
              (1) body velocity and          
              (2) normalized joint positions        
              (3) joint velocities.
-Apply PCA to reduce feature dimension to 80. Classify by DNN of 3
  layers of 50x50x50 (or switching to other classifiers in one line). 
-Mean filtering the prediction scores between 2 frames. Add label
  above the person if the score is larger than 0.8. 

During this locked down, I got time to research more on this topic. But using video frames as dataset was a challenge as My laptop didn't had much space. But one day morning my ubuntu partition was deleted accidentally and to resolve it I had to buy a software. As the only important document I had in Ubuntu was my project, decided to reinstall the OS. I started the project again from scratch. But,the good news was now I had enough memory space. This journey of my project taught me that, everything happens for a reason. We just have to find the bright side.

I will updating the project code in a week.

I can't believe four years have passed and I am going to be a graduate soon.

Thank you GitHub. The place where I learned to grow.