GitHub Finish-Up-A-Thon Challenge Submission
What I Built
I built a Sign Language Recognition System that uses computer vision and machine learning to recognize sign language gestures and convert them into text in real time.
The goal of the project is to make communication easier for people who use sign language by creating an accessible software-based solution that can recognize hand signs through a camera.
The system currently supports American Sign Language (ASL) alphabets and several static hand signs. While it is still a work in progress, the model is capable of recognizing many common gestures with reasonable accuracy and serves as a foundation for future improvements.
Tech Stack
- Python
- OpenCV
- MediaPipe
- TensorFlow / Keras
- NumPy
- Scikit-learn
- GitHub Copilot
Demo
GitHub Repository: https://github.com/Tahoora-Tabassum/Sign2Speech-Converter
Video Walkthrough: https://youtu.be/qeveizcQq7E
The Story Behind the Project
This project originally started as my final-year computer science project.
At the time, I had very little understanding of how machine learning systems were trained or deployed. While looking for project ideas, I came across a video about researchers who had developed gloves capable of converting sign language into speech in real time.
I found the idea fascinating, but it relied on specialized hardware. That made me wonder:
"What if I could build a software-only version that uses a regular camera instead?"
That question became the starting point of this project.
Because it was a college project with strict deadlines, I focused on getting a working prototype completed in time for submission. The system could recognize some signs, but the dataset was limited, the model wasn't trained enough, and many parts of the workflow were still manual.
After the project was submitted, development slowed down and the repository remained mostly untouched.
When the GitHub Finish-Up-A-Thon challenge came along, I decided it was the perfect opportunity to return to the project and improve it properly.
Where It Was Before
- Limited training data
- Recognition accuracy was inconsistent
- Data collection worked only through a live camera feed
- Training interruptions meant restarting from the beginning
- No efficient way to expand the dataset
- Several repetitive tasks required manual work What I Improved During the Challenge Better Dataset Collection
Before vs After
![Before and After Comparison]
Left: Original final-year project version. Right: Improved version developed during the GitHub Finish-Up-A-Thon.
One of the biggest limitations of the original project was data collection.
Initially, I could only collect gesture data directly from a webcam. During this challenge, I expanded the pipeline so data can also be collected from device-stored videos.
I also added support for extracting gesture frames from YouTube videos, making it significantly easier to build larger datasets without recording every sample manually.
Improved Training Workflow
Previously, if model training stopped because of an interruption or system issue, the entire training process had to start from scratch.
I implemented checkpoint-based training so the model can resume from where it left off instead of losing all progress.
This saves time and makes experimentation much more practical.
Expanded and Refined Dataset
I collected additional sign language samples from multiple sources and used them to improve the training dataset.
While the model still has room for improvement, the larger dataset helped improve recognition performance compared to the original version.
Accuracy Improvements
I spent considerable time tuning the training process, improving preprocessing steps, and experimenting with model configurations.
The result is a system that performs noticeably better than the original college submission and can reliably recognize ASL alphabets and several static hand gestures.
Better Project Structure
I reorganized parts of the codebase to make future development easier and reduce the effort required to add new signs and datasets.
My Experience with GitHub Copilot
GitHub Copilot played an important role in helping me improve the project.
One area where it helped significantly was feature development. When I wanted to extend the data collection pipeline beyond live camera input, Copilot assisted in generating boilerplate code for handling video processing workflows.
It also helped me implement training checkpoints so interrupted training sessions could resume automatically rather than starting over from the beginning.
Throughout development, Copilot accelerated repetitive coding tasks, suggested implementation approaches, and helped me focus more on experimenting with the machine learning pipeline rather than spending time writing routine code.
While the overall project architecture, training strategy, and system design decisions were still my responsibility, Copilot acted as a valuable development companion that increased my productivity throughout the challenge.
Looking Ahead
This project is still evolving.
Future improvements I would like to explore include:
- Dynamic sign recognition
- Sentence-level recognition
- Text-to-speech output
- Larger datasets
- Support for additional sign languages
- Improved real-time performance
The GitHub Finish-Up-A-Thon gave me the motivation to revisit a project that started as a college requirement and turn it into something significantly more capable than the original prototype.
Thank you for reading, and I'd love to hear your feedback.
Top comments (0)