Last year (2 months ago), I created a game using p5.js and ml5.js that uses Machine Learning to detect the ASL alphabet. I made this project for my...
For further actions, you may consider blocking this person and/or reporting abuse
Hey Francis! 👋 This was such a fun read. I love how you combined ASL recognition, machine learning, MQTT, and multiplayer gameplay into one project. Using MQTT for a game instead of the usual IoT examples was especially creative. 😄
Also, the workaround for the X/Y training issue shows real persistence and problem-solving. Thanks for sharing the challenges along with the wins—it makes the journey even more interesting. Looking forward to seeing what you build next!
Wow, Francis, that's incredible work! You've really made this project come together, and I'm loving the fact that you were able to get MQTT working for multiplayer functionality. I can imagine how tough it must have been to figure out the Hand pose model and train it, but you persisted and got great results.
Thanks Aryan! It did took me a bit since I was conformable doing Image Classification. What helped was watching an Overview of the Hand pose from The Coding Train. I think @richardpascoe would love this channel since he does web development related stuff in terms of Creative Coding.
But overall, I was happy to make the multiplayer game work since that was one of my goals I wasn't able to accomplish perfectly till now. Thanks for your reply!
Wow - just wow - congratulations on accomplishing that goal you set for yourself! Looking forward to learning so much more from you!! This is great!
after reading it made me feel I too want to try something like this
Glad it gave you motivation Naveen! Thanks for reading :D
This is a really creative use of MQTT beyond the usual IoT sensor examples.
Using it just for multiplayer game state + combining it with ML hand-pose detection and Tauri is a very cool mix of technologies. Also loved the practical workaround you found for the X/Y training issue — that’s real problem-solving in action.
Curious: did you notice any latency differences between MQTT and WebSockets during gameplay?
Hey Bhavin! Hope you are well!
For context, I use Web Sockets when I was building my Chat App using Flask back around 2023-2024.
With that said, I didn't see any latency differences when it comes to using MQTT compare using Web Sockets, which was surprising giving the fact that I was publishing data points and the skeleton coordinates to the broker and then receiving that and displaying on the user side, which is a lot!
The only thing I notice was that one of the requirements for this project was making an offline mode. Because I was using libraries, I had to get it locally installed and direct it on file instead of the traditional API calling to the web.
Not only that, the libraries I locally instead ALSO does an API call to the web. For example, when I installed ml5.js locally, I notice that it was calling TensorFlow Mediapipe on the Web since ml5.js is built on TOP of TensorFlow. So, I had to install Mediapipe files locally.
It was a massive headache, but it improve responsiveness DRASTICALLY, which was awesome to see!
Thank you for the reply and hope you have a good week!
Thanks Richard! Let me know your thoughts when you deep dive into the post!
it's really cool man! Congrats!
Thanks for reading Fit!
Really impressive project! Love how you combined ML with multiplayer functionality—makes me think of innovative ways to enhance interactive gameplay.
Thanks Liam! ML with Multiplayer is quite interesting to experience!