Last year (2 months ago), I created a game using p5.js and ml5.js that uses Machine Learning to detect the ASL alphabet. I made this project for my...
For further actions, you may consider blocking this person and/or reporting abuse
Wow, Francis, that's incredible work! You've really made this project come together, and I'm loving the fact that you were able to get MQTT working for multiplayer functionality. I can imagine how tough it must have been to figure out the Hand pose model and train it, but you persisted and got great results.
Thanks Aryan! It did took me a bit since I was conformable doing Image Classification. What helped was watching an Overview of the Hand pose from The Coding Train. I think @richardpascoe would love this channel since he does web development related stuff in terms of Creative Coding.
But overall, I was happy to make the multiplayer game work since that was one of my goals I wasn't able to accomplish perfectly till now. Thanks for your reply!
Thanks for the heads-up, Francis! Will check it out, and re-read this post, when I have a little more time! Sounds rather interesting though!
Wow - just wow - congratulations on accomplishing that goal you set for yourself! Looking forward to learning so much more from you!! This is great!
Due to a meeting delay, had time to read the post this morning - truly incredible stuff and testament to your hard work and determination. Will keep this bookmarked to re-read again, so I get a deep understanding of what was involved. Well done!
Thanks Richard! Let me know your thoughts when you deep dive into the post!
Will do, Francis!
This is a really creative use of MQTT beyond the usual IoT sensor examples.
Using it just for multiplayer game state + combining it with ML hand-pose detection and Tauri is a very cool mix of technologies. Also loved the practical workaround you found for the X/Y training issue — that’s real problem-solving in action.
Curious: did you notice any latency differences between MQTT and WebSockets during gameplay?
Hey Bhavin! Hope you are well!
For context, I use Web Sockets when I was building my Chat App using Flask back around 2023-2024.
With that said, I didn't see any latency differences when it comes to using MQTT compare using Web Sockets, which was surprising giving the fact that I was publishing data points and the skeleton coordinates to the broker and then receiving that and displaying on the user side, which is a lot!
The only thing I notice was that one of the requirements for this project was making an offline mode. Because I was using libraries, I had to get it locally installed and direct it on file instead of the traditional API calling to the web.
Not only that, the libraries I locally instead ALSO does an API call to the web. For example, when I installed ml5.js locally, I notice that it was calling TensorFlow Mediapipe on the Web since ml5.js is built on TOP of TensorFlow. So, I had to install Mediapipe files locally.
It was a massive headache, but it improve responsiveness DRASTICALLY, which was awesome to see!
Thank you for the reply and hope you have a good week!
it's really cool man! Congrats!
Thanks for reading Fit!
Really impressive project! Love how you combined ML with multiplayer functionality—makes me think of innovative ways to enhance interactive gameplay.
Thanks Liam! ML with Multiplayer is quite interesting to experience!