Hello guys,
Good news! Pictograms are now the official sport in #Tokyo2020.
So, I have developed an application with Pose Detection in TensorFlow.js, that we can become pictograms :D
In this article, I will explain about this application.
DEMO→https://pictogram-san.com/
github→https://github.com/tommy19970714/pictogram-san
※ I developed this application with my friends, Tommy, Waserin, Nishikawa, and mikkame.
Why Pictograms?
Now now the pictograms are official in Olympic Games.
The Tokyo Organising Committee of the Olympic and Paralympic Games (Tokyo 2020) today unveiled the official sport pictograms of the Olympic Games Tokyo 2020
Tokyo 2020 unveils Olympic Games sport pictograms
You want to try pictograms? Just try this application!
Usage
Visit https://pictogram-san.com/, then click Start Game or Take photo.
When you click Start Game, the music starts now!
Let's pose as per the subject! lol
When you click Take photo, You can get a screen shot.
If you want switch a camera between in-camera and rear camera, you can just click the following button! (Only SP is available. When you try on PC, this button is disabled.)
Features
This application has the following functions.
- Make a pictogram of the image acquired by the webcam.
- Split the screen in two vertically, and play the pictogram on the top and the webcam video on the bottom.
- By clicking the play button, the music automatically starts playing.
- Switch a camera between in-camera and rear camera
- Take a screenshot of the screen when the music ends.
Here, we used a model called pose-detection in TensorFlow.js and this application is built with Next.js and TypeScript.
I can't write about all the technical details, so now I'll just write some points important.
If you need, please check our github.
Any issues and PRs are really welcome!
Set up to use pose-detection with Next.js and TypeScript
$ yarn add @tensorflow-models/pose-detection @tensorflow/tfjs-core @tensorflow/tfjs-converter @tensorflow/tfjs-backend-webgl
※ I wrote in another article about the reason why I don't do just yarn add @tensorflow/tfjs. Have a look if you need.
If the model you use is based on wasm, you must install @tensorflow/tfjs-backend-wasm instead of @tensorflow/tfjs-backend-webgl.
Here, because the pose-detection model is based on webgl, I installed webgl.
Then, load PoseNet.
If necessary, you can specify the architecture, etc.
const modelName = SupportedModels.PoseNet
const net = await createDetector(modelName, {
quantBytes: 2,
architecture: 'MobileNetV1'
outputStride: 16,
inputResolution: resolution,
})
There are two architectures, MobileNetV1 and ResNet50. ResNet50 gives higher accuracy, but it is quite heavy, especially on a heavy phone, so we used MobileNetV1 here.
After the PoseNet has been loaded, put the image or video data in the argument of estimatePoses.
In this case, we used the webcam data as is.
const predictions = await net.estimatePoses(webcam, {
maxPoses: 1,
flipHorizontal: false,
})
Then, you can use the detected predictions and draw them as you like.
Start pose detection as soon as the webcam starts up
In order to use the information from the webcam, we need to wait until the video tag is loaded.
So, I wrote like the following.
if (webcamRef.current.video.readyState === 4) {
// Write the process of using TensorFlow.js
}
However, since the readyState is not 4 when the page is loaded, this processing part will be passed forever.
So, I tried to wait until readyState becomes 4 in the following code.
const handleLoadWaiting = async () => {
return new Promise((resolve) => {
const timer = setInterval(() => {
if (webcamRef.current?.video?.readyState === 4) {
resolve(true)
clearInterval(timer)
}
}, 500)
})
}
const handleStartDrawing = async () => {
await handleLoadWaiting()
// Here write something
}
By the way, I had used useEffect in the past my application that puts on the mask on your face.
※ Since it has no loading, you may be confused because the mask is not displayed at first, but if you wait for some seconds, the mask will be attached.
useEffect(() => {
runFaceDetect();
}, [webcamRef.current?.video?.readyState])
// https://github.com/yuikoito/mask-app/blob/master/src/App.tsx#L43-L46
Here, it was better to wait until the loading is completed with handleLoadWaiting function and I used it.
Then don't forget to specify the size of the input element and the size of the output element as well.
const webcam = webcamRef.current.video as HTMLVideoElement
const canvas = canvasRef.current
webcam.width = webcam.videoWidth
webcam.height = webcam.videoHeight
canvas.width = webcam.videoWidth
canvas.height = webcam.videoHeight * 2
At first, because I totally forgot to specify the width of the input webcam element, the width of the input value was recognized as 0, and I was troubled by the error that roi was 0. So, DON'T forget!
webcam.width = webcam.videoWidth
webcam.height = webcam.videoHeight
Switch a camera between in-camera and rear camera
To switch between the in-camera and rear camera, simply switch the facingMode between 'user' and 'environment'.
const [facingMode, setFacingMode] = useState<'user' | 'environment'>('user')
const videoConstraints = {
width: // specify width you want,
height: // specify height you want,
facingMode: facingMode,
}
// webcam part
<Webcam
audio={false}
mirrored={true}
videoConstraints={videoConstraints}
ref={webcamRef}
/>
You can then switch the value of facingMode by clicking a button.
References
This idea is based on Mr. Takahashi.
https://github.com/Kazuhito00/Tokyo2020-Pictogram-using-MediaPipe
That's it!
As I wrote, I developed this application with my friends, Tommy, Waserin, Nishikawa, and mikkame.
We developed for this two days like a Hackathon with connecting to discord.
It was a lot of fun, and I'd like to do it again.
If you want to develop with me, please contact :)!
===
This article is the eleventh week of trying to write at least one article every week.
If you'd like, please take a look at my previous weekly posts!
See you soon!
Contact
Please send me a message if you want to offer a job or ask me something.
yuiko.dev@gmail.com
https://twitter.com/yui_active
Thank you!
Top comments (1)
Your work is amazing!