In this program, puppet doll
mimic the pose of the smartphone. (smartphone built-in inertia sensor used)
mimic the pose of your face. (Web camera + PoseNet used)
stare at you. (Web camera + PoseNet used)
You can switch between these 3 modes.
Two servo motors are used, and in each mode, the doll can change the posture up and down, left and right.
Materials
- obniz
- Micro servo SG-5010 x 2 pieces https://obniz.io/sdk/parts/ServoMotor/README.md?iframe=false
- Pin header (both ends long) 1 x 6 http://akizukidenshi.com/catalog/g/gC-09055/
- PC / Smartphone (PoseNet can be used)
- Powerful double-sided tape
- Puppet doll
- Mechanism for fixing the puppet doll and servomotor. (A bar shaped object, L - shaped bracket, etc.)
- Base and weight
Operation confirmation
Since the smartphone's inertial sensor is used in mode 1, and web camera and PoseNet is used in mode 2 or 3, this program may not work depending on the device. (It is not necessary to connect to obniz if you want to do only the operation confirmation below.)
Operation confirmation of the inertial sensor
Execute the program, select mode 1 and press the SET button, then yaw, pitch, and roll values are displayed on the line labeled "RAW:" in the "Status" view at the bottom of the screen.
When keeping the smartphone horizontal and rotating it like a compass needle on the horizontal plane, if the leftmost value (yaw) changes between 0 and 360 degrees, it works properly.
We confirmed the operation with these environments.
- Xperia Z3 (Android 6.0) + Chrome
- AQUOS PHONE (Android 8.0) + Chrome
Operation confirmation of PoseNet
Execute the program (HTTPS connection required), select mode 2 or 3, wait a while, and if the image of the camera appears and the position of eyes and nose are recognized, it is working properly.
We confirmed the operation with these environments.
- AQUOS PHONE (Android 8.0) + Chrome
- MacBook Pro 2017 + Chrome In the case of smartphones, in low speed mode of about 1 fps in the mode using the camera, in MacBook Pro, it got about 10 fps.
How to use
When obniz is connected, select the mode and press the SET button to start the operation. In Mode 1, the operation does not start until the START button is pressed in order to align the orientation of the smartphone with the doll. The STOP button stops the servo motors as it is. The RESET button returns the servo motors to the initial position and stops.
Also, in order to prevent the servo motors from moving rough, Max Speed and LPF can be adjusted with bars. LPF is the coefficient of the low pass filter. The filter on the right is stronger and stops at the far right. The smaller the Max Speed and the larger the LPF, the servo motors move gently, but the slower the reaction. If it is reversed, the reaction is early, but it gets rough.
How to make (hardware)
The circuit is just connecting two servo motors to obniz. They could be connected to anywhere, but I connected them as below.
Yaw direction servo motor
io0: signal line (yellow)
io1: VCC (red)
io2: GND (brown)
Pitch direction servo motor
io3: signal line (yellow)
io4: VCC (red)
io5: GND (brown)
Two servo motors are fixed in such a direction as the picture, the lower servomotor is for the yaw axis and the upper servo motor is for the pitch axis.
Regarding the fixing of the servo motors, it is necessary to strongly bond the servo motors because the force due to the inertia of the doll when the servo motor moves quickly adds a large force to the contacts between the servo motors. This time I used the largest pedestal attached to the servomotor, I used 3M "super strong double-sided tape" with cushion.
Also a blue stick extends upwards, covered with a doll. It is necessary to devise to prevent the doll from idling.
How to make (software)
Mode selection
Elements whose classes are specified as “view-contents-1, 2, 3” are displayed only in each mode. You can select the mode from 1 to 3 with the select element. Pressing the SET button toggles the display according to the value of the mode selected at that time and executes bindPage() when using PoseNet and start posture estimation.
Getting inertial sensor values
In mode 1 the inertial sensor value of the device is used. $(Document).ready is executed when loading of HTML is completed, and here
every time the pose of the device is obtained and yaw and pitch are calculated from eventData, and it is made to be able to refer at any time as smp.yaw and smp.pitch. The value is adjusted using the “constrain” function because the servo motor rotates from 0 to 180 degrees.
Pose estimation with PoseNet
In mode 2 and 3, the pose estimation by Web camera and PoseNet is used. I was used this site as a reference.
http://developers.gnavi.co.jp/entry/posenet/hasegawa
Set up of PoseNet and web camera is done in bindPage(); and start pose estimation with detectPoseInRealTime(video, net);
net.estimateSinglePose(video, imageScaleFactor, flipHorizontal, outputStride); is actually the part where the pose is estimated.
In drawPoints(poses, ctx);, get the coordinates of the parts, such as eyes and nose from the pose estimation results, and in the canvas, points and their part names are drawn at the place of each part.
In addition, for the mode 2, the yaw and pitch values of the orientation of your face are simply calculated by trigonometric functions, and the values are stored in the object “mypose”. for the mode 3, the yaw and pitch values the doll should face is calculated simply from the position of the nose in the screen, and the values are stored in the object “myposition”.
Driving of the servo motors
The angles of servo motors are updated by using the setInterval function in startServo function.
When starting the respective modes,
Mode 1 ... startServo (smp, 50)
Mode 2 ... startServo (mypose, 100);
Mode 3 ... startServo (myposition, 100)
are executed respectively, for example in the mode 1, with reference to the variable “smp”, “updateServo” function which is to update the angle of the servo motor in accordance with the values of yaw and pitch of “smp” will be called every 50 ms.
The angle of the servomotor is calculated and updated in this function:
moveServoToward (src.yaw, src.pitch, max_deg, min_deg);
In this function, a simple low-pass filter is implemented as a program to smooth the motion of the servomotor. lpf_a is the coefficient of the low-pass filter, s_yaw, s_pitch is the value of yaw and pitch of the servo motor before updating. A function that updates the servo motor angle directly is in setServo, and at the same time, that angle is assigned to s_yaw and s_pitch.
Program
https://obniz.io/explore/36
Top comments (0)