Tutorial: Hacking a robot while respecting fundamental laws of robotics

This past week I got invited to attend Twilio Signal conference in San Francisco as part of their Champions program.

I had the chance to meet other awesome Champions at our Summit. You should all follow them, they are all doing amazing things in their communities all around the world. Find more about the Twilio Champions program over here.

The conference itself was full of great announcements around the Twilio eco-system. I am mostly excited by Conversations, as a way to connect many people uing different systems (SMS, WhatsApp, chat...) into the same conversation. The new Twilio CLI to do everything around Twilio directly in the terminal looks pretty awesome too.

Another big announcement at Signal was version 3 of TwilioQuest available on Desktop. If you are not familiar with TwilioQuest, imagine a video game where you gain XP points by solving coding challenges.
It's a fun way to learn about the Twilio world and programming in general.

Screenshot of TwilioQuest

The main character you interact within TwilioQuest, is named Cedric and is a friendly robot that is guiding you through your quest against the Legacy Systems.

The thing I did not realize: Cedric is real! He and his friends attended Signal too. They were so many roaming the expo floor and booths.

There even was a special Signal mission on TwilioQuest to earn XPs if you were spending some time with the team of Misty Robotics (parent company of the Misty robot, Cedric's robot family).

And this exactly what I did! I went over to meet the Misty Robotics team to learn more about those friendly robots🤖

The team was organizing Developers testing sessions, and inviting people to try their Hello World tutorial to "hack" the robot.

Misty is an incredible robot, among many things she has multiple sensors to move around your house without bumping into anything, a speaker to express herself, a screen to show some emotions, and a camera with face recognition capability. And all that is hackable through an SDK! 🎉

Isn't she lovely?

I had a lot of fun following the tutorial. I made Misty move, make her blink her light and even taught her to recognize me. At the end of the session, the team was handing out surveys to collect feedback. And as you may have already guessed it was a Typeform! 🤩

Then I had no option... I had to connect Misty to Typeform.

I wanted to make Misty react every time someone was filling up the survey.

Here is how I managed to connect the two.

Overall principle 📖

We rely on Typeform webhooks functionality to react immediately after someone answered.

The webhook is received by an app that can call the Misty API and make the robot react.

But to be able to communicate with the Misty robot, your app needs to be on the same WIFI network, so it's most likely that you will run it locally on your laptop.

We will use ngrok to expose localhost to the world 🌍.

Make the Misty move her arm

We create a small express app with only one endpoint in a file named app.js. This endpoint will receive webhook notification and then call Misty API.

This is how it looks:

const express = require('express')
const app = express()
const port = 3000

const axios = require('axios')
const bodyParser = require('body-parser')

// parse application/x-www-form-urlencoded
app.use(bodyParser.urlencoded({ extended: false }))

// parse application/json
app.use(bodyParser.json())

const ROBOT_IP = '10.41.129.96' //change to your own
app.post('/', (req, res) => {
  res.sendStatus(200)

  axios({
    method: 'POST',
    url: `http://${ROBOT_IP}/api/arms`,
    data:{
        Arm: "left",
        Position: -90,
        Velocity: 100,
        Units: "degrees"
      }
  }).catch(function(err){
    console.log(err)
  })
})

app.listen(port, () => console.log(`Example app listening on port ${port}!`))

You can run the app by using the command node app.js

And to make her arm go up, run the following command curl -X POST https://localhost:300 in your terminal.

And if everything works well you should see Misty's left arm going up 🎉

Connect it to Typeform 🔗

All this is working well locally, but wouldn't it be cool to connect it to other services?

To make it happen we need the help of an awesome tool called ngrok. ngrok will expose your localhost and make it available to rest of the internet by giving it a URL. This URL is unique to you and every time you launch ngrok. Make sure to read its getting started guide.

Once you have ngrok installed you can run the command ngrok http 300.
This should give you a URL back. We are going to use it to call our app from the outside.

Screenshot of ngrok

On your Typeform dashboard, we can now select the form we want to connect to Misty, and under Connect > Webhooks add a new webhook with this URL.

And voilà 🎉
You now have a robot that raises her arm every time someone fills up your typeform.

This will work until you kill ngrok or the node app. Ngrok will give you a new URL every time you launch it, so just keep it open while you are developing.

Go beyond 🚀

After seeing Misty raising her arm the first time, I could not restrain myself, and I shouted a loud "Woo-hoo" on the conference floor 😊.

But I could not stop jus now when I was so close to building something even cooler.

My ultimate goal was to build an app that will make Misty speak and react to the things that were posted on the typeform.

Make her arm move dynamically 👋

So I built a simple typeform, where people could leave their name and decide which arm Misty should raise.

To make it easier to extract data from typeform webhook payload, I had to update the ref of my questions. On your typeform select the question and look at the bottom of the sidebar and click Edit. There you can change it to whatever you please. I called mine username and arm_choice.

Edit Ref on typeform question

Now let's see how to extract those values from the webhook payload 👇

  const { form_response } = req.body
  const {answers} = form_response

  let arm_field = answers.find((a) => a.field.ref === 'arm_choice')
  let selected_arm = arm_field.choice.label

  let username_field = answers.find((a) => a.field.ref === 'username')
  let username = username_field.text

Then you can pass select_arm value to Misty API, and raise the corresponding arm.

axios({
    method: 'POST',
    url: `http://${ROBOT_IP}/api/arms`,
    data:{
        Arm: selected_arm,
        Position: -90,
        Velocity: 100,
        Units: "degrees"
      }
  }).catch(function(err){
    console.log(err)
  })

Maker her speak 🤖🗣️

Misty REST API lets you send WAV audio files using the SaveAudio endpoint (doc) but... they have to be encoded in base64 😤

So I had to find a hack to generate a .wav file and convert it to base64.

It took me a couple of trials to find the perfect library to do it, but say.js saved my day!

const say = require('say')
say.export(`Hello ${username}`, 'Alex', 0.75, `hello_${username}.wav`, (err) => {
  if (err) {
    return console.error(err)
  }
})

Using those few lines, it creates a .wav file named hello_${username}.wave with a voice that says Hello nicolas for example.
Unfortunately, I could not find the name for the female voices on my mac, so we are stuck with Alex for now 🤷‍♂️

cont fs = require('fs');
let wav = new WaveFile(fs.readFileSync(`hello_${username}.wav`));
let wav64 = wav.toBase64()

  axios({
    method: 'POST',
    url: `http://${ROBOT_IP}/api/audio`,
    data:{
      FileName: `hello_${username}.wav`,
      Data: wav64,
      ImmediatelyApply: true, //make Misty play it right away
      OverwriteExisting: true
    }
  }).catch(function(err){
    console.log(err)
  })

And this 🔝 is how I read the file, convert it to base64 and call Misty API to make it play on the robot.

And this how it finally looks 🤩
Liquid error: internal

The code for the entire app is available here👇

Hope that got you excited to hack things around and play with Misty 🤖.
Their crowdfunding campaign is over, but you can already pre-order it.

🙏 Special thanks to Twilio Champions team for inviting me to be part of this great adventure 😃