If you're like me, then you'd do pretty much anything to have your own R2-D2 or BB-8 robotic buddy. Just imagine the adorable adventures you'd have together!
I'm delighted to report that the Anki Cozmo is the droid you've been looking for.
Cozmo is big personality packed into a itty-bitty living space. You don't need to know how to code to play with Cozmo - but if you do - then Cozmo has even more phenominal cosmic power.
In this post, I'm going to show you how you can teach your own Cozmo to recognize everyday objects using transfer learning with TensorFlow on FloydHub.
virtualenv ~/.env/cozmo -p python3
git clone https://www.github.com/whatrocks/cozmo-tensorflow
pip install -r requirements.txt
Getting enough training data for a deep learning project can be a pain. But thankfully we have a robot who loves to run around and take photos with his camera, so let's just ask Cozmo to take pictures of things we want our robot to learn.
Let's start with a can of delicious overpriced seltzer. Place Cozmo directly in front of a can of seltzer. Make sure that your robot has enough space to rotate around the can while it is taking pictures. Be sure to enter the name of the object that Cozmo is photographing when you run the
python3 cozmo-paparazzi.py seltzer
Repeat this step for as many objects (labels) as you want Cozmo to learn! You should now see all your image labels as subdirectories within the
/data folder of your local directory.
Next up - let's upload our images to FloydHub as a FloydHub Dataset. This will allow us to mount these images during our upcoming model training and model serving jobs on FloydHub. Datasets on FloydHub are an easy way for your training jobs to reference a version-controlled dataset.
floyd data init cozmo-images
floyd data upload
In our case, I've named this image dataset
cozmo-images. I've made it a public dataset, so feel free to use it in your own Cozmo projects!
And now the fun begins. First, Make sure you are in the project's root directory, and then initialize a FloydHub project so that we can train our model on one of FloydHub's fully-configured TensorFlow cloud GPU machines.
Side note - if that last sentence sounded like a handful, then just know that FloydHub takes care of configuring and optimizing everything on your cloud machine so that it's ready for your GPU-powered deep learning experiments. You can specify the exact deep learning framework you'd like to use - whether that's TensorFlow 1.4 or PyTorch 0.3 or more - and FloydHub will make sure your machine has everything you need to start training immediately.
Okay, back to business, let's initialize our project:
floyd init cozmo-tensorflow
Now we're ready to kick off a deep learning training job on FloydHub.
A few things to note:
- We'll be doing some simple transfer learning with the Inception v3 model provided by Google. Instead of training a model from scratch, we can start with this pre-trained model, and then just swap out its final layer so that we can teach it to recognize the objects we want Cozmo to learn. Transfer learning is a very useful technique, and you can read more about it on TensorFlow's website.
- We're going to be mounting the images dataset that Cozmo created with the
--dataflag at the
/datadirectory on our FloydHub machine.
- I'm enabling Tensorboard for this job with the
--tensorboardflag so that I can visually monitor my job's training process
- I've edited the
retrain.pyscript (initially provided by the TensorFlow team) to write its output to the
/outputdirectory. This is super important when you're using FloydHub, because FloydHub jobs always store their outputs in the
/outputdirectory). In our case, we'll be saving our retrained ImageNet model and its associated training labels to the job's
floyd run \
--data whatrocks/datasets/cozmo-images:data \
'python retrain.py --image_dir /data'
That's it! There's no need to configure anything on AWS or install TensorFlow or deal with GPU drivers or anything like that. (If you're paying close attention, I didn't include the
--env flag in my job command - that's because FloydHub's default environment includes TensorFlow 1.1.0 and Keras 2.0.6, and that's all I need for my training 😎).
Once your job is complete, you'll be able to see your newly retrained model in your job's output directory.
I recommend converting your job's output into a standalone FloydHub Dataset to make it easier for you to mount our retrained model in future jobs (which we're going to be doing in the next step). You can do this by clicking the 'Create Dataset' button on the job's output page. Check out the Dataset called cozmo-imagenet to see my retrained model and labels.
We can test our newly retrained model by running another job on FloydHub that:
- Mounts our retrained model and labels
- Sets up a public REST endpoint for model serving
Model-serving is an experimental feature on FloydHub - we'd love to hear your feedback on Twitter! In order for this feature to work, you'll need to include a simple Flask app called
app.py in your project's code.
For our current project, I've created a simple Flask app that will receive an image from Cozmo in a POST request, evaluate it using the model we trained in our last step, and then respond with the model's results. Cozmo can then use the results to determine whether or not it's looking at a specific object.
floyd run \
--data whatrocks/datasets/cozmo-imagenet:model \
Finally, let's run our
cozmo-detective.py script to ask Cozmo to move around the office to find a specific object.
python3 cozmo-detective.py toothpaste
Every time that Cozmo moves, the robot will send an black and white image of whatever it's seeing to the model endpoint on FloydHub - and FloydHub will run the model against this image, returning the following payload with "Cozmo's guesses" and how long it took to compute the guesses.
If Cozmo is at least 80% confident that it is looking at the object in question, then the robot will run towards it victoriously!
Once Cozmo's found all your missing objects, don't forget to shut down your serving job on FloydHub.
It's a magical world, Cozmo, ol' buddy... let's go exploring!
I'm eager to see what you and your Cozmo can find together, along with a little help from your friends at FloydHub. Share your discoveries with us on Twitter!
This project is an extension of @nheidloff 's Cozmo visual recognition project and the Google Code Labs TensorFlow for Poets project. I also wrote about this project on my personal site - except with a lot more references to Short Circuit and The Legend of Zelda.