I am a software engineer who has mostly worked on the web. With all of the AI boom and all of the different models, our everyday work has changed a lot. Manual typing is gone and now we are mostly scoping and reviewing. So I have been curious about robotics and the whole physical world. How is it going there? And why do I still not have my own humanoid that cooks for me?
So I decided to dive a little deeper than just AI summaries and take a first step, to understand it and run into the first problems by myself. I bought a Raspberry Pi with a webcam to recreate a robot playground, but decided to skip the movement part to keep it simple and not wait too long for the delivery.
It was also interesting to me to explore the developer experience around it. I know how it works in the cloud. You can easily deploy or update any version just by connecting your repository for any part of the app. Of course if you want something specific you have to set it up on your own, but it is easy and quick to make, for example for an MVP. So is it that easy to push your own code to production when production is your own robot? How fast can I bring my own solution to my clients and how many steps does it take?
I took the Raspberry Pi as the main part of the robot, the actual production environment and not just a local on the laptop and the camera to create a visible part of the robot that can break or lag.
And I came up with Vision Stand. It has simple object detection that runs on the device, a snapshot feature and I also built a flow around it to push updates from my GitHub repository, to reduce the chance of breaking something or at least have a nice fallback.
It runs fully on ROS 2 and I tried to reuse as much of the existing web infrastructure as possible, because it is already well known and probably a lot of engineers with web experience will move towards it. And it actually works. I reused most of the things, specifically Docker, the container registry and the Ubuntu setup, the same as I would for my own VPS.
There were quite a lot of problems, but most of them were not about the code itself. The code was written fully by Claude. I had no experience with ROS 2, but with a plan it took me around 12 hours and went smoothly. I was especially impressed that Claude, with enough permissions, can connect over SSH directly to the Raspberry Pi and fix bugs there. Not all of them, but still, that was quite cool.
Compute
The first thing I hit was the compute. I was physically limited by my device. I cannot just top up the balance of my account somewhere on a website and increase the power of my service to get more frames from my camera. Once I set up object detection, 640 pixel input was too slow, about 2 frames per second and basically I could not do anything with that. So I had to switch to 320 pixel input to get 11 or 12 frames per second in the live stream with object detection, even with the compressed stream. And if I want something better, I have to go and buy a new device, wait for the delivery and set it up. There is no other way to get more power. It was also pretty loud once I ran it, because the Raspberry Pi fan was always trying to cool it down. I always needed to turn it on and off so I could sleep. That is also some extra time in the whole flow of working on it and testing it.
My MacBook against me
The whole environment on my MacBook was against my tests. First of all, I had to set up a local virtual machine just to run ROS and test it, because ROS does not really support macOS right now. So you have to be ready for that and get some extra space on your disk. I had three separate environments to work with, macOS, Ubuntu in a virtual machine and Ubuntu on my Raspberry Pi. They are all near to me, but it still took time just to set them up. And the second problem comes right after that, the connection between them. The local network permission on MacBook was dropped a few times during the work, without any clear error and it was blocking the connection to my Raspberry Pi through Lichtblick, an app to see the live stream. Maybe I was just unlucky, but it took me hours to solve it.
Camera
I also ran into some physical problems with my camera and how it connects to the Raspberry Pi. It was mostly solved by Claude as well, but it took me some effort to understand it and find out where the problem is. There were two things. The first one is the device name. If you replug your USB camera, it can end up on a different port and show up under a different device name. The second one was the streaming. Your app has to understand that the stream is gone and reconnect or it stays broken until you restart everything. I only have one camera and one Raspberry Pi. But I can imagine a full humanoid with a lot of parts of the system that connect and communicate with each other. You always need to think about what happens if one of them breaks and handle it well while the robot is running, even with the node architecture of ROS 2.
Offline
Another thing is that your machine is not always connected to the internet. At some point you can be offline and nobody guarantees it like a cloud provider does. For example, in the cloud you do not keep your images on your own server. You upload them into a bucket and all of your services, even if you have many of them, share the same images, logs or state. Here I have a simple snapshot feature where I save one frame of my live stream. It lives only on the device and it may never get uploaded to a web server because the connection is down or something breaks and you need to handle that part too. You have to think not only about the snapshot but about the state of the application, the whole working environment and what happens when the connection drops.
Push vs pull
My main goal was to recreate the experience of connecting your repository, pushing one commit, creating a release and having your new version on the Pi. On the web one push event goes directly to production. Here you cannot have a push signal, because sometimes the connection or the app is down. Instead the device needs to pull, constantly checking GitHub on a schedule and asking if a new version exists. This is a slightly different architecture and there can be some lag before your new version is running.
My favorite part was the rollback. I created a failed release on purpose and the device ran it, understood that there was no stream from the camera, rolled back automatically and remembered to never try it again. The camera was not working for a bit, but I did not do anything to recover it. You can also pin a specific version in the environment file.
There is no simple way to manage the versions on a device that stays with you in the same room. Of course you can do all of this manually over SSH and nothing is wrong with that. But if I have an idea for my own MVP, I do not want to think about versioning and fallbacks. I just want one clean way to use the infrastructure around it and ship the features related to my own project, not recreate a lot of stuff.
Updating the updater
Quite a funny part. The updater can update everything around it but not itself. During the implementation I had to improve my updater script a few times and every time I had to go directly to the device and update it there. It is the only thing I still have to run by myself on the device. The whole idea of the project is about updates and it still has a problem you never think about in the web, because it was solved by a lot of other companies, products and layers and you just use them.
There are some projects that handle OTA updates for physical devices, but it seems there is no standard for the ROS ecosystem. Let me know if you know a good tool.
Next
So why do I still not have a humanoid that cooks my chicken and rice? I do not know. But it looks like it is a bit harder than just code. And there are a few iterations we need to go through before I get a good meal.
I really enjoyed this project. Of course it is not a real robot, without any actuator and not solving a real problem. But I touched the physical world. My code runs on a device that I can just plug in, leave in another room and it works all the time.
I think the whole robotics developer experience and the infrastructure around it will take some time to mature. As AI goes further, it creates more of a trend toward robotics and the physical AI. Probably more engineers will come, solve these problems for themselves and then share it with others. Let’s see how it goes and how big the market is here. I am curious about it. Tell me what problems you run into. The code is open.
Top comments (0)