Being a robotics and vision AI grad student at NAIST, I spend my days with a real robot arm, cameras, and a long list of failed experiments. Here's what I've learned about where things actually break.
The boring problem
People assume the hard part is the model, but for us, it's writing the Python that glues perception to motion, and every time we swapped models, we ended up rewiring the same motor functions. What changed our workflow was treating robot capabilities as reusable skill bundles (Skillware): write the deterministic script once, and any model can call it. You swap the brain without rewiring the motor functions. The problem is Skillware is mostly software focused, and although it gives you a good baseline to create your own skills, it still takes time to write scripts the first time, but then it works like magic.
Determinism is a feature, not a limitation
A robot has a fixed set of skills, and it cannot invent new ones mid-task, which is exactly what you want, that's why a deterministic skill either works or it doesn't, fits robots the best. No grey zone, no hallucinated torque values, and honestly, half of our bugs aren't AI bugs at all, but unit mismatches, degrees fed into radian-expecting nodes, mg instead of m/s².
The tradeoff is context I guess, when 20+ skills loaded in full, the context gets cluttered fast. So we only append short descriptions of what each skill does and when to use it, and the full context unlocks only when a skill is triggered. It basically doubles as smart memory management.
Knowing why the robot did something
On top of the skills, we run a harness (aura) with a native sequencer, which gives us a real audit trail, like what the robot did, when, why, and where the reasoning came from, was it the model's own thought, something from a skill bundle, or something picked up from an internet post? With physical hardware, "it kind of seemed like a good idea" is not an acceptable answer.
The real world refuses to cooperate
In vision, the hardest part is lighting and unpredictable background noise, people walking by, hands entering the scene, and here's the ironic one: even when the robot does everything perfectly, strict guardrails can make it freeze. It sees a hand in the frame, can't compute distance reliably from a frozen snapshot, decides that moving might risk contact, and freezes rather than move, which yes, is perfectly safe, but also completely useless. The robot isn't wrong, but our guardrails just have no concept of "re-plan from a new viewpoint."
So the model is maybe 20% of the problem. The rest is deterministic skills, context management, audit trails, and a world that refuses to hold still. If you're building agent pipelines for hardware, invest there first. What keeps breaking in your robot stack? Happy to trade war stories :D
Top comments (4)
So nice to read your post Rin, feels like a glimpse into your personal life, and learning something new in a short read.
Thank you so much blade! It means the world.
Really enjoyed reading this, Rin! 😊
Coming from a systems architecture background, I can relate to the idea that the hardest problems often live between components rather than inside them. Your point about deterministic skills particularly resonated with me. Separating what the AI decides from what the hardware is actually allowed to execute seems like a solid architectural boundary. One thing I’d be curious about: even with deterministic skills, the physical outcome isn’t always deterministic because of sensor noise, timing, and environmental changes. Do you also validate preconditions and postconditions at the skill execution level?
I also found your example of the robot freezing when a hand enters the frame really interesting. It highlights the difference between stopping safely and recovering safely. A system that can detect uncertainty is useful, but one that can actively reduce that uncertainty through a new observation or a safe re-planning step seems even more valuable.
Thanks for sharing these real-world lessons. It’s refreshing to read about the engineering challenges beyond the AI model itself. Wishing you the best with your research! 🤖
Thank you so much Mustafa for the thoughts and positivity. These are good questions we are trying to address ourselves in the lab, but yeah overall it is never a waterfall solution, but a bunch of moving pieces (literally in robotics), have to work in symphony to achieve somewhat reliable results, beyond a nice demo.