Robots and AI are often talked about as if they are the same project, but they are not. The AI research that gets headlines (big vision models, reinforcement learning, large language models) mostly runs in simulation or in a data center. The robot has to deal with gravity, friction, battery drain and the fact that a dropped object does not generate a loss function. This post looks at where the two fields actually meet today, what each AI technique contributes to a physical machine, and where the real bottlenecks are.
Where AI Actually Touches a Robot
A robot's software stack usually has three layers: perception, planning, and control. AI has made real progress in the first two. Control, the part that turns a plan into motor commands, is still mostly classical: PID loops, model predictive control, inverse kinematics. The AI work sits on top of that layer, not inside it.
That split explains why a robot can have a strong vision model and still fail to pick up a cup. The model tells it where the cup is. It does not tell the motors how much force to apply before the cup slips.
Vision Models: Seeing Before Moving
Vision is the most mature AI contribution to robotics. Object detection and segmentation models let a robot find items on a shelf, identify a surgical landmark, or track a package on a conveyor. Depth estimation, often from stereo cameras or LIDAR fused with a learned model, gives the robot a 3D sense of its surroundings instead of a flat image.
# simplified perception loop
frame = camera.capture()
detections = vision_model.predict(frame)
for obj in detections:
depth = depth_estimator.estimate(obj.bbox, frame)
world_pose = camera_to_world(obj.bbox, depth)
planner.add_target(world_pose)
This loop runs several times a second on a real robot. The model's job ends at producing a pose. Everything after that, approaching the object and gripping it with the right force, is a separate problem.
Reinforcement Learning: Learning to Move
Reinforcement learning (RL) trains a lot of locomotion and manipulation policies. Instead of hand coding a walking gait, you simulate the robot thousands or millions of times, reward it for staying upright and moving forward, and let the policy emerge.
# simplified RL training loop
for episode in range(num_episodes):
state = env.reset()
done = False
while not done:
action = policy(state)
next_state, reward, done = env.step(action)
buffer.store(state, action, reward, next_state)
state = next_state
policy.update(buffer.sample())
Boston Dynamics' Atlas and Tesla's Optimus both lean on this kind of training for balance and recovery. Sim-to-real transfer is the hard part: a policy that works in simulation often needs real-world fine-tuning because friction, motor backlash and sensor noise never match the simulator exactly.
Humanoid and Mobile Robots in Production Today
- Warehouse and logistics: Amazon's fleet of mobile robots moves shelves and packages across fulfillment centers, guided by AI path planning that reroutes around people and obstacles in real time.
- Humanoids: Figure's robots and Tesla's Optimus are being tested on tasks like sorting and simple assembly, using vision-language-action models to map a camera feed and a spoken instruction to a sequence of movements.
- Surgical robots: Intuitive Surgical's da Vinci system still relies on a human surgeon for decisions, but AI assists with instrument tracking and tissue recognition.
- Autonomous drones: AI-driven drones now handle inspection of power lines, crop monitoring and search and rescue, using onboard vision models for obstacle avoidance where GPS is unreliable.
What Large Language Models Add
LLMs are showing up in robotics as planners, not controllers. Given a high-level instruction like "clear the table," an LLM can break it into subtasks (locate objects, pick each one up, place it in a bin) that get handed to lower-level motion policies. Google's RT-2 and similar vision-language-action models go further, mapping images and text directly to robot actions. The LLM does not move a single motor; it decides what should happen next and leaves the execution to the control stack described earlier.
The Real Limitations
- Power: A humanoid robot like Optimus runs for roughly 2-5 hours on a charge doing moderate work, far short of a human shift.
- Dexterity: Human hands have 27 degrees of freedom and a sense of touch that current robot grippers still cannot match, which is why most warehouse robots move boxes rather than individual items.
- Cost: A capable manipulation arm with force sensing still costs tens of thousands of dollars, which keeps humanoid robots out of most workplaces for now.
- Safety: A robot operating near people needs certified force limits and fail-safes, which slows down deployment far more than the AI model does.
Where This Is Headed
The next few years look less like "robots get smarter" and more like "robots get cheaper and more reliable at narrow tasks." Expect AI to keep improving perception and task planning while mechanical engineering, battery density and manufacturing cost close the gap on the physical side. The bottleneck has shifted: it used to be the software. Increasingly, it is the hardware underneath it.
Top comments (0)