With the rapid development of LLMs (Large Language Models) and VLMs (Vision-Language Models), a new type of model called VLA (Vision-Language-Action) has been attracting increasing attention in robotics.
In this series, I will use Hugging Face LeRobot and the relatively low-cost SO-101 robot arm to experiment with imitation learning, robot learning, and eventually VLA models on real hardware.
In this Part 0, I will first cover:
- What VLA is
- What LeRobot is
- Why I chose LeRobot instead of starting with ROS + Raspberry Pi
- What hardware I purchased to build the SO-101
In the following articles, I plan to move from assembling the robot to calibration, teleoperation, dataset collection, imitation learning, and eventually VLA experiments.
What Is VLA?
VLA stands for Vision-Language-Action.
In a very simplified form, it looks like this:
Vision
"What can the robot see?"
+
Language
"What does the human want?"
↓
VLA
↓
Action
"How should the robot move?"
LLMs have already become widely known through systems such as ChatGPT.
An LLM primarily works with language.
From there, VLMs (Vision-Language Models) expanded AI systems so that they could process both language and visual information such as images.
A simplified progression looks like this:
LLM
Understand and generate language
↓
VLM
Understand language + images
↓
VLA
Understand language + images
and generate physical actions
Up to the VLM stage, AI primarily observes information in a digital environment and responds to it.
VLA goes one step further.
The goal is for an AI model to:
observe the physical world, understand a human instruction, and actually control a robot to perform the task.
For example, suppose a camera sees:
- a red block
- a blue box
- a robot arm
and the user says:
"Put the red block into the blue box."
A VLA system could generate a sequence of robot actions such as:
Move the arm toward the red block
↓
Open the gripper
↓
Grasp the block
↓
Lift the block
↓
Move toward the blue box
↓
Open the gripper
The important difference is that the AI does not simply respond:
"There is a red block in the image."
Instead, it takes action in the physical world.
Physical AI
Combining AI with robots, cameras, sensors, motors, and other physical systems so that the AI can
perceive → reason → act
in the real world is increasingly described as Physical AI.
VLA is one important approach toward building Physical AI systems.
For my experiments, I will use:
Hugging Face LeRobot
SO-101 robot arm
to explore this area using relatively inexpensive hardware.
Physical AI Is Becoming Accessible to Individuals
Traditionally, buying a robot for serious robotics research could be extremely expensive.
Even relatively small research or industrial robot systems could easily cost tens of thousands of dollars.
In that environment, a robot costing approximately as much as a passenger car could almost be considered "inexpensive" for research use.
However, the situation has been changing rapidly.
Today we have access to increasingly affordable:
- servo motors
- cameras
- Raspberry Pi computers
- NVIDIA Jetson devices
- small robots
- drones
- 3D printers
- open-source robotics software
- open AI models
Because of this, robotics is gradually moving away from a world where meaningful experiments require expensive proprietary hardware.
Instead, individuals can now assemble relatively inexpensive components and experiment with robotics and AI themselves.
The SO-101 is a good example of this trend.
Instead of purchasing an expensive research-grade robot arm, it is possible to build a small robot arm for only a few hundred dollars and use it to experiment with technologies such as:
- imitation learning
- robot learning
- VLA
- Physical AI
For me, this is one of the most exciting aspects of the SO-101.
Why LeRobot Instead of ROS + Raspberry Pi?
Of course, SO-101 + LeRobot is not the only way to combine inexpensive hardware with AI.
For example, it is possible to connect motors, sensors, and cameras to a Raspberry Pi and build a small robot car or robot arm.
I previously experimented with this approach by modifying a very inexpensive toy robot car and combining it with an LLM.
You can find that experiment here(Sorry Japanese only now but you can translate by LLM/VLM, I will translate and publish here soon):
https://zenn.dev/takeofuture/articles/458c1a3abcce04
Experimenting with an LLM and a Low-Cost Robot Car
For more advanced robotics projects, another very common option is ROS (Robot Operating System).
A typical architecture might look like this:
Camera / LiDAR / Sensors
↓
Raspberry Pi
/ Jetson
↓
ROS
↓
Motor Controller
↓
Robot
ROS provides a common framework for many important robotics functions, including:
- acquiring images from cameras
- reading LiDAR and other sensors
- controlling motors
- estimating robot position
- planning paths
- exchanging data between different programs and devices
For this reason, ROS is one of the most important platforms in modern robotics.
So why am I using LeRobot instead of starting with Raspberry Pi + ROS?
The Goal Is Not Just to Program the Robot
The goal of this project is not simply:
"Write a program that moves a robot."
Instead, I want to explore the following process:
Have a human demonstrate a task, collect the robot data, train an AI model using that data, and finally let the AI control the robot by itself.
This is where LeRobot becomes especially interesting.
LeRobot is an open-source framework from Hugging Face designed for robot learning.
At a high level, the workflow looks like this:
Human operates the robot
↓
Teleoperation
↓
Record camera images,
joint states, and actions
↓
Dataset
↓
Train an AI model
↓
ACT / SmolVLA / π0 / etc.
↓
The trained AI controls the robot
For example, imagine that a human demonstrates the following task dozens of times:
- Pick up the red block and put it into the box.
During each demonstration, LeRobot can record data such as:
Camera images
+
Robot joint states
+
Actions performed by the human
This becomes a training dataset.
The dataset can then be used to train an imitation-learning policy or a VLA model.
After training, the goal is for the robot to observe the camera image and perform the same task without continuous human control.
In a very simplified comparison:
ROS
Focus:
"How do I build and control the robot system?"
LeRobot
Focus:
"How do I collect robot data, train a model,and let AI control the robot?"
This is obviously an oversimplification, but I think it is a useful way to understand the difference.
ROS and LeRobot are also not competing technologies.
They can be used together.
For example:
Sensors / Camera
↓
ROS
↓
Hardware Control
↓
LeRobot
↓
Dataset / Training
↓
Robot AI
ROS could handle hardware interfaces and sensors while LeRobot manages robot-learning datasets and AI policies.
A Simple SO-101 Setup
The SO-101 is a relatively simple robot arm.
Because of that, I do not necessarily need to place Raspberry Pi or ROS between the computer and the robot.
I can start with a much simpler architecture:
Camera
↓
PC running Ubuntu
↓
LeRobot
↓ USB
Servo Controller
↓
SO-101
For this project, my main goal is not to build every part of the robot infrastructure from scratch.
Instead, I want to reach the robot AI experiments as quickly as possible.
In particular, I want to experiment with:
- teleoperation
- dataset collection
- imitation learning
- learned robot policies
- VLA models
That is why I chose LeRobot + SO-101 for this project.
SO-101 Hardware
So what do we actually need to build an SO-101?
It is possible to purchase a completely assembled SO-101, but the mechanical structure itself is relatively simple.
The frame can also be produced using a 3D printer.
In other words, you do not necessarily need to purchase a pre-made frame kit.
A basic system could be assembled from:
3D-printed frame
+
Servo motors
+
Servo controller
+
PC
This can reduce the cost even further.
I think this is another very interesting part of modern low-cost robotics.
Instead of purchasing an entire robot from a manufacturer, we can download designs, print the mechanical parts ourselves, purchase inexpensive motors and controllers, and assemble the system.
I would eventually like to try this approach with other robots or drones and build the mechanical structure myself using a 3D printer.
However, that is not the main goal of this particular project.
My priority this time is:
Build the SO-101 as quickly as possible and move on to LeRobot, imitation learning, and VLA experiments.
Therefore, I decided to purchase a pre-made SO-101 Follower Arm Frame Kit instead of printing every frame component myself.
I also purchased an Electronics Kit rather than sourcing every servo motor, cable, controller, and power supply individually.
In other words, my goal is not:
Build every component of the robot completely from scratch.
My goal is:
Reach the robot-learning and Physical AI experiments as quickly as possible.
What I Purchased
I purchased two main kits.
I currently live in the United States, so I purchased them through Amazon.com.
Similar SO-101 kits and components are also available from various sellers in other countries.
- SO-101 Follower Arm Frame Kit The frame kit contains the mechanical components required to assemble the robot arm. The kit includes items such as:
- 3D-printed frame parts
- gripper components
- camera mount
screws and assembly hardware
Amazon.com Product:
SO-101 Follower Arm Frame Kit
ASIN:B0GH429G4X
SO-101 Follower Arm Electronics Kit
The Electronics Kit contains most of the electronic components required to operate the SO-101.
The kit includes:6 × Feetech STS3215 servo motors
Waveshare Serial Bus Servo Controller
12V 5A power supply
connection cables
Amazon.com
Product:
SO-101 Follower Arm Electronics Kit
ASIN:B0GH35175P
Hardware Architecture
With these components, the basic architecture looks like this:
PC running Ubuntu
↓ USB
Waveshare
Serial Bus Servo Controller
↓
Feetech STS3215 × 6
↓
SO-101 Frame
After adding a camera, the system becomes:
Camera
↓
PC
↓
LeRobot
↓
Servo Controller
↓
SO-101
This will allow me to start experimenting with:
Teleoperation
↓
Dataset Collection
↓
Model Training
↓
AI-controlled Robot
What's Next?
The next step is to assemble the SO-101.
However, before physically assembling the robot arm, there is one important task:
assigning an ID to each servo motor.
The SO-101 uses multiple servo motors connected through the same serial bus.
Because of this, each servo must have its own unique ID so that the controller can identify and control it independently.
The next article covers this servo ID configuration process in detail:
SO-101 Servo Motor ID Configuration
After that, I plan to proceed roughly in the following order:
Servo ID Configuration
↓
SO-101 Assembly
↓
Calibration
↓
Teleoperation
↓
Dataset Collection
↓
Imitation Learning
↓
VLA / Physical AI
My final goal is not simply to build the robot arm.
I want to use this inexpensive hardware platform to explore how far an individual can go today with robot learning, VLA, and Physical AI.
In the next article, I’ll walk through how to assign an ID to each motor and prepare the SO-101 for the next steps.
👉 Part 1:
https://dev.to/takeofuture/trying-vla-part-1-preparing-lerobot-and-assigning-ids-to-the-motors-538g
Top comments (0)