DEV Community

Glen Yu
Glen Yu

Posted on

What I learned training an ACT policy for the LeRobot SO-ARM101

I’ve been learning a lot about robotics in the last little while I’m here to document my journey and share my learnings starting with Action Chunking with Transformer (ACT) policies.

HF LeRobot banner

Introduction

I have been fascinated by robotics for a long time, but for years the space felt fragmented. There were plenty of robot arms on the market, but rarely a cohesive framework or active open-source community tying hardware and modern machine learning together. That changed when I discovered the SO-ARM101 within Hugging Face’s LeRobot ecosystem. A low-cost, 3D-printed arm backed by standardized datasets, accessible imitation learning pipelines, and an active community gave me the confidence to finally take the leap into physical AI.

And what a journey it has been. From initial setup quirks to training my first Action Chunking with Transformer (ACT) policy, getting everything to run reliably taught me plenty of lessons that are not spelled out in the official documentation. Here is everything I learned along the way so you can skip the trial and error.

Setup

Development environment

I use pyenv on my MacBook and set up a miniconda3-3.12 environment as the base. The initial setup steps outlined in the official documentation were accurate and straightforward. I installed the LeRobot package from PyPI with the necessary extras for CLI scripts and Feetech bus servo control:

pip install 'lerobot[core_scripts,feetech]'
Enter fullscreen mode Exit fullscreen mode

How many cameras do you need?

When designing your physical workspace, I recommend at least two cameras: a wrist camera and an overhead or side camera.

  • Wrist camera: Provides fine-grained visual feedback on where target objects are relative to the gripper. Much like a racehorse with blinders, its field of view is narrow and focused entirely on the manipulation point.
  • Top/Overhead camera (Bird’s Eye view): Gives the model global context for where the arm and objects sit on the table. An overhead perspective avoids the blind spots created by side-mounted cameras, where the arm body can obstruct the view.
  • Front-facing base camera (Optional third): Depending on your task complexity, adding a camera at the base facing forward provides depth perspective that complements the top-down and wrist views.

I have seen basic budget setups use the stock 15 FPS wrist camera paired with a built-in MacBook webcam, but a dedicated multi-camera layout yields far better policy consistency.

Hardware Selection

  • 2 x InnoMaker 1080P USB2.0 UVC camera: A drop-in upgrade for the stock 15 FPS wrist camera that came with the arm. Its compact form factor and optional aluminum case also made it easy to mount at the arm base as a front-facing camera.
  • Pro HD 1080P/30FPS webcam with tabletop tripod: Used for the bird’s eye view. Placing it on a nearby shelf with the tripod made it easy to dial in the height and tilt angle without buying specialized clamp mounts

IMPORTANT: I purchased 1920x1080 resolution cameras, but while the detail is nice, it does make your dataset massive. Much like training traditional ML models for object detection, the data doesn’t have to be in high resolution (that’s for humans). Instead, configure the cameras to operate at a lower resolution such as 640x480 during training and model rollout. You still capture all the important details and metrics. This is drastically speed up training and lower memory usage.

Dataset recording and thread tuning

When recording episodes, allocate at least two encoder threads per camera (dataset.encoder_threads=2) to avoid dropping frames during teleoperation. For my 3-camera setup, I assigned 3 threads per camera, utilizing 9 CPU cores on my MacBook M3 Pro.

While running cameras at 15 FPS reduces dataset size and CPU overhead, modern USB cameras natively default to 30 FPS. Standardizing on 30 FPS across all sensors avoids framerate mismatch issues during episode logging.

Downscaling resolution for faster training

While 1080p cameras provide crisp video for human eyes, high-resolution feeds create bloated datasets, spike VRAM usage, and slow training to a crawl. Standard computer vision backbones in ACT do not need full HD detail to learn spatial features. Resize your camera feeds to lower resolutions like 640x480 for both training and policy evaluation. This will keeping storage requirements low and training fast.

Training strategy

For your first ACT policy, eliminate as much physical and visual variability as possible. Use a single-color Lego brick (or build a slightly larger block using bricks of the same color) and place it against a clean, uniform background. For the target container, choose a simple box or bowl with a matte, single color. Highly reflective, translucent, or patterned objects add unnecessary visual noise that makes early policy convergence much harder.

NOTE: Before settling on a Lego brick, I initially tested with a cube-shaped USB wall charger. This turned out to be a poor choice. Depending on how it rested on the table, the prongs and varying face textures created significant visual variability from different angles. That extra visual noise impacted the gripper’s consistency in gripping the block.

How much training is needed?

A common question when moving from classical deep learning to imitation learning is how training steps map to traditional epochs. In a 3-camera setup, 100 recorded episodes typically yield around 45,000 rows of Parquet data containing synchronized camera images and servo motor joint states (each row of data corresponds to a single frame of video). If you train for 20,000 steps with a batch size of 32, the relationship is:

Epochs = Total Steps x Batch Size / Dataset Size

Plugging in those numbers:

Epochs = 20000 x 32 / 45000 = 14.2

An epoch represents one full pass through your training data. In standard PyTorch or TensorFlow workflows, 14 epochs usually still falls a bit short for a vision-based policy to generalize; 25 to 50 epochs is a more reliable baseline for basic convergence.

Task complexity vs. dataset size

Epoch count is only one metric. Training for hundreds of epochs will not improve policy quality if your underlying dataset lacks sufficient coverage. You want a dataset with broad enough coverage to match the complexity of your task, and then pair it with enough training steps to allow the model to converge:

  • Fixed start and fixed target (Low variance): If the arm picks up a block from a static Point A and drops it into a container at a static Point B, a modest dataset of 30 episodes trained for 15 epochs can yield a functional baseline policy.
  • Randomized start and fixed target (Medium variance): If the Lego block can appear anywhere within the workspace while the container stays fixed, you will likely need around 100 episodes trained for 30 epochs to properly cover the workspace distribution.
  • Randomized start and randomized target (High variance): If both the block and the container change positions across every demonstration, you should aim for 250+ episodes trained for ~50+ epochs so the Transformer encoder can learn generalized spatial relationships rather than memorized trajectory paths.

Interpreting L1 Loss

When monitoring your training curves, ‭L1 Loss serves as your primary convergence indicator. While there is no universal cutoff for success, ACT policies for manipulation tasks typically begin showing reliable trajectories when the ‭L1 Loss falls between 0.01 and 0.05.

What does an ‭L1 Loss of 0.02 actually mean in practice? It represents the mean absolute error between the model’s predicted trajectory chunk and the ground-truth teleoperated actions across normalized joint space. Keep in mind that ‭L1 Loss is an aggregated average across all degrees of freedom. On a 6-DOF arm, five joints might track with near-zero error while a single critical joint (such as the wrist pitch or gripper closing action) exhibits higher error, yet the overall average can still look deceptively low at 0.02. Ultimately, running real-world policy rollouts will always be the true test of performance, and seeing your arm execute the task autonomously is easily the most rewarding part.

My approach

My goal was to train an ACT policy capable of picking up a Lego block and placing it inside a bowl, where both the block and the bowl could be placed at arbitrary positions across the workspace. I set a target success rate of 75% or higher.

I recorded all demonstration episodes locally on a makeshift workbench. Because the LeRobot pipeline integrates seamlessly with the Hugging Face Hub, I pushed the datasets upstream and ran the resource-heavy training jobs on an NVIDIA DGX Spark at the office. This allowed me to train with larger batch sizes and higher step counts far faster than running on my MacBook.

I started with a medium-variance baseline: 50 episodes (~21.5k frames) where the Lego block was placed in different locations around the workspace while the destination bowl remained fixed in one spot.

  • Training configuration: 20,000 steps, batch size of 32 (~30 epochs).
  • Compute: Training took roughly 8 hours. A batch size of 32 consumed approximately 22GB of VRAM.
  • Result: (50 episodes for 20k training steps, L1 Loss = 0.074)

With the single-target baseline proven, I introduced full spatial variance. I recorded 150 new episodes where the positions of both the Lego block and the bowl changed with every demonstration, then merged them with the initial dataset for a total of 200 episodes (~83.5k frames).

  • Training configuration: 100,000 steps, batch size of 32 (~38 epochs).
  • Compute: Training time was roughly 41 hours.
  • Result: (200 episodes for 100k training steps, L1 Loss = 0.052)

While the v1 model performed well overall, I wanted to push reliability higher. I recorded an additional 125 episodes specifically covering awkward placements, tight table boundaries, and difficult angles. Merging these gave a final dataset of 325 episodes (~141k frames).

  • Training configuration: 250,000 steps, batch size of 32 (~57 epochs).
  • Compute: Total training time ran for approximately 104 hours.
  • Result: (325 episodes for 250k training steps, L1 Loss = 0.040)

Training for 104 hours might sound like overkill, but leveraging the DGX Spark compute allowed the Transformer to generalize smoothly across tricky spatial boundaries and hit the 75%+ success target consistently.

What’s next?

Working through the “hello world” of robotic manipulation with the SO-ARM101 was a lot of fun, but it is just the starting point. Next, I plan to tackle more complex, deformable object manipulation tasks, such as folding clothes with an ACT policy.

Beyond pure imitation learning, I am also excited to experiment with Vision-Language-Action (VLA) models such as Hugging Face’s SmolVLA, Physical Intelligence’s Pi, and NVIDIA’s GR00T to see how generalist multimodal backbones handle zero-shot spatial reasoning and natural language task prompting.

(And of course, I plan to share my learnings with these tools as well)

What I would love to see

From a tooling perspective, native macOS support for simulation environments like NVIDIA Isaac Sim would be a huge quality-of-life improvement for Apple Silicon developers. Currently, Isaac Sim remains restricted to Linux and Windows with dedicated NVIDIA RTX hardware.

In an ideal setup, I would love to have a dedicated local AI workstation at home, powered by an NVIDIA DGX Spark or an AMD Ryzen AI Halo. Having native Linux and serious local compute would let me run GPU-accelerated physics simulations in Isaac Sim and train large physical AI policies right from my home workbench.

If you found this post helpful or are currently troubleshooting your own LeRobot build, feel free to connect or share your setup in the comments below.

Top comments (0)