DEV Community

CAVEDU Educaion
CAVEDU Educaion

Posted on Originally published at blog.cavedu.com AI-assisted

Cosmos 3 Edge, Part 3: SO-101 Sim-to-Real and Robot Control on Jetson Thor

In the first two articles, we introduced Cosmos 3 Edge on Jetson and explored the Physical AI workflow from visual understanding to action reasoning. The remaining step is to turn an AI decision into a machine's movement: the action in perceive, reason, and act must change something in the physical environment.

NVIDIA's SO-101 Sim-to-Real Learning Path offers a useful development route, from calibration and simulation through data collection, VLA training, autonomous tests on a real robot, and Cosmos-based dataset augmentation.

One distinction is essential: the SO-101 learning path uses GR00T VLA as its main robot policy. Cosmos is used for dataset augmentation, not as a direct replacement for GR00T in that course. NVIDIA's separate Cosmos 3 Edge route demonstrates a post-trained robot policy running on Jetson AGX Thor.

This final article connects these complementary pieces of the Physical AI ecosystem.

Read the series

  1. Model capabilities and a first look at Physical AI
  2. Workflows, action reasoning, and action chunks
  3. SO-101 sim-to-real and on-device robot control

These links open the original CAVEDU articles in Traditional Chinese.

What does a robot need beyond its controller?

A conventional robot controller accepts low-level inputs such as joint positions, velocities, or other control signals and drives the motors accordingly. It does not, by itself, understand a request such as “Pick up the industrial object and put it in the yellow container.”

That apparently simple task involves several questions:

  • What counts as yellow under this lighting?
  • Which object matches the request?
  • Where is the destination?
  • What are the arm's current position and orientation?
  • From which direction should it approach?
  • When should the gripper close?
  • How should it move after grasping?
  • What if the object is not where the system expected it to be?

The NVIDIA GR00T architecture combines visual observations with language for task understanding, followed by action output for the robot.

NVIDIA GR00T architecture as illustrated in the original article

A Physical AI system connects sensors, perception, world understanding, a policy, actions, and the robot. It also needs feedback. After every action, the environment has changed. New observations must influence the next decision. This is closed-loop operation.

The SO-101 learning path builds that whole loop

Train an SO-101 Robot From Sim-to-Real With NVIDIA Isaac is more than a guide to moving an arm. It describes a workflow involving:

  1. Robot calibration.
  2. Simulation.
  3. Teleoperation and data collection.
  4. VLA policy training.
  5. Simulation evaluation.
  6. Real-robot evaluation.

The learning path also includes Cosmos augmentation and autonomous tests. Choosing a model is only one part of the challenge. Data, training, simulation, and physical evaluation must form a coherent process.

SO-101 workflow illustration from CAVEDU

At a 333 Robotics Community meetup in September 2026, the original author made a related observation about LeRobot: the robot hardware is not the whole story. Data and policy are central to the ecosystem.

CAVEDU author speaking at the September 2026 333 Robot Meetup

First, calibrate the robot's body

Physical AI draws attention to the AI model, but practical robot development often starts with calibration. Both the SO-101 follower and its teleoperation leader arm need calibration so the system knows their joint states and operating ranges.

If the model's representation of the robot does not match the physical mechanism, a more capable model will not simply erase that mismatch. The system's understanding must align with the actual robot body.

Why start in simulation?

Testing directly on a real robot can be costly. Failures can damage equipment, disrupt a test setup, or stop a production line. Isaac Sim provides an environment for teleoperation, data collection, and policy evaluation before transferring a candidate policy to the real robot.

The development sequence is simulation, data collection, policy training, evaluation, and physical deployment. Simulation helps explore and test the policy; real-world evaluation remains necessary.

Simulation and reality differ

A policy that performs well in simulation may still fail on the physical robot. The source groups sim-to-real differences into four categories:

  • Sensing gap: Real cameras introduce noise, blur, lighting variation, lens effects, and depth errors.
  • Actuation gap: Real motors and mechanisms have friction, backlash, control timing, and other hardware behavior.
  • Physics gap: Contact, collision, elasticity, and friction are complex and difficult to reproduce exactly.
  • Modeling gap: Geometry, mass, and inertia in the simulated model may differ from the real hardware.

Many small differences are difficult to represent fully. See the learning-path discussion referenced by the source.

AI-generated conceptual illustration of common sim-to-real gaps

The original article identifies this image as AI-generated.

What can Cosmos contribute?

In the SO-101 learning path, Cosmos appears as a dataset-augmentation strategy. The idea is to increase the diversity of training data rather than rely only on a limited set of demonstrations.

Simulation demonstrations, real data, and generated variations can contribute to a broader training dataset. Text-to-video and image-to-video workflows can introduce visual variation, including backgrounds, lighting, appearances, and environments.

The goal is not attractive video for its own sake. The variations must help the trained policy generalize to different conditions, while preserving the task-relevant information needed for training.

The original article also points readers to Cosmos 3 Nano on NVIDIA's model platform. Cosmos-family capabilities extend beyond visually realistic generation to relationships among observations, motion, and actions; the exact interface depends on the selected model and workflow.

Cosmos 3 Nano model page on NVIDIA NIM, as shown in the source article

The separate Cosmos 3 Edge robot-control route

Returning to Cosmos 3 Edge, NVIDIA describes another path toward edge robot control:

  1. Start from Cosmos 3 Edge.
  2. Perform robot-policy post-training.
  3. Deploy the policy on Jetson AGX Thor.
  4. Run on-device robot control.

See the model introduction and on-device robot-control post-training guide.

These are distinct roles within the ecosystem. In the SO-101 course, GR00T VLA provides the manipulation-policy route and Cosmos supplies augmentation. The separate Cosmos 3 Edge demonstration deploys a post-trained policy on Thor. Combining their names does not automatically create a supported plug-and-play integration.

Jetson AGX Thor as the onboard AI computer

In the cited on-device workflow, the policy server runs on Jetson AGX Thor. This brings policy inference close to the robot rather than requiring every observation and action decision to travel through a remote AI server.

Jetson is therefore more than a camera-processing box. It can serve as the robot's onboard AI computer, subject to the workload's memory, compute, software, and integration requirements.

Action chunks connect model timing to robot timing

As discussed in Part 2, a policy can predict a sequence of actions for the robot to execute. The system observes, predicts an action chunk, begins execution, observes again, and replans.

The key question is whether inference timing and action execution fit together. Repeatedly pausing the robot while waiting for a large model to decide on one small movement is different from generating motion segments and updating them through feedback.

Chunking does not remove the need for a controller, robot-specific action conversion, or validation. It provides a way to organize policy predictions across time.

Two routes toward the same Physical AI goal

Route Main development sequence Cosmos's role
SO-101 learning path Robot calibration, simulation, data collection, GR00T/VLA training, real-robot evaluation Dataset augmentation within the workflow
Cosmos 3 Edge robot policy Model, robot-policy post-training, Jetson AGX Thor deployment, on-device control The post-trained model supports the robot policy

Both aim to connect perception, decision-making, and physical action. A working robot still needs cameras, perception, a policy, actions, a controller, the physical mechanism, and feedback.

The SO-101 workflow tackles the broader development problem: how can a policy trained and evaluated in simulation remain useful on real hardware? Cosmos 3 Edge demonstrates a path for world/action modeling to become an edge robot policy.

“Cosmos × SO-101” is not one button

The current learning path should not be read as “Download Cosmos 3 Edge, connect an SO-101, and press Run.” The policy, data, embodiment, camera arrangement, and controller must be matched and evaluated.

Even with a model, data, Jetson, and an arm, the practical question remains: What success rate do we obtain after the real robot starts moving?

This is why real evaluation matters. Simulation measures performance in the simulated environment. Deployment on the physical robot reveals how the system handles the actual world.

Looking back at the series

Part 1 introduced how Cosmos 3 Edge helps interpret observations. Part 2 connected world understanding to action decisions. This final article places those capabilities inside a full robot-development workflow.

The destination is a continuous loop: observe the world, understand the task, choose actions, execute them, and observe the changed world again. Answering a visual question or generating an impressive video is only one part of that process.

From “it moves” to “it works reliably”

Alongside task success, the original article highlights intervention rate: how often must a human step in per 100 cycles? A system requiring frequent rescue is not yet a dependable production tool.

Completing a task once may be luck. Repeating it 10, 50, or 100 times under different lighting, object positions, and environments provides more useful evidence. Better data, more accurate simulation, stronger policies, faster edge inference, and reliable closed-loop control all matter.

Physical AI depends on a robust workflow connecting robot, simulation, data, VLA, Cosmos, and real-robot evaluation. The next topic is preparing OpenUSD assets for simulation—the bridge toward SimReady content.

Resources


English translation and editorial clarification of CAVEDU's original article. Original author: CAVEDU 阿吉. Original images and diagrams retain their source labels and may contain Traditional Chinese. The sim-to-real gap illustration is identified above as AI-generated.

Top comments (0)