In Part 1, we introduced NVIDIA Cosmos 3 Edge and its deployment on Jetson. Traditional vision AI asks, “What objects are in front of me?” Physical AI goes further: What is happening now? What could happen next? If I take an action, how will the world change?
World models aim to connect objects, space, motion, actions, and outcomes. World Action Models (WAMs) extend that relationship toward robot behavior. This article moves from “Can the model run?” to “Can it help decide what to do next?”
Read the series
- Model capabilities and a first look at Physical AI
- Workflows, action reasoning, and action chunks
- SO-101 sim-to-real and on-device robot control
These links open the original CAVEDU articles in Traditional Chinese.
From Vision AI to Physical AI
Imagine a banana, a plate, and a robot arm on a table. The original article illustrates this with an SO-101 simulation in Isaac Sim: watch the demonstration.
Object detection can return the banana, plate, and arm with their coordinates. A vision-language model can describe a banana beside a plate. A robot needs something more: how should it pick up the banana and put it on the plate?
The task becomes a continuous loop:
- Observe the world.
- Understand the scene and task.
- Predict possible outcomes.
- Choose and execute actions.
- Observe the changed world and repeat.
This feedback loop is central to Physical AI and physical agents.
Moving a world model toward actions
Cosmos 3 Edge is an approximately 4B member of NVIDIA's Cosmos 3 family. Beyond multimodal observations, its workflows include action trajectories, world understanding, future prediction, and action reasoning. See Part 1 for the model discussion.
A useful distinction is:
- A VLM describes what the world looks like now.
- A WAM relates actions to how the world may change.
“Pick up the banana and put it on the plate” sounds effortless to a person. For a robot, it involves locating both objects, choosing an approach, closing the gripper, moving above the plate, releasing the banana, and checking the result. Vision, spatial reasoning, motion, and outcomes must be connected.
What is an action?
“Move your hand to the right” is not directly executable motor control. Robot commands are numerical representations over time. Depending on the robot and control interface, they can encode translation, rotation, joint targets, or gripper state.
Conceptually, a trajectory contains a sequence of targets at successive times. The original article describes Cosmos 3 as using a shared geometric representation across embodiments, including translation, rotation, and manipulation state. The output can therefore go beyond a sentence describing an action and include an action trajectory.
The important relationship becomes what the robot sees, what it does, and what may happen next. It no longer stops at describing what it sees.
Observation to action is a closed loop
A more complete pipeline connects cameras and sensors to observations, world understanding, a robot policy, actions, and the physical robot. The next sensor observation closes the loop.
A single successful model call is not the goal. The system must repeatedly observe, predict, act, and observe again. The policy output also has to be converted into commands accepted by the controller.
Why use action chunks?
The Cosmos 3 Edge robot-policy materials describe 32 actions per inference for the referenced workflow. Rather than request a fresh large-model inference for every tiny movement, the system predicts a segment of motion and lets the controller execute it.
The Jetson AI Lab guide describes action generation using 640 × 360 observations on Jetson Thor, with repeated replanning to stay aligned with the task.
The pattern is:
- Observe the current scene.
- Predict a chunk of upcoming actions.
- Start executing those actions.
- Collect a new observation.
- Replan the next chunk.
Low-level robot control may require much faster updates than a large model can generate. Action chunks connect these different time scales. Single-step policies are not inherently impossible; the practical issue here is avoiding a slow large-model call for every low-level control update.
Why deploy at the edge?
Sending camera observations to a remote GPU introduces network latency, bandwidth requirements, and dependence on the connection. On-device inference brings the policy closer to the robot's observations and actuators.
In the official example discussed by the source, a post-trained Cosmos 3 Edge policy runs on Jetson AGX Thor. Reported action-chunk generation takes approximately 1.53 seconds, while each chunk corresponds to about 2.13 seconds of robot motion. This gives the system a window to prepare another chunk before the current motion finishes.
The reported 15 Hz is an action execution rate, not 15 full model inferences every second. Chunking and receding-horizon control make continued motion and replanning possible. These timing figures describe the cited deployment, not every Jetson or robot.
Jetson's changing role
Earlier edge-vision applications often followed a recognition-oriented pattern: camera, Jetson, detector, object label. Physical AI adds task reasoning and action generation before the robot executes its next movement.
Jetson becomes more than a device that “sees.” It can serve as an onboard platform for understanding observations and supporting decisions. Software compatibility still depends on the module and JetPack release; select the stack appropriate to the actual device.
Do not connect an arbitrary arm and expect it to work
Generating actions does not mean that downloading a model and plugging in a robot produces an autonomous worker.
NVIDIA's on-device robot-control post-training article reports a 22.9% success rate for the referenced post-trained policy on the RoboLab closed-loop tasks. The original article identifies this evaluation as 120 language-conditioned manipulation tasks. That result belongs to a particular benchmark and configuration; it is not a general success rate for all robot applications.
Policies still need task- and embodiment-specific data and post-training. The released Cosmos3-Edge-Policy-DROID uses the DROID dataset.
Changing the arm, camera placement, action space, or task can require major adaptation. A world/action model supplies foundational capabilities, not a universal robot controller. The physical world does not become cooperative simply because the prompt is impressive.
From running a model to building a useful system
The development path can be described in five levels:
- World understanding: Recognize the banana on the table.
- World prediction: Relate candidate movements to possible changes in the scene.
- Action generation: Produce a proposed motion sequence.
- Closed-loop control: Execute that motion on the robot.
- Physical feedback: Detect a failed grasp, a moved object, or another change, then observe and plan again.
The last stage is especially difficult. The real world is dynamic and full of disturbances. Even sophisticated simulation cannot reproduce every small contact, sensing, and mechanical detail.
The value is more than the parameter count
The interesting part is not simply that 4B is relatively small. It is that world understanding and action reasoning can become part of a machine's onboard workflow.
The robot observes its physical surroundings; an edge computer reasons about what it sees, predicts consequences, and supports the next action. AI becomes part of the machine rather than only a server beside it.
Yet “runs at the edge” and “reliably controls the physical world” remain different achievements. The reported RoboLab result makes that distinction concrete. Progress requires better robot data and demonstrations, stronger action prediction, faster inference, better closed-loop control, safety mechanisms, and both simulation and real-world evaluation.
What comes next?
Part 1 introduced the model's understanding capabilities. This article connected world models, actions, and edge computing in a Physical AI workflow. Part 3 takes the next step: how do these ideas relate to a real robot, its calibration, its sensors, and the sim-to-real development process?
The useful loop remains observe, understand, predict, act, and observe again.
Resources
- NVIDIA World Action Model glossary
- Cosmos 3 Edge on Jetson AI Lab
- Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models
- RoboLab
A World Action Model learns relationships between how a world may change and the actions a robot can take to influence that change. Connecting predicted states to actions can support planning and adaptation to new tasks and environments.
English translation and editorial clarification of CAVEDU's original article. Original author: CAVEDU 阿吉. Benchmark figures refer to the cited configurations. Original illustrations retain their source labels and may include Traditional Chinese.




Top comments (0)