Building an End-to-End Physical AI Software Stack with ROS 2

Physical AI systems combine robotics, perception, machine learning, planning, and real-time control. The challenge is not simply running an AI model—it is building a software stack that can reliably move information from sensors to decisions and from decisions to actuators.

This tutorial presents a practical architecture for an end-to-end Physical AI stack using ROS 2 as the communication backbone.

What We Will Build

A simplified robot pipeline looks like this:

Sensors
   ↓
Sensor Drivers
   ↓
ROS 2 Topics
   ↓
Perception
   ↓
World Model / State Estimation
   ↓
Planning
   ↓
Control
   ↓
Actuators

ROS 2 nodes communicate using topics for continuous streams, services for short request/response operations, and actions for longer-running behaviors with feedback. ROS 2 documentation

1. Define the Robot Architecture

Before writing code, divide the system into logical responsibilities:

• Hardware layer — cameras, LiDAR, IMU, encoders, motors
• Driver layer — converts hardware data into ROS messages
• Perception layer — object detection, segmentation, depth estimation
• State estimation — localization and sensor fusion
• Planning layer — global and local motion planning
• Control layer — velocity, steering, or joint commands
• Application layer — missions and high-level behaviors
• Monitoring layer — diagnostics, logging, metrics, and safety

Keeping these responsibilities separate makes the system easier to test and replace.

2. Create a ROS 2 Workspace

A typical workspace can be organized as:

physical_ai_ws/
├── src/
│   ├── robot_bringup/
│   ├── sensor_drivers/
│   ├── perception/
│   ├── localization/
│   ├── planning/
│   ├── control/
│   └── robot_interfaces/
├── build/
├── install/
└── log/

Create the workspace:

mkdir -p ~/physical_ai_ws/src
cd ~/physical_ai_ws
colcon build
source install/setup.bash

3. Connect Sensors

A camera node might publish:

/camera/image_raw
/camera/camera_info

An IMU could publish:

/imu/data

A LiDAR driver could publish:

/scan

Use standard message types where possible. This keeps components interoperable.

For continuous sensor streams, ROS 2 topics are generally the appropriate interface. ROS 2 interfaces

4. Add a Perception Node

The perception node subscribes to sensor data and produces semantic information.

For example:

/camera/image_raw
        ↓
 Object Detector
        ↓
/detections

A Python-based prototype could use OpenCV and an inference runtime:

class Detector:
    def detect(self, image):
        # Run model inference here
        return detections

The ROS node should focus on communication and orchestration rather than containing every inference detail.

5. Build a World Model

Raw detections are not enough for autonomous behavior.

Create a state representation containing:

• Robot pose
• Detected objects
• Obstacles
• Map information
• Velocity
• Sensor timestamps
• Confidence scores

For example:

WorldState
├── robot_pose
├── robot_velocity
├── obstacles[]
├── objects[]
└── map

This gives planning components a consistent view of the environment.

6. Add Navigation and Planning

For mobile robots, Nav2 provides a ROS 2 navigation framework for autonomous navigation, including planning, control, and obstacle avoidance. Nav2 documentation

A typical flow is:

Goal Pose
   ↓
Global Planner
   ↓
Path
   ↓
Local Controller
   ↓
Velocity Command
   ↓
Robot

The planner should not directly manipulate motors. Instead, it should produce a representation that the control layer can safely execute.

7. Add the Control Layer

The controller converts planned motion into actuator commands.

For a mobile robot:

/cmd_vel
   ↓
Motor Controller
   ↓
Left Motor + Right Motor

For a robotic arm, the output may instead contain joint positions, velocities, or torques.

Keep safety limits close to the actuator interface. AI-generated commands should never bypass physical constraints.

8. Use Actions for Long-Running Behaviors

A mission such as:

Navigate to loading station

is better represented as an action than as a simple service because it can provide feedback and support cancellation.

Conceptually:

Client
  ↓ goal
Navigation Action Server
  ↓ feedback
Client
  ↓
Result

ROS 2 actions are designed for long-running operations that need feedback and cancellation. ROS 2 Actions

9. Add Observability

Production robots need more than logs.

Track:

• Sensor frequency
• Message latency
• Inference latency
• CPU/GPU utilization
• Planner duration
• Controller frequency
• Dropped messages
• Localization confidence
• Safety events

ROS 2 also provides topic statistics that can help monitor message age and timing behavior.

10. Test in Simulation First

Before connecting real motors, test the complete graph in simulation.

A useful progression is:

Unit Tests
   ↓
Node Tests
   ↓
Simulation
   ↓
Hardware-in-the-Loop
   ↓
Real Robot

This makes failures cheaper and safer to diagnose.

11. Production Architecture

A production deployment might look like:

             ┌───────────────┐
             │ Mission Layer │
             └───────┬───────┘
                     ↓
             ┌───────────────┐
             │   Planning    │
             └───────┬───────┘
                     ↓
┌──────────┐   ┌───────────────┐
│ Cameras  │ → │  Perception   │
│ LiDAR    │ → │ State Est.    │
│ IMU      │ → │ World Model   │
└──────────┘   └───────┬───────┘
                       ↓
                ┌────────────┐
                │  Control   │
                └─────┬──────┘
                      ↓
                ┌────────────┐
                │ Actuators  │
                └────────────┘

Key Design Principles

1. Keep hardware interfaces replaceable.
2. Keep AI inference independent from ROS transport.
3. Use typed interfaces between components.
4. Treat timestamps as first-class data.
5. Separate planning from low-level control.
6. Put safety constraints below high-level AI.
7. Test every layer independently.
8. Design for observability from the beginning.

Conclusion

An end-to-end Physical AI system is a collection of cooperating software layers rather than a single AI model. ROS 2 provides a strong communication foundation, while perception, state estimation, planning, control, and safety form the rest of the autonomy stack.

The most important engineering decision is to define clean boundaries between these layers. Once those boundaries are stable, individual AI models, sensors, planners, and hardware components can evolve without requiring a complete rewrite.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx