DEV Community

vmodal_ai
vmodal_ai

Posted on

Building a Multi-Layer Robot Software Architecture from Sensors to AI

Building a Multi-Layer Robot Software Architecture from Sensors to AI

A modern robot combines hardware drivers, sensor processing, AI models, localization, planning, control, and user applications. Without clear boundaries, these components quickly become difficult to maintain.

This tutorial shows how to organize a robot software stack from the physical sensor layer to AI-driven applications.

The Seven-Layer Model 

Layer 7 — Applications / Mission
Layer 6 — AI Reasoning
Layer 5 — Planning
Layer 4 — World Model / Localization
Layer 3 — Perception
Layer 2 — ROS 2 Middleware / Interfaces
Layer 1 — Hardware & Drivers
Enter fullscreen mode Exit fullscreen mode

Each layer should have a clear responsibility.

Layer 1: Hardware and Drivers

Examples include:

  • RGB cameras
  • Depth cameras
  • LiDAR
  • IMU
  • GPS
  • Wheel encoders
  • Motor controllers

Drivers translate hardware-specific protocols into standardized software interfaces.

Camera SDK
    ↓
Camera Driver
    ↓
ROS 2 Image Message
Enter fullscreen mode Exit fullscreen mode

Layer 2: ROS 2 Communication

ROS 2 nodes communicate through a graph. Topics are useful for continuous streams, services for short request/response operations, and actions for long-running operations with feedback. ROS 2 documentation

Example graph:

camera_node
     ↓
/camera/image_raw
     ↓
perception_node
     ↓
/detections
     ↓
planner_node
Enter fullscreen mode Exit fullscreen mode

This decouples producers from consumers.

Layer 3: Perception

Perception converts raw sensor signals into meaningful information.

Typical components:

Image → Object Detection
Image → Segmentation
Depth → 3D Points
LiDAR → Obstacles
IMU → Motion Information
Enter fullscreen mode Exit fullscreen mode

A perception output could look conceptually like:

{
  "label": "person",
  "confidence": 0.94,
  "position": [2.1, 0.7, 0.0]
}
Enter fullscreen mode Exit fullscreen mode

Do not let application code depend directly on model-specific output. Define a stable perception interface.

Layer 4: World Model and Localization

The robot needs to combine observations into a representation of the world.

Sensors
   ↓
Sensor Fusion
   ↓
Localization
   ↓
World Model
Enter fullscreen mode Exit fullscreen mode

The world model may contain:

  • Robot pose
  • Occupancy information
  • Dynamic objects
  • Static landmarks
  • Navigation goals
  • Velocity
  • Uncertainty

This layer provides planning with structured information rather than raw sensor streams.

Layer 5: Planning

Planning answers:

What should the robot do next?

For navigation:

Current Pose + Map + Goal
             ↓
          Planner
             ↓
           Path
Enter fullscreen mode Exit fullscreen mode

Nav2 is a ROS 2 navigation framework that provides components for autonomous navigation and is designed to work across different robot configurations. Nav2

Planning can also include:

  • Task planning
  • Motion planning
  • Manipulation planning
  • Behavior trees
  • Recovery behaviors

Layer 6: AI Reasoning

AI can operate above traditional robotics algorithms.

For example:

User: "Bring the box from the storage room."

AI Reasoner
    ↓
Task decomposition
    ↓
Navigate → Find box → Pick → Return
Enter fullscreen mode Exit fullscreen mode

The AI should produce structured intentions:

{
  "task": "navigate",
  "destination": "storage_room"
}
Enter fullscreen mode Exit fullscreen mode

A deterministic task executor then validates and executes that intent.

Layer 7: Application and Mission Layer

The highest layer manages user-facing workflows.

Examples:

  • Warehouse missions
  • Delivery tasks
  • Inspection workflows
  • Human-robot interaction
  • Fleet management

A mobile dashboard could visualize:

Robot
├── Position
├── Battery
├── Mission
├── Camera
├── Detected Objects
└── Health
Enter fullscreen mode Exit fullscreen mode

This layer should not contain low-level motor control.

Define Interfaces Between Layers

A useful rule is:

Layer N should know the interface of Layer N-1,
not its implementation.
Enter fullscreen mode Exit fullscreen mode

For example:

Planner → Localization API
Enter fullscreen mode Exit fullscreen mode

rather than:

Planner → Specific GPS Driver
Enter fullscreen mode Exit fullscreen mode

This makes components replaceable.

Example Repository Structure 

robot_stack/
├── drivers/
├── interfaces/
├── perception/
├── localization/
├── world_model/
├── planning/
├── control/
├── mission/
├── ai/
├── monitoring/
└── deployment/
Enter fullscreen mode Exit fullscreen mode

Testing Strategy

Test every layer independently.

Driver Tests
     ↓
Interface Tests
     ↓
Perception Tests
     ↓
Planning Tests
     ↓
Control Tests
     ↓
Integration Tests
     ↓
Simulation
     ↓
Hardware
Enter fullscreen mode Exit fullscreen mode

For AI components, maintain recorded sensor datasets so model changes can be evaluated against identical inputs.

Performance Considerations

Measure:

  • Sensor-to-perception latency
  • Perception-to-planning latency
  • Planning duration
  • Control-loop jitter
  • Message frequency
  • CPU/GPU utilization
  • Memory usage

A robot can have highly accurate AI and still fail if decisions arrive too late.

Conclusion

Layering a robot software stack creates clear boundaries between hardware, communication, perception, planning, AI, and applications.

The most valuable benefit is replaceability. You can change a camera, AI model, planner, or user interface without rewriting the entire robot.

Useful Links

Website: www.v-modal.com
SDK Flutter: https://github.com/v-modal/vmodal_sdk_flutter
SDK Android: https://github.com/v-modal/vmodal_sdk_android
Discord: https://discord.gg/K72z28KUx

Top comments (0)