STATERA Library · Series 08 · Volume 12 · Chapter 02 · EN
Source: https://github.com/Statera-Library/08_Vol_12_Autonomous_Driving_Software/blob/main/Chapter_02_AV_Software_Architecture/Chapter_02_AV_Software_Architecture_EN.md
Volume 12. Autonomous Driving Software
Chapter 02. AV Software Architecture
02.01. AV SW Stack Layered Architecture Overview
An autonomous vehicle software stack is commonly organized as a layered architecture in which each layer transforms information into progressively higher-level representations and actions. Raw physical signals enter through sensors, pass through hardware interfaces and processing modules, become an interpreted representation of the environment, and finally drive planning and control decisions. This decomposition allows complex autonomy software to be developed, tested, replaced, and validated as manageable subsystems.
At the lowest level, the hardware and device-interface layer connects computing platforms with cameras, LiDARs, radars, GNSS receivers, IMUs, wheel encoders, vehicle networks, and actuators. Device drivers convert hardware-specific protocols into standardized software interfaces while preserving timestamps, calibration parameters, diagnostic states, and sensor metadata. Hardware abstraction reduces dependencies between autonomy algorithms and particular sensor or vehicle configurations.
Above the device interfaces, the sensor-processing layer converts raw measurements into representations suitable for perception and localization. Camera pipelines may perform decoding, distortion correction, rectification, exposure handling, and image synchronization, while LiDAR pipelines perform point-cloud decoding, filtering, motion compensation, and coordinate transformation. Radar, GNSS, and inertial data undergo corresponding preprocessing before being distributed to downstream modules.
Time synchronization and coordinate management form important cross-layer services rather than isolated algorithms. Autonomous systems combine measurements generated at different frequencies, transported with different delays, and expressed in different coordinate frames. Accurate timestamps, clock synchronization, calibration, transformation trees, and latency tracking therefore determine whether apparently correct sensor measurements describe the same physical state when they are fused.
The localization layer estimates the vehicle\'s position, orientation, velocity, and motion state relative to a local or global reference. GNSS, RTK, IMU, wheel odometry, LiDAR matching, visual odometry, and map information may be combined according to the operational environment. The resulting vehicle-state estimate provides a common spatial reference for perception, mapping, prediction, planning, and control, making localization one of the central dependencies of the autonomy stack.
Perception transforms processed sensor data into an understanding of the surrounding environment. Typical outputs include detected and tracked objects, free space, drivable areas, road boundaries, terrain properties, semantic classes, occupancy representations, and obstacle geometry. Modern stacks increasingly combine camera, LiDAR, and radar information to improve coverage and robustness rather than relying on a single sensing modality, particularly under difficult environmental conditions.
A world representation layer organizes localization, perception, map, and vehicle-state information into a coherent model consumed by decision-making modules. Depending on the architecture, this representation may include object lists, occupancy grids, semantic maps, lane structures, terrain maps, or learned latent features. Separating world representation from individual perception algorithms helps downstream components operate against stable interfaces even when sensing algorithms evolve.
Prediction extends perception from understanding the current scene to estimating how relevant dynamic entities may evolve. It can estimate future positions, velocities, trajectories, intentions, or multiple possible behaviors of pedestrians, vehicles, bicycles, and other agents. Prediction uncertainty is especially important because planning should not treat every estimated future state as deterministic. The planning system must reason about plausible alternatives while maintaining safety margins.
Behavior planning determines what the autonomous system should do at a semantic level. Decisions may include following a route, yielding, stopping, overtaking, avoiding an obstacle, entering an intersection, approaching a docking location, or transitioning into a minimal-risk condition. Behavior logic may use finite-state machines, rule systems, optimization, probabilistic reasoning, machine learning, or hybrid methods while remaining constrained by operational and safety requirements.
Motion planning converts behavioral intent into a geometrically and dynamically feasible path or trajectory. It considers vehicle dimensions, kinematic and dynamic constraints, obstacle geometry, predicted agent motion, terrain conditions, speed limits, comfort, and safety margins. In outdoor AMRs, the planner may additionally account for uneven terrain, slopes, curbs, narrow passages, surface traversability, wheel slip, and restricted maneuvering areas that differ from conventional road-driving assumptions.
The control layer transforms a planned trajectory into commands that the physical platform can execute. Longitudinal control regulates velocity, acceleration, braking, or drive torque, while lateral control regulates steering or differential wheel motion depending on vehicle configuration. Feedback from odometry, steering, IMU, and actuator states closes the loop. Control typically operates at higher frequencies and tighter timing constraints than perception or behavioral planning.
A vehicle-interface layer bridges autonomy software and the drive-by-wire or robot motion system. It translates normalized commands such as target velocity, steering angle, acceleration, braking demand, or wheel torque into platform-specific CAN, Ethernet, serial, or controller messages. In the opposite direction, it publishes actuator states, wheel speeds, fault codes, battery information, emergency-stop status, and other feedback required by autonomy and safety software.
Middleware provides the communication fabric connecting these layers. Publish-subscribe messaging, service calls, shared memory, data serialization, discovery, quality-of-service policies, and process lifecycle management allow independently developed modules to operate as one distributed system. The attached structure treats middleware selection, real-time separation, sensor-to-actuation data flow, and hardware abstraction as dedicated architectural concerns within the AV software architecture chapter.
Safety functions should span the entire stack instead of being implemented as a single final layer. Runtime monitors can check sensor health, localization confidence, perception consistency, trajectory feasibility, communication status, actuator responses, and computing resources. Watchdogs, redundant sensing, fallback planners, emergency braking, safety cages, and minimal-risk behaviors provide independent mechanisms capable of restricting or overriding nominal autonomy when predefined safety conditions are violated.
System management and observability are similarly cross-cutting capabilities. Logging, tracing, diagnostics, configuration management, health monitoring, resource supervision, software versioning, and performance measurement allow engineers to understand the behavior of a distributed autonomy system. Recording timestamps and intermediate outputs across the processing chain is particularly valuable because failures observed at the actuator may originate much earlier in sensing, synchronization, localization, or planning.
Layered architecture does not imply that every module must execute sequentially. Production autonomy systems contain parallel pipelines, asynchronous data streams, feedback paths, shared services, GPU workloads, and real-time control loops operating at different frequencies. Camera perception may execute at tens of frames per second while low-level control operates much faster. Architecture must therefore define not only functional boundaries but also timing, scheduling, concurrency, and data ownership.
Real-time and non-real-time workloads should be separated according to their timing consequences. Vehicle control, emergency response, critical state estimation, and selected safety functions may require deterministic or bounded execution, whereas map management, logging, visualization, fleet communication, and some AI processing can tolerate greater latency variation. Explicit separation prevents computationally expensive perception or background services from interfering with time-critical vehicle functions.
Well-defined interfaces are therefore as important as individual algorithms. Every interface should specify message semantics, coordinate frames, units, timestamp conventions, update rates, validity states, uncertainty, timeout behavior, and failure handling. A perception message without a clear timestamp or coordinate frame can be more dangerous than a missing message because downstream software may interpret valid numerical values within the wrong temporal or spatial context.
The layered approach also supports portability across vehicle platforms. If algorithms depend on abstract vehicle-state, sensor, and actuator interfaces rather than vendor-specific hardware details, the same perception and planning software can be deployed on multiple AMR sizes or vehicle configurations. Platform-specific differences can remain within drivers, calibration packages, vehicle interfaces, and configuration data, reducing the amount of autonomy software that must be rewritten for each platform.
Deployment architecture must map these logical layers onto actual computing resources. Sensor processing and deep neural networks may execute on GPUs or dedicated accelerators, planning may execute on CPU cores, and safety-critical control may run on isolated processors or controllers. Communication between computing domains introduces additional latency and failure modes, so software architecture and hardware architecture must be designed together rather than treated as independent engineering activities.
A mature AV stack is ultimately a closed-loop cyber-physical system rather than a simple chain from sensors to actuators. Sensors observe the environment and vehicle, software constructs state and world representations, planning selects future motion, controllers generate commands, and physical motion changes the next sensor observations. Continuous feedback makes latency, uncertainty, synchronization, and failure propagation fundamental architectural properties rather than implementation details.
For outdoor AMRs, the classical AV layering remains useful but must be adapted to lower-speed yet highly variable operational environments. The software may need to handle mixed pedestrian spaces, unstructured roads, construction areas, slopes, rough surfaces, GNSS degradation, changing payloads, and docking or mission workflows. This explains why the broader volume separates architecture from sensor processing, drivable-area estimation, detection, behavior planning, trajectory generation, control, safety, localization, and SOTIF validation.
The most effective layered architecture therefore combines modular functional decomposition with explicit cross-layer contracts. Sensors, localization, perception, prediction, planning, control, and vehicle interfaces provide the main autonomy flow, while middleware, timing, safety, diagnostics, data management, and hardware abstraction support the entire stack. This structure creates a foundation on which individual algorithms can evolve without losing system-level traceability, deterministic behavior, testability, portability, and operational safety.
02.02. Autoware Universe Architecture and Module Map [w/Code]
Autoware Universe is organized as a modular autonomous-driving software platform in which perception, localization, planning, control, vehicle interface, and system-management functions are separated into cooperating software components. Its architecture follows the general autonomous-driving software-stack concept, but provides a practical implementation structure in which individual functions can be developed, tested, configured, and replaced without redesigning the entire system. This modularity is particularly important for adapting the stack to different vehicles, sensors, operational environments, and autonomy requirements.
At the foundation of the architecture, ROS 2 provides the communication and execution framework connecting autonomous-driving modules. Nodes or composable components exchange structured messages through publish-subscribe communication, services, and actions. Topics carry continuously changing information such as sensor measurements, localization states, detected objects, trajectories, and vehicle states. The middleware therefore acts as the communication backbone that allows independently developed modules to operate as one distributed autonomy system.
The sensing portion of the module map receives raw information from cameras, LiDARs, radars, GNSS receivers, IMUs, and vehicle sensors. Sensor drivers and preprocessing components convert hardware-specific data into standardized representations that downstream modules can consume. Typical processing includes point-cloud filtering, image preprocessing, coordinate transformation, synchronization, and sensor-specific feature extraction. The separation between hardware drivers and higher-level autonomy functions allows different sensor configurations to be integrated without rewriting the complete software stack.
Localization modules estimate the vehicle\'s pose and motion relative to a map or reference coordinate system. Autoware-based systems can combine GNSS, IMU, wheel odometry, LiDAR matching, and other available measurements depending on the deployment environment. Localization outputs provide the spatial reference required by perception and planning. The architecture must therefore maintain consistent coordinate frames, timestamps, covariance or confidence information, and state validity so that downstream modules can determine whether the estimated vehicle state is suitable for autonomous operation.
The perception section transforms sensor observations into structured environmental information. Its module map can contain object detection, object tracking, traffic-related recognition, point-cloud processing, occupancy or free-space estimation, and other scene-understanding functions. Camera and LiDAR perception may operate independently or contribute to sensor-fusion pipelines. The resulting objects and environmental representations are published through defined interfaces so that prediction and planning modules do not need to know the internal implementation details of each perception algorithm.
Planning is generally divided into multiple levels because an autonomous vehicle must first determine an appropriate behavior and then generate a physically executable motion. Mission or route planning establishes the intended route, while behavior and scenario-level planning determine decisions such as stopping, yielding, following, avoiding obstacles, or changing lanes where applicable. Motion planning then generates a trajectory that satisfies vehicle constraints and environmental conditions. This separation prevents high-level decisions from becoming unnecessarily coupled to low-level trajectory generation.
Trajectory planning connects behavioral decisions with vehicle control. A trajectory normally contains spatial and temporal information such as position, orientation, velocity, acceleration, and sometimes curvature or additional dynamic constraints. The planner evaluates collision risk, vehicle dimensions, road or terrain constraints, and trajectory feasibility before publishing a candidate motion. The control system then attempts to follow this trajectory rather than directly interpreting high-level behavioral commands, creating a clear boundary between planning and physical execution.
The control portion converts the planned trajectory into commands appropriate for the vehicle platform. Longitudinal control manages velocity, acceleration, and braking-related behavior, while lateral control manages steering or equivalent directional motion. Controllers use vehicle-state feedback to compensate for disturbances, modeling errors, actuator delay, and trajectory-tracking errors. Because control is timing-sensitive, its execution requirements differ from computationally intensive perception or map-processing functions, making scheduling, latency, and deterministic communication important architectural considerations.
The vehicle interface forms the boundary between Autoware software and the actual vehicle control system. It translates abstract autonomy commands into platform-specific signals and receives feedback such as vehicle velocity, steering state, acceleration, gear state, actuator status, and diagnostic information. A properly designed vehicle interface isolates hardware and vehicle-specific implementation from the rest of the autonomy stack. This allows a common planning and perception architecture to be reused across different vehicle platforms while replacing only the necessary interface and configuration components.
System-management modules coordinate the operational state of the autonomous-driving system. They can manage module activation, lifecycle states, health information, operational modes, diagnostics, and transitions between autonomous and fallback conditions. A complete autonomy stack must know not only what each module calculates but also whether the module is alive, producing valid data, and operating within its expected timing and quality limits. System management therefore provides an important supervisory layer above individual functional modules.
The module map should also be understood as a data-flow architecture rather than merely a collection of software packages. Sensor data moves toward perception and localization, these modules produce environmental and vehicle-state representations, planning consumes those representations to generate a trajectory, and control converts the trajectory into vehicle commands. Feedback then returns through vehicle-state measurements and updated environmental observations. This closed-loop flow means that message definitions, timestamps, coordinate frames, update frequencies, quality-of-service settings, and failure states are fundamental parts of the architecture.
For practical customization, Autoware Universe allows a deployment to select and configure the modules appropriate for its operational design domain rather than treating every available capability as mandatory. An outdoor AMR, for example, may require LiDAR-based localization, terrain-aware perception, drivable-area estimation, obstacle tracking, and low-speed trajectory control while not requiring every road-vehicle function. The architecture can therefore be interpreted as a reusable autonomy framework whose modules are assembled according to vehicle characteristics, sensor configuration, mission requirements, and environmental constraints.
The most important architectural concept is the separation of responsibilities combined with well-defined interfaces. Perception should produce environmental information, localization should provide vehicle state, planning should determine feasible future motion, control should execute that motion, and the vehicle interface should translate software commands into physical actuation. ROS 2 communication connects these responsibilities while system management and safety-related mechanisms supervise their operation. This structure makes Autoware Universe useful not simply as a collection of algorithms, but as a reference architecture for constructing and customizing complete autonomous-driving systems.
02.03. Apollo AD Platform Architecture Overview [w/Code]
Apollo is an open-source autonomous-driving platform developed around a modular software architecture for integrating perception, localization, prediction, planning, control, routing, and vehicle interfaces. Its architecture is designed to connect heterogeneous sensors and vehicle platforms with a coordinated autonomy pipeline. Within the Volume 12 structure, Apollo is positioned after the general AV software-stack architecture and alongside Autoware Universe, providing a platform-level reference for understanding how a large autonomous-driving software system organizes functional modules and their interactions.
A central architectural characteristic of Apollo is the separation of autonomous-driving functions into relatively independent modules connected through defined communication interfaces. Sensor data enters through hardware and sensor drivers, is processed into perception and localization information, and is subsequently consumed by prediction and planning components. Control modules transform the resulting trajectory into vehicle commands, while vehicle-interface components connect those commands to the physical platform. This separation allows individual modules to evolve without requiring the complete autonomy system to be implemented as one monolithic application.
The sensor and hardware layer provides the connection between physical devices and Apollo\'s software environment. Cameras, LiDARs, radars, GNSS receivers, IMUs, and vehicle-state sensors produce measurements that must be decoded, timestamped, calibrated, and transformed before higher-level algorithms can consume them. Hardware-specific differences should remain concentrated near this boundary whenever possible. Such abstraction allows the same perception or planning concepts to be reused when sensors, computing platforms, or vehicle hardware are changed.
Localization provides the spatial and temporal reference required by the rest of the autonomous-driving system. Apollo-style architectures can combine positioning sources such as GNSS, inertial measurements, odometry, and map-based localization to estimate vehicle pose and motion. The localization result must be associated with consistent coordinate frames, timestamps, and uncertainty information. Accurate localization is particularly important because perception objects, map elements, predicted trajectories, and planned paths must all be interpreted within a common spatial reference.
The perception layer converts sensor observations into structured descriptions of the surrounding environment. Depending on the configuration, this may include camera-based recognition, LiDAR-based object detection, traffic-related perception, lane or road understanding, obstacle estimation, and sensor fusion. The objective is not simply to identify individual objects but to generate information that can support downstream decision-making. Perception outputs therefore need stable semantics, coordinate conventions, timestamps, confidence values, and validity states so that prediction and planning can interpret them consistently.
Prediction operates between perception and planning by estimating how relevant dynamic objects may evolve. Current observations provide information about where an object is and how it is moving, while prediction attempts to estimate possible future states or trajectories. Multiple hypotheses may be necessary because human drivers, pedestrians, and other agents do not behave deterministically. Planning can then use these predicted possibilities to evaluate collision risk, maneuver feasibility, and appropriate safety margins instead of relying exclusively on the current scene state.
Planning transforms environmental understanding into an intended vehicle motion. A large-scale autonomous-driving architecture generally separates routing or mission-level planning from behavior and motion planning. Routing determines where the vehicle should travel, behavior planning determines what maneuver or action should be performed, and motion planning generates a geometrically and dynamically feasible trajectory. This decomposition allows high-level decisions to remain distinct from the numerical generation of vehicle motion.
The planning system must continuously integrate information from localization, perception, prediction, maps, vehicle state, and operational constraints. A planned trajectory may contain position, orientation, velocity, acceleration, curvature, and timing information. The planner must evaluate whether the trajectory is collision-free and executable by the vehicle. When the environment changes, new objects appear, localization confidence decreases, or the current trajectory becomes infeasible, the planning pipeline must be capable of updating its decision and generating an alternative motion.
The control layer provides the connection between planned motion and physical vehicle behavior. It receives the selected trajectory and vehicle-state feedback and generates steering, acceleration, braking, or other motion commands. Longitudinal and lateral control can be implemented as separate control functions while sharing common vehicle-state information. Control operates under tighter timing constraints than many high-level autonomy functions, so computational scheduling, communication latency, actuator delay, and deterministic execution become important architectural concerns.
The vehicle interface isolates Apollo\'s autonomy logic from platform-specific actuation mechanisms. A vehicle may expose steering, throttle, brake, gear, wheel-speed, chassis-state, and diagnostic interfaces through CAN, Ethernet, or other communication mechanisms. The vehicle interface converts abstract software commands into the signals required by the actual platform and publishes feedback back to the autonomy stack. This boundary is essential when adapting an autonomous-driving platform to different vehicle architectures because the higher-level planning and perception modules should not depend directly on individual actuator implementations.
Communication middleware forms the connective structure between functional modules. Autonomous-driving applications exchange large quantities of sensor data, perception results, localization states, predicted objects, trajectories, and vehicle feedback. The communication architecture must therefore support appropriate message serialization, publish-subscribe behavior, data delivery policies, synchronization, and process management. Middleware design also affects latency, determinism, bandwidth consumption, fault propagation, and the ability to distribute workloads across multiple computing processes or machines.
Apollo\'s architecture should be considered as a distributed closed-loop system rather than a simple one-way pipeline. Sensor observations produce updated vehicle and environmental states, those states influence prediction and planning, planning produces a trajectory, control executes the trajectory, and vehicle motion generates new sensor observations. Errors or delays at one stage can therefore propagate into downstream decisions. Architectural interfaces must consequently define not only normal data values but also timestamps, validity, uncertainty, timeout behavior, and failure states.
A practical Apollo deployment also requires explicit management of computational resources. Perception workloads can be computationally intensive and may benefit from GPU acceleration, while planning and control require predictable CPU execution and communication timing. Logging, visualization, diagnostics, map processing, and other supporting functions can operate with different timing requirements. Separating workloads according to their computational and temporal characteristics helps prevent resource contention from degrading safety-relevant functions.
System management and operational supervision provide another important architectural dimension. A complete autonomous-driving platform must coordinate module startup, shutdown, configuration, health monitoring, diagnostics, and operational-mode transitions. It must also detect when a module stops producing valid information or exceeds an expected latency boundary. These mechanisms allow the autonomy system to distinguish between normal operation, degraded operation, and conditions requiring a fallback or controlled intervention.
For an outdoor AMR, the Apollo architectural concepts can be reused while adapting the functional scope to a different operational domain. An AMR may operate at lower speeds than a passenger vehicle but encounter unstructured roads, pedestrian areas, construction zones, rough terrain, restricted spaces, changing payloads, and GNSS degradation. Consequently, Apollo-inspired modularity can be retained while emphasizing terrain perception, traversability, obstacle avoidance, low-speed trajectory generation, docking behavior, and mission-oriented autonomy where appropriate.
The comparison with Autoware is useful because both architectures demonstrate how complex autonomous-driving functions can be decomposed into reusable software modules, although their implementation structures and ecosystem assumptions differ. The purpose of examining Apollo in this chapter is therefore not to treat one platform as universally applicable, but to understand how a large autonomous-driving platform organizes functional responsibilities, communication, data flow, and vehicle integration. This architectural understanding can then inform the design of a customized AV or outdoor-AMR software stack.
For system architects, the most important lesson is that Apollo should be understood at the level of interfaces and responsibilities rather than merely as a collection of individual algorithms. Sensor processing, localization, perception, prediction, planning, control, and vehicle integration must form a coherent closed loop. Middleware, configuration, diagnostics, timing, resource management, and safety supervision must support that loop without creating hidden dependencies. This architectural viewpoint makes it possible to reuse selected concepts while designing a platform appropriate for the specific vehicle, sensors, computing hardware, and operational design domain.
02.04. Custom AV SW Architecture for Outdoor AMR [w/Code]
A custom autonomous-vehicle software architecture for an outdoor AMR should begin with the operational characteristics of the robot rather than directly copying a passenger-vehicle AV stack. Outdoor AMRs may operate on roads, sidewalks, industrial yards, construction areas, campuses, ports, and mixed pedestrian spaces. Their software must therefore combine autonomous navigation with terrain awareness, mission execution, docking, obstacle avoidance, and vehicle-specific motion control. Within the Volume 12 structure, this topic follows the general AV architecture and the Autoware and Apollo platform examples, providing a basis for designing a customized architecture.
The architecture should maintain a clear separation between hardware-dependent functions and autonomy intelligence. Cameras, LiDARs, GNSS RTK, IMUs, wheel encoders, ultrasonic sensors, radar, and vehicle controllers may differ between AMR platforms. Device drivers, calibration, hardware abstraction, and vehicle interfaces should isolate these differences from perception, localization, planning, and AI modules. This allows the same autonomy software to operate across multiple robot configurations while platform-specific information remains concentrated in well-defined interface layers.
The sensor-processing layer converts heterogeneous sensor outputs into synchronized and geometrically consistent information. Camera images require decoding, calibration, distortion correction, and synchronization, while LiDAR data may require filtering, ground segmentation, motion compensation, and coordinate transformation. GNSS and IMU measurements must be time-aligned with other sensors, and wheel or actuator feedback must be associated with accurate timestamps. For outdoor AMRs, sensor preprocessing should also account for vibration, dust, rain, changing illumination, and mechanical mounting differences.
Localization should be designed as a multi-source estimation system because outdoor AMRs cannot always depend on a single positioning technology. GNSS RTK can provide highly accurate global positioning in open environments, while LiDAR matching, IMU integration, wheel odometry, and map-based localization can maintain local motion estimation when GNSS quality decreases. The architecture should expose position, orientation, velocity, covariance or confidence, coordinate frames, and validity states through a stable interface so that planning and control can react appropriately to degraded localization.
Perception for an outdoor AMR should extend beyond conventional road-vehicle object detection. The system may need to recognize pedestrians, vehicles, obstacles, curbs, slopes, vegetation, road boundaries, terrain structures, free space, and traversable surfaces. Multi-sensor fusion can combine complementary information from cameras, LiDAR, radar, and ultrasonic sensors. The output should be expressed in a common coordinate system and should include confidence and temporal information so that downstream modules can distinguish reliable observations from uncertain or outdated information.
A dedicated drivable-area and traversability representation is particularly important for outdoor AMRs. A passenger vehicle often assumes that road geometry provides a relatively strong constraint, whereas an AMR may encounter unstructured surfaces and areas where the concept of a conventional lane is not meaningful. Traversability estimation should consider slope, roughness, obstacle height, surface condition, vehicle dimensions, wheel configuration, clearance, and expected wheel slip. The resulting representation can guide both global navigation and local motion planning.
The world model should integrate localization, perception, maps, terrain information, vehicle state, and mission context into a consistent representation. Depending on the application, this may include occupancy grids, point-cloud maps, semantic maps, object lists, terrain maps, and local cost representations. The world model should not become an uncontrolled data repository; its interfaces should define ownership, update rates, timestamps, coordinate frames, and validity conditions. This makes the information predictable for planning and reduces hidden dependencies between modules.
Mission management is another important difference between an outdoor AMR and a conventional passenger AV. An AMR may be assigned missions such as patrol, inspection, transportation, scanning, docking, charging, or repeated route execution. A mission manager should convert these high-level objectives into operational states and navigation goals. It should also coordinate task completion, interruption, charging requirements, manual intervention, recovery behavior, and return-to-base operations without embedding these responsibilities directly into low-level motion algorithms.
Behavior planning should connect mission objectives with environmental conditions. The AMR may need to decide whether to continue along a route, stop for a pedestrian, avoid an obstacle, wait for a blocked passage, reroute around an inaccessible region, approach a docking station, or enter a minimal-risk condition. A hybrid architecture combining deterministic rules with learned components can provide flexibility while keeping safety-critical decisions constrained by explicit policies. The behavior layer should produce understandable intentions that can be evaluated independently from trajectory generation.
Motion planning converts those intentions into executable trajectories while respecting the physical characteristics of the AMR. Vehicle length, width, wheelbase, steering geometry, turning radius, maximum acceleration, braking capability, payload, ground clearance, suspension behavior, and terrain constraints must be included in feasibility checks. For high-payload or off-road platforms, the planner may also need to consider wheel slip, uneven ground, slopes, narrow passages, and stability limitations. The trajectory should therefore represent more than a geometric path when dynamic execution constraints are significant.
The control architecture should remain closely connected to the actual vehicle dynamics. Lateral motion may use steering control or differential-drive control depending on the platform, while longitudinal motion may regulate velocity, acceleration, braking, or wheel torque. Feedback from wheel encoders, IMU, steering sensors, and vehicle-state interfaces closes the control loop. Because control requires predictable timing, real-time execution should be separated from computationally variable workloads such as large neural-network inference, map processing, visualization, and logging.
The vehicle interface should provide a stable boundary between the customized autonomy stack and the AMR\'s embedded controllers. Commands such as target velocity, steering, braking, drive torque, or wheel commands should be converted into platform-specific CAN, Ethernet, or other control messages. Feedback should include actuator state, vehicle velocity, battery state, faults, emergency-stop status, and other diagnostic information. This separation makes it possible to reuse the autonomy architecture when the underlying motor controller, drive system, or vehicle platform changes.
ROS 2 or another suitable middleware can connect the functional modules, but the middleware should not define the autonomy architecture itself. The architecture should first establish data ownership and functional responsibilities, then select communication mechanisms that satisfy latency, reliability, bandwidth, quality-of-service, and real-time requirements. High-rate sensor streams, perception results, trajectories, vehicle states, diagnostics, and mission events may require different communication policies. This separation prevents middleware configuration from becoming an implicit dependency between autonomy functions.
Safety and fault handling should be designed across the entire architecture. Runtime monitors can evaluate sensor health, localization confidence, perception validity, trajectory feasibility, communication status, actuator response, and computing resources. When necessary, the system should transition from nominal autonomy to degraded operation, controlled stopping, fallback navigation, or a minimal-risk condition. Watchdogs, emergency-stop interfaces, independent safety controllers, and actuator-level protections should provide layers of defense rather than relying exclusively on the autonomy software.
Computing architecture should reflect the different characteristics of each workload. GPU resources may be assigned to deep-learning perception, sensor fusion, and other AI workloads, while CPU resources handle planning, system management, communication, and general processing. Real-time processors or dedicated controllers may execute safety-critical control functions. The architecture should explicitly manage memory, inter-process communication, data transfer, and computational budgets because an overloaded AI workload must not prevent the vehicle from maintaining safe control behavior.
The resulting custom architecture should preserve the proven functional structure of modern AV platforms while adapting it to the actual outdoor AMR mission. Sensor processing, localization, perception, traversability estimation, world representation, mission management, behavior planning, motion planning, control, and vehicle integration form the main autonomy loop. Middleware, safety, diagnostics, timing, configuration, and computing infrastructure support this loop across all layers. The objective is not to reproduce Autoware or Apollo completely, but to select appropriate architectural principles and combine them into a maintainable, testable, portable, and safety-oriented AMR software platform.
02.05. Real Time and Non Real Time Module Separation
An autonomous vehicle software system contains workloads with fundamentally different timing requirements. Vehicle control, emergency braking, actuator supervision, and selected state-estimation functions may require deterministic or tightly bounded execution, while perception, mapping, logging, visualization, diagnostics, and AI model processing can tolerate greater timing variation. Separating these workloads is therefore an architectural requirement rather than merely an optimization. The module structure places real-time and non-real-time separation within the AV software architecture, alongside middleware, data flow, hardware abstraction, and safety architecture.
Real-time execution means that correctness depends not only on producing the correct result but also on producing it within a defined temporal constraint. A steering command that is mathematically correct but arrives too late may produce unacceptable vehicle behavior. Similarly, a delayed emergency-stop response can be more important than the numerical accuracy of a perception result. Real-time modules therefore require bounded latency, predictable scheduling, controlled resource access, and carefully defined execution periods. The objective is usually deterministic or sufficiently bounded behavior rather than simply achieving the highest possible average processing speed.
Typical real-time functions include low-level vehicle control, actuator command generation, emergency braking, watchdog supervision, critical vehicle-state estimation, and selected safety monitoring. These functions often execute at relatively high frequencies and interact directly with physical hardware. Their execution must remain stable even when computationally intensive perception or AI workloads experience temporary load increases. For this reason, real-time functions should have protected CPU resources, controlled memory usage, predictable communication paths, and clearly defined timing budgets. Hardware controllers or dedicated processors may also be used when stronger isolation is required.
Non-real-time modules have different characteristics and can accept greater execution-time variation. Deep neural-network inference, point-cloud processing, object detection, map generation, visualization, data recording, fleet communication, and offline analytics may consume substantial computational resources without requiring every execution cycle to finish at exactly the same time. This does not mean that these modules can tolerate unlimited latency. Instead, their timing requirements are generally expressed through acceptable ranges, deadlines, throughput targets, or freshness constraints rather than strict deterministic cycles.
The separation should occur at several architectural levels. At the process level, critical functions can execute in dedicated processes or containers, while non-critical workloads run separately. At the CPU level, specific cores or scheduling classes can be reserved for real-time execution. At the memory level, critical processes should avoid uncontrolled allocation and excessive memory contention. At the communication level, high-priority control messages should not be delayed by large sensor streams or background data transfers. These mechanisms collectively reduce interference between workloads with different timing characteristics.
Middleware configuration is particularly important because communication can undermine otherwise well-designed real-time separation. A real-time controller may have deterministic computation but still experience unpredictable behavior if its messages are delayed by queue congestion, excessive serialization, network contention, or inappropriate quality-of-service settings. Communication channels should therefore be designed according to message priority, frequency, reliability, bandwidth, and latency requirements. High-rate sensor data and low-rate diagnostic information should not automatically share identical communication policies.
The perception pipeline illustrates why separation is necessary. Camera and LiDAR processing may generate large volumes of data and may use GPU acceleration, asynchronous execution, batching, and variable processing times. A temporary increase in scene complexity can increase inference or point-cloud processing time. If perception executes on the same unrestricted resources as the control loop, this computational variation can propagate into control latency. Isolating perception resources and defining bounded interfaces allows perception to degrade or drop data without directly disrupting the vehicle\'s fundamental motion-control loop.
Planning occupies an intermediate architectural position. Mission planning, behavior planning, and trajectory generation may not require the same deterministic frequency as low-level control, but their outputs must remain sufficiently fresh for the current operating situation. A planner that produces a trajectory several seconds after the environment has changed may generate an internally valid but operationally obsolete result. Planning should therefore use deadline monitoring, trajectory freshness checks, cancellation or replacement mechanisms, and explicit timeout behavior. The architecture should distinguish computational flexibility from unlimited delay.
Control requires the strongest timing discipline because it closes the loop between software commands and physical vehicle motion. A controller normally receives vehicle-state feedback, calculates an updated command, and sends that command to the actuator interface at a defined control period. Jitter, communication delay, sensor delay, and actuator response time all contribute to the effective control-loop latency. The architecture should therefore define the expected period, maximum allowable jitter, input freshness, execution deadline, and behavior when a cycle is missed.
Safety functions should remain capable of operating when non-real-time modules fail or become overloaded. A runtime safety monitor, emergency-stop path, watchdog, or independent safety controller may need to detect excessive latency, missing heartbeats, invalid commands, or communication failures without depending on the normal perception or planning pipeline. This creates a layered safety architecture in which failure of a computationally complex module does not automatically imply loss of the ability to place the vehicle into a safer state. The separation boundary must therefore include explicit failure and fallback behavior.
For an outdoor AMR, real-time separation becomes especially relevant because the robot may combine high-compute AI perception with relatively simple but safety-critical motion control. Cameras, LiDARs, radar, terrain analysis, object detection, and traversability estimation can consume significant GPU and CPU resources, while wheel control, braking, steering, and emergency stopping require predictable execution. An appropriate architecture can allow AI workloads to operate flexibly while protecting the motion-control and safety paths. This is particularly important when payload, terrain, wheel slip, or operating speed changes the vehicle\'s dynamic response.
The practical design should define a timing contract for every important interface. Such a contract can specify execution period, expected latency, maximum latency, jitter tolerance, message freshness, queue depth, timeout threshold, and fallback behavior. Monitoring these values during operation makes timing behavior observable rather than relying on assumptions. When a module exceeds its timing contract, the system can record the event, reduce operational capability, replace the module output, or transition toward a controlled stopping or minimal-risk condition depending on the severity.
Real-time and non-real-time separation should ultimately be treated as a system-level design principle connecting software, middleware, computing hardware, and safety engineering. The objective is not to make every module real-time, because doing so can increase complexity and resource requirements without providing meaningful benefit. Instead, each function should receive the timing guarantees appropriate to its physical and operational consequences. This approach creates an architecture in which computationally intensive autonomy functions can evolve independently while critical control and safety functions retain predictable execution, clear timing boundaries, and controlled behavior under degraded conditions.
02.06. AV Middleware Selection ROS2 Cyber RT DDS [w/Code]
Middleware is the communication foundation that connects sensing, localization, perception, prediction, planning, control, vehicle interfaces, diagnostics, and system management within an autonomous vehicle. Middleware selection therefore affects far more than message transport. It influences latency, determinism, data distribution, component isolation, scalability, debugging, deployment, and hardware integration. In the AV software architecture, ROS 2, Cyber RT, and DDS represent important technologies for understanding how distributed autonomy modules exchange information.
An AV middleware must support communication patterns ranging from high-bandwidth camera and LiDAR streams to small but timing-critical control and safety messages. These data flows have different requirements for frequency, reliability, latency, queue depth, durability, and message freshness. Applying one communication policy to every channel can create unnecessary overhead or unpredictable delay. Middleware architecture should therefore begin by classifying data according to operational importance and temporal behavior rather than simply connecting every software module through default settings.
ROS 2 provides a distributed robotics framework in which nodes communicate through topics, services, and actions. Its publish-subscribe model is well suited to modular autonomy software because sensor processing, perception, localization, planning, and control can be implemented as independent components with explicit message interfaces. ROS 2 also provides lifecycle management, parameter handling, component composition, logging, tooling, and ecosystem integration. The broader robotics software structure treats ROS 2 fundamentals, communication, real-time behavior, security, lifecycle management, deployment, and AMR applications as dedicated middleware concerns.
A major architectural characteristic of ROS 2 is its use of DDS-oriented communication through the ROS middleware abstraction layer. DDS provides a data-centric publish-subscribe model designed for distributed systems and exposes Quality of Service policies that control communication behavior. Reliability, durability, history depth, deadline, liveliness, and related policies allow communication to be adapted to different data classes. This flexibility is useful in autonomous systems, but incorrect QoS combinations can also cause communication failures or unexpected buffering, making configuration part of system architecture rather than a secondary deployment detail.
DDS should be understood as a communication standard and middleware foundation rather than as a complete autonomous-driving framework. It provides mechanisms for discovery, data distribution, publisher-subscriber communication, and configurable delivery behavior across distributed processes and machines. Different DDS implementations can provide different performance characteristics and tooling while following common concepts. In an AV architecture, DDS can therefore serve as the transport foundation beneath higher-level robotics or autonomy frameworks while the application layer defines perception, planning, control, and vehicle-specific semantics.
Cyber RT was developed as the runtime framework associated with Apollo\'s autonomous-driving software ecosystem. Its design addresses high-throughput data communication, component scheduling, task execution, and communication between autonomy modules. Rather than providing the broad general-purpose robotics ecosystem associated with ROS 2, Cyber RT is closely aligned with Apollo\'s component architecture and autonomous-driving data flow. This makes it useful when studying how a large AV platform integrates runtime scheduling and communication around a defined autonomy software framework.
The choice between ROS 2 and Cyber RT should therefore be considered together with the surrounding software ecosystem. A system based heavily on Autoware, ROS-compatible sensors, robotics packages, and existing ROS development tools naturally benefits from ROS 2 integration. A system derived from Apollo modules may benefit from Cyber RT because the runtime and component model are aligned with Apollo\'s architecture. Middleware selection should consequently consider software reuse, developer expertise, available drivers, integration cost, debugging tools, long-term maintenance, and deployment requirements in addition to raw communication performance.
Latency is a critical selection criterion, but average latency alone is insufficient. Autonomous systems must also consider latency distribution, worst-case delay, jitter, queue accumulation, serialization overhead, memory copies, and scheduling interference. A middleware configuration that achieves excellent average throughput but occasionally produces large delays may be inappropriate for control-related communication. Performance evaluation should therefore measure end-to-end behavior under realistic concurrent workloads, including cameras, LiDARs, perception inference, planning, logging, and vehicle communication.
High-bandwidth sensor transport introduces another important design issue. Transferring large images and point clouds between processes can consume memory bandwidth and CPU resources through repeated serialization and copying. Intra-process communication, shared-memory transport, loaned-message concepts, component composition, or other zero-copy-oriented techniques can reduce this overhead where supported. The architectural objective is to avoid unnecessary data movement while preserving module isolation and maintainability. Optimization should focus especially on communication paths that carry large data at high frequency.
Real-time requirements further influence middleware configuration. Low-level control, actuator supervision, emergency response, and selected safety functions may require bounded communication behavior, while mapping, visualization, logging, and fleet communication can tolerate more variation. Middleware cannot make an inherently non-deterministic application real-time by itself, but it can support a real-time architecture through appropriate scheduling, queue control, memory management, QoS configuration, and transport selection. Communication design must therefore be coordinated with operating-system and process-level real-time strategies.
Reliability requirements also vary according to message semantics. Losing an occasional camera frame may be acceptable if a newer frame is already available, whereas losing a mode-transition or critical vehicle-state message may have different consequences. Conversely, guaranteeing delivery of every high-rate sensor message can cause queues to grow when consumers fall behind. The middleware policy should therefore reflect whether the application requires the newest available sample, guaranteed delivery, bounded history, transient persistence, or explicit acknowledgement behavior.
Distributed deployment introduces network-level considerations. Autonomous systems may place sensor processing, AI inference, planning, control, logging, or supervision on different processors or computers. Communication then crosses Ethernet or other networks and becomes sensitive to bandwidth, congestion, packet loss, clock synchronization, and topology. Middleware discovery and data distribution must operate predictably across these computing domains. Where critical traffic shares a physical network with bulk data, network segmentation, traffic prioritization, and bandwidth budgeting may be necessary.
Security is another middleware-selection dimension because AV and AMR communication can cross process, computer, vehicle, and fleet boundaries. Authentication, access control, encryption, identity management, secure discovery, and software configuration become increasingly important as systems become connected. Security mechanisms introduce computational and operational overhead, so they should be designed together with communication performance rather than added after deployment. The robotics software structure consequently treats ROS 2 security and robot cybersecurity as dedicated architectural areas.
Observability and debugging should also influence the middleware decision. Engineers need to inspect message rates, latency, dropped samples, queue behavior, node health, topology, timestamps, and resource usage when diagnosing autonomy failures. A communication framework that provides strong runtime introspection and mature tooling can significantly reduce integration effort. Recorded data should also preserve sufficient timing and message information for offline replay, regression testing, and failure reconstruction, especially when problems appear only during complex field operations.
For an outdoor AMR, ROS 2 is particularly relevant when the system combines robotics navigation, custom AI perception, heterogeneous sensors, vehicle controllers, and fleet interfaces. DDS-based QoS can be configured differently for camera streams, LiDAR data, localization, trajectories, diagnostics, and control states. Cyber RT remains valuable as an architectural reference and as a runtime choice when Apollo components form a substantial part of the software platform. The decision should follow the intended software ecosystem rather than assuming that one middleware is universally superior.
A hybrid system may also contain multiple communication domains. Real-time vehicle controllers can communicate through CAN, EtherCAT, or dedicated embedded interfaces, while ROS 2 or Cyber RT connects higher-level autonomy processes and Ethernet-based sensors. Cloud or fleet communication may use separate API or messaging technologies. Middleware architecture should therefore define explicit gateways between domains instead of forcing every communication requirement into a single technology. This preserves clear timing, safety, security, and ownership boundaries.
The final middleware architecture should be selected through workload-driven validation. Representative sensor rates, message sizes, process placement, CPU and GPU loads, network conditions, and failure cases should be reproduced before deployment. End-to-end latency, jitter, throughput, packet loss, CPU consumption, memory usage, and recovery behavior can then be measured. The purpose of middleware selection is not merely to choose ROS 2, Cyber RT, or DDS by name, but to establish a predictable communication infrastructure that supports modularity, performance, safety, maintainability, and future expansion of the autonomous-driving or outdoor-AMR software stack.
02.07. AV Data Flow Architecture Sensor to Actuation
Autonomous vehicle data flow architecture defines how information moves from physical sensing to vehicle actuation through the complete autonomy stack. Sensors observe the environment and vehicle state, processing modules transform raw measurements into usable representations, planning modules determine intended motion, and controllers convert trajectories into actuator commands. The architecture must preserve timing, coordinate frames, confidence, validity, and ownership throughout this flow. Within the Volume 12 structure, sensor-to-actuation data flow follows middleware selection and precedes hardware abstraction and safety architecture.
The data flow begins with physical sensors such as cameras, LiDAR, radar, GNSS, IMU, wheel encoders, ultrasonic sensors, and vehicle-state interfaces. Each sensor produces data with different rates, formats, bandwidths, coordinate systems, and uncertainty characteristics. Camera streams may contain millions of pixels per frame, while an IMU generates compact measurements at much higher frequency. The architecture should therefore treat sensor data as heterogeneous time-dependent observations rather than as interchangeable messages.
Sensor drivers form the first software boundary between physical devices and the autonomy stack. They receive device-specific packets, decode proprietary formats, verify communication status, and convert measurements into standardized internal representations. Accurate timestamps should be assigned as close to the measurement source as practical because downstream fusion depends on temporal consistency. Drivers should also expose device health, sequence information, frame identifiers, calibration references, and error states instead of forwarding only measurement values.
Preprocessing transforms raw sensor measurements into representations suitable for downstream algorithms. Camera pipelines may perform decoding, rectification, resizing, and image normalization, while LiDAR pipelines can apply filtering, motion compensation, downsampling, and coordinate transformation. Radar processing may produce detections or tracks, and GNSS, IMU, and wheel measurements may undergo validation and synchronization. This stage should preserve the relationship between processed data and its original measurement time so that later modules can reason about data freshness.
Time synchronization is a fundamental property of the complete sensor-to-actuation path. Sensors rarely capture measurements at exactly the same instant, and communication or computation introduces additional delay. The system may therefore use hardware timestamps, synchronized clocks, buffering, interpolation, or temporal alignment to create consistent observations. Every important data product should carry sufficient timing information to determine when the underlying physical event occurred, when processing was completed, and whether the information remains valid for the current control cycle.
Coordinate transformation provides the spatial counterpart to time synchronization. Sensor measurements may originate in camera, LiDAR, radar, IMU, vehicle, local-map, or global-map coordinate frames. Calibration and transformation chains convert these measurements into common frames so that information can be fused correctly. Incorrect frame definitions can create errors that appear to be perception or localization failures. The data architecture should therefore manage frame identifiers, static calibration, dynamic transforms, and transformation timestamps as explicit parts of the information flow.
Localization converts GNSS, IMU, wheel odometry, LiDAR matching, visual information, or map references into an estimate of vehicle pose and motion. Its output normally includes position, orientation, velocity, and some representation of confidence or uncertainty. This state becomes a major reference for perception, mapping, planning, and control. Because localization quality can change rapidly, downstream modules should receive not only the estimated pose but also validity, covariance, source status, and timestamp information that indicate how strongly the estimate can be trusted.
Perception transforms synchronized sensor information into an environmental representation that is useful for autonomous decision making. Object detection, tracking, free-space estimation, road or terrain segmentation, obstacle extraction, and traversability analysis may operate in parallel. Their outputs can include object lists, occupancy information, semantic regions, terrain properties, and predicted motion. These products should use clearly defined coordinate frames and timestamps so that a planner does not unknowingly combine observations representing different locations or different moments.
A world-model or environment-model layer integrates perception, localization, maps, and vehicle state into a coherent representation. Rather than forcing every planning module to interpret raw sensor outputs independently, this layer can expose objects, free space, occupancy, semantic information, terrain costs, route context, and dynamic predictions through controlled interfaces. The model should define data ownership and update rules because multiple modules may otherwise attempt to modify the same representation asynchronously and create inconsistent interpretations of the environment.
Prediction extends the current environmental state into possible future states. Tracked pedestrians, vehicles, or other dynamic objects may be associated with estimated velocities, trajectories, intentions, or uncertainty envelopes. For outdoor AMRs, prediction may also include changes in traversable space caused by moving equipment, temporary obstacles, or human activity. Prediction data must remain temporally connected to the observation from which it was generated because an accurate prediction based on stale input can still be inappropriate for current planning.
Behavior planning interprets mission objectives, route information, world-model state, and predicted interactions to determine what the vehicle should do next. The output may represent intentions such as continue, yield, stop, avoid, reroute, dock, or enter a fallback state. This layer reduces complex environmental information into a decision context for motion planning. The interface should communicate both the requested behavior and constraints associated with that behavior so that trajectory generation does not need to reconstruct high-level mission logic from raw perception data.
Motion planning converts behavioral intent and environmental constraints into an executable path or trajectory. The resulting data may contain position, heading, curvature, velocity, acceleration, and time references along the future motion sequence. Vehicle geometry, steering limits, braking capability, obstacle boundaries, terrain conditions, and dynamic constraints should be incorporated before the trajectory is released to control. A trajectory should also contain identifiers, generation timestamps, validity intervals, and status information so that obsolete trajectories can be detected and rejected.
The control layer receives the desired trajectory together with current vehicle-state feedback and computes commands required to reduce tracking error. Longitudinal control may determine velocity, acceleration, braking, or drive torque, while lateral control can calculate steering angle, curvature, or differential wheel commands. The controller operates at a higher and more regular frequency than many planning functions, so it must continue receiving fresh state estimates between trajectory updates. Timing consistency between reference trajectory and feedback state is therefore essential.
Vehicle-interface software translates generic autonomy commands into platform-specific actuator communication. Steering, propulsion, braking, gear, wheel torque, or differential-drive commands may be encoded into CAN, Ethernet, EtherCAT, serial, or proprietary messages depending on the platform. The same interface returns measured velocity, steering state, wheel speed, actuator status, battery information, faults, and emergency conditions. This bidirectional path closes the information loop between autonomy software and the physical vehicle.
Actuation is not the end of the data flow because every physical command changes the state that subsequent sensors observe. Steering changes vehicle orientation, propulsion changes velocity, braking changes acceleration, and terrain interaction can introduce slip or disturbance. Sensors measure the resulting motion and feed it back into localization, perception, state estimation, and control. Autonomous driving should therefore be understood as a continuously operating closed-loop data system rather than a one-directional pipeline from sensors to actuators.
Safety monitoring should observe data across the entire pipeline rather than operating only at the final actuator interface. Missing sensor updates, stale localization, inconsistent perception, invalid trajectories, control overruns, communication loss, and actuator faults can all indicate unsafe conditions. Health and validity information should propagate alongside functional data so that downstream modules can respond to degraded inputs. Independent watchdogs or safety controllers may additionally supervise critical commands and force controlled stopping when normal data flow becomes unreliable.
The complete architecture should finally be evaluated through end-to-end latency and data provenance. Engineers should be able to trace an actuator command backward through control, trajectory generation, behavior decisions, environmental representation, perception, and the sensor measurements that contributed to it. Timestamps, sequence identifiers, frame information, validity states, and diagnostic metadata make this trace possible. A well-designed sensor-to-actuation architecture therefore provides not only efficient data movement but also temporal consistency, spatial consistency, observability, fault containment, and a verifiable closed loop between the physical environment and autonomous vehicle behavior.
02.08. Hardware Abstraction for Multi Platform AV SW
Hardware abstraction allows autonomous-driving software to operate across different vehicle and robot platforms without requiring perception, localization, planning, and autonomy logic to understand every hardware-specific implementation. Cameras, LiDARs, GNSS receivers, IMUs, motor controllers, steering systems, and computing devices may change between platforms, but the upper software layers should interact with stable logical interfaces. Within the Volume 12 architecture, hardware abstraction follows sensor-to-actuation data flow and precedes AV software safety architecture.
A multi-platform AV architecture should separate physical devices from functional capabilities. Instead of allowing an application to depend directly on a specific LiDAR model, CAN message identifier, steering controller, or GNSS receiver, the software should request capabilities such as point-cloud input, vehicle pose, steering command, wheel velocity, or actuator status. Device-specific drivers then translate between these logical capabilities and actual hardware. This separation reduces coupling and makes hardware replacement possible without redesigning the complete autonomy stack.
The lowest abstraction layer normally contains device drivers and communication adapters. Drivers communicate with sensors, embedded controllers, and actuators through interfaces such as Ethernet, CAN, CAN FD, EtherCAT, USB, serial communication, or vendor-specific protocols. They decode packets, verify communication state, manage device configuration, and expose measurements or commands through standardized software interfaces. Hardware-specific behavior should remain concentrated within this layer rather than spreading into perception, planning, or control modules.
Sensor abstraction should normalize information while preserving the characteristics required by autonomy algorithms. Different cameras may provide different resolutions, frame rates, encoding formats, and calibration models, while LiDAR devices may differ in channel count, scan pattern, field of view, and point representation. An abstraction layer can expose common image or point-cloud structures, timestamps, coordinate frames, calibration information, and health status. It should not hide meaningful physical differences that affect algorithm performance or safety.
Vehicle-state abstraction performs a similar role for chassis and motion information. Different platforms may represent velocity, steering angle, wheel speed, gear state, brake status, battery condition, or fault information using completely different communication messages. A standardized vehicle-state interface converts these signals into a common internal model. Localization, planning, control, diagnostics, and safety modules can then consume consistent vehicle information without depending directly on platform-specific CAN databases or embedded-controller protocols.
Command abstraction is particularly important because vehicle actuation varies substantially among platforms. Ackermann-steered vehicles may require steering-angle and longitudinal commands, differential-drive AMRs may require left and right wheel velocities, and more complex platforms may expose torque, brake pressure, wheel steering, or individual motor commands. The autonomy stack should generate a platform-independent motion request whenever practical, while a vehicle adapter converts that request into commands appropriate for the actual drivetrain and steering architecture.
A capability-based interface is often more scalable than an interface based only on device names. Each platform can declare whether it supports functions such as steering control, independent wheel torque, service braking, parking braking, reverse motion, GNSS RTK, radar, or redundant localization. Higher-level modules can configure themselves according to available capabilities instead of assuming identical hardware. This approach becomes increasingly useful when one autonomy software family must support indoor AMRs, outdoor patrol robots, heavy logistics platforms, and other mobile systems.
Configuration management separates platform differences that can be represented as parameters from differences that require software changes. Vehicle dimensions, wheelbase, turning radius, sensor mounting positions, communication addresses, control limits, topic names, calibration files, and device identifiers should normally be loaded from platform-specific configuration. The executable software can remain common while deployment packages select the appropriate configuration. Version control of these parameters is essential because an incorrect configuration can produce behavior as dangerous as an incorrect algorithm.
Coordinate-frame abstraction is necessary when sensors are mounted differently across platforms. The autonomy algorithms should operate with clearly defined frames such as sensor, base, vehicle, odometry, local-map, and global-map frames rather than embedding physical mounting coordinates in application code. Platform-specific calibration defines the transformations between these frames. When a sensor position changes, updating calibration should be sufficient wherever possible, preventing geometry assumptions from being duplicated across multiple perception and localization modules.
Computing hardware also requires abstraction. One platform may use an embedded CPU and GPU, another may use a high-performance edge computer, and another may distribute workloads across several computers. Software should minimize unnecessary assumptions about processor model, GPU count, storage device, or network topology. Hardware-aware optimization can still be provided through backend implementations, acceleration libraries, or deployment profiles while the functional interface presented to autonomy modules remains stable.
Middleware contributes to hardware abstraction by separating communication endpoints from their physical location. A perception module can consume a standardized sensor topic regardless of whether the producer runs in the same process, another container, or another computer. ROS 2, DDS, Cyber RT, or comparable communication mechanisms can provide this logical connectivity. Middleware alone, however, does not create a good abstraction; message definitions, ownership rules, timing semantics, and error behavior must also remain stable across platform implementations.
Hardware abstraction must include timing and synchronization semantics because two devices that produce similar data may behave differently in time. Sensor latency, timestamp source, trigger mechanism, clock domain, update frequency, and communication delay can affect sensor fusion and control. Standard interfaces should therefore describe not only the value of a measurement but also when it was captured and how timing validity is determined. A replacement device should not be considered equivalent merely because it produces the same nominal data type.
Fault and health abstraction is equally important. Vendors may report communication loss, overheating, degraded accuracy, saturation, initialization failure, or internal faults using different mechanisms. The hardware abstraction layer should translate these conditions into a consistent health model that system management and safety modules can interpret. Common states such as initializing, operational, degraded, unavailable, and faulted can simplify supervisory logic while detailed vendor information remains accessible for diagnostics and maintenance.
Safety boundaries should prevent hardware abstraction from concealing conditions that require platform-specific protection. Maximum steering rate, braking capability, actuator timeout, payload limitation, thermal limits, emergency-stop behavior, and safe-state transitions may differ significantly among vehicles. These properties should be represented through explicit constraints and safety interfaces rather than hidden behind a generic command API. The abstraction should simplify integration while preserving every hardware characteristic that can influence safe motion.
Testing becomes significantly easier when hardware interfaces are abstracted. Real devices can be replaced by simulators, recorded-data sources, software mocks, or hardware-in-the-loop systems while the upper autonomy modules continue using the same interface. This enables perception, planning, control, and fault-handling logic to be tested before the target vehicle is available. The same architecture also supports regression testing when a sensor, controller, or computing platform is replaced during the product lifecycle.
For outdoor AMRs, abstraction should accommodate substantial differences in size, payload, steering geometry, terrain capability, sensor configuration, and computing performance. A small patrol AMR and a heavy logistics vehicle may share localization, perception, mission management, and planning concepts while requiring different motion models and actuator interfaces. Platform profiles can bind common autonomy modules to specific sensor suites, vehicle dynamics, safety limits, and control backends without creating separate software stacks for every robot.
The resulting multi-platform architecture should therefore define a stable boundary between reusable autonomy intelligence and replaceable hardware implementations. Device drivers, vehicle adapters, calibration, configuration, capability descriptions, timing semantics, health models, and hardware-specific safety constraints form the abstraction layer beneath the autonomy stack. When these boundaries are designed carefully, new sensors, controllers, computers, and vehicle platforms can be integrated with limited impact on higher-level software, enabling portability, reuse, testing, maintainability, and systematic expansion across multiple autonomous vehicle and AMR families.
02.09. AV SW Safety Architecture SOTIF Alignment
Autonomous vehicle software safety architecture must address not only failures of hardware and software components but also hazards that can occur when correctly functioning autonomy functions behave inadequately in difficult operating conditions. Perception uncertainty, incomplete environmental understanding, ambiguous scenes, unusual road geometry, and unexpected interactions can produce unsafe behavior without a conventional component failure. SOTIF provides an important framework for addressing these limitations of intended functionality. Within the Volume 12 structure, this topic follows hardware abstraction and precedes Autoware ROS 2 package customization.
Traditional functional safety and SOTIF address related but different sources of risk. Functional safety primarily considers hazards resulting from malfunctioning behavior, while SOTIF focuses on hazards caused by functional insufficiencies or foreseeable misuse when the system may technically operate as designed. An autonomous-driving architecture should therefore avoid treating safety as a single monitor added after development. Safety requirements must influence sensing, perception, localization, planning, control, interfaces, computing, diagnostics, and operational constraints throughout the software architecture.
The safety architecture begins with a clear definition of intended functionality and operating conditions. The system should specify where, when, and under what environmental and vehicle conditions autonomous operation is expected to perform safely. Road type, terrain, weather, illumination, speed, sensor visibility, localization availability, communication conditions, and interaction with pedestrians or vehicles may all affect capability. These assumptions form an operational boundary against which perception limitations, planning behavior, and fallback strategies can be evaluated.
SOTIF-oriented analysis places particular emphasis on functional insufficiencies. A camera may operate correctly but fail to detect an object under glare, a LiDAR may provide valid measurements while sparse returns reduce classification quality, or localization may remain active while uncertainty becomes too large for precise navigation. These situations are not necessarily conventional faults. The architecture must therefore represent uncertainty, confidence, validity, environmental conditions, and performance limitations explicitly instead of reducing every module output to a simple available-or-failed state.
Perception safety requires multiple mechanisms because environmental understanding is inherently uncertain. Sensor diversity, cross-checking, confidence estimation, temporal consistency, plausibility checks, and redundancy can help identify unreliable perception results. A detected object should include confidence and timing information, while free-space or traversability outputs should indicate validity and uncertainty where appropriate. The safety architecture can then distinguish between high-confidence environmental knowledge and situations where the vehicle should reduce speed, increase margins, request additional evidence, or stop.
Localization safety follows similar principles. Position estimates should include covariance, integrity, source status, and freshness rather than providing only a pose. GNSS RTK, IMU, wheel odometry, LiDAR localization, and map matching can provide complementary information, but agreement between sources should be monitored. Sudden jumps, excessive drift, stale updates, inconsistent coordinate transforms, or degraded GNSS conditions should trigger defined responses. Planning should never assume that a numerically available localization result is automatically trustworthy.
The world model represents another important safety boundary because it integrates information used by downstream decision making. Objects, occupancy, free space, terrain, routes, and vehicle state may be generated at different rates and from different sources. Each data product should carry timestamps, frame information, validity, and ownership. Consistency checks can detect stale or contradictory information before it reaches planning. This prevents apparently valid planning decisions from being based on an internally inconsistent representation of the environment.
Planning safety should constrain behavior and trajectory generation even when perception inputs appear valid. Behavioral decisions must respect operational rules, exclusion zones, speed limits, stopping requirements, and mission constraints. Trajectory generation should verify collision clearance, curvature, acceleration, braking distance, vehicle geometry, dynamic feasibility, and terrain constraints. Independent trajectory validation can provide an additional protection layer by checking a proposed trajectory before control execution rather than assuming that the planner always produces safe output.
Control safety requires predictable timing and explicit command limits. Steering, velocity, acceleration, braking, torque, and wheel commands should remain within platform-specific physical and safety boundaries. Control modules should detect stale trajectories, missing feedback, excessive tracking error, actuator saturation, and missed execution deadlines. A command that is mathematically reasonable may still be unsafe if it is based on outdated state information. Timing validity must therefore be considered part of command validity.
Runtime monitoring connects these safety mechanisms across the software stack. Monitors can observe sensor health, perception confidence, localization integrity, world-model consistency, planner status, trajectory validity, control timing, middleware communication, computing load, and actuator response. Monitoring should identify both explicit faults and gradual degradation. The resulting safety state can be represented through defined modes such as nominal operation, degraded operation, restricted operation, controlled stop, or minimal-risk condition.
Fallback behavior must be designed before failures or functional limitations occur. When confidence decreases, the system may reduce speed, increase following distance, restrict maneuver options, switch localization sources, disable a mission function, reroute, request remote assistance, or perform a controlled stop. The appropriate response depends on the vehicle, operating environment, and severity of the condition. A fallback strategy should itself have clearly defined prerequisites because a maneuver cannot be considered safe if the sensors or actuators required to execute it are unavailable.
Independent safety paths can reduce dependence on the main autonomy pipeline. An emergency-stop system, safety controller, watchdog, collision monitor, or low-level actuator protection may operate separately from high-compute perception and planning. Such mechanisms should not blindly duplicate the complete autonomy stack; instead, they provide simpler protective functions with limited dependencies. Architectural independence can help prevent a single software overload, middleware failure, corrupted state, or computing fault from eliminating every available safety response.
SOTIF alignment also requires systematic treatment of known and unknown hazardous scenarios. Development should identify conditions in which functional limitations are already understood and search for additional triggering conditions through simulation, recorded-data replay, scenario generation, field testing, and edge-case analysis. Rare combinations of weather, lighting, object appearance, terrain, sensor degradation, and traffic interaction are especially important. Newly discovered scenarios should feed back into requirements, datasets, algorithms, monitors, and operational constraints.
Safety evidence depends heavily on observability and traceability. When unsafe or unexpected behavior occurs, engineers should be able to reconstruct which sensor observations, localization states, perception outputs, world-model information, planning decisions, and control commands contributed to the event. Consistent timestamps, software versions, calibration identifiers, configuration versions, diagnostic states, and event logs support this reconstruction. Traceability also helps connect identified hazards with requirements, implementation changes, tests, and verification results.
For outdoor AMRs, SOTIF considerations extend beyond conventional road-driving scenarios. Uneven terrain, slopes, curbs, vegetation, dust, rain, low illumination, GNSS degradation, wheel slip, temporary obstacles, mixed pedestrian areas, and changing payloads can create functional limitations without producing explicit hardware faults. Traversability estimation, localization confidence, vehicle stability, stopping capability, and terrain-dependent control should therefore participate in the safety architecture. Operational limits may need to change dynamically as environmental confidence decreases.
The resulting AV software safety architecture should combine preventive design, runtime monitoring, validation, fallback behavior, and continuous scenario-based improvement. SOTIF alignment does not mean adding a single SOTIF software module; it means ensuring that functional limitations and triggering conditions are considered throughout perception, localization, world modeling, planning, control, hardware abstraction, middleware, and system management. The objective is an autonomy system that can recognize uncertainty and degradation, constrain its behavior accordingly, and transition toward a safer state when the assumptions required for intended autonomous operation are no longer sufficiently satisfied.
02.10. Autoware ROS2 Package Customization for AMR [w/Code]
Customizing Autoware ROS 2 packages for an autonomous mobile robot requires adapting a road-vehicle-oriented autonomy framework to the geometry, dynamics, sensors, operating environment, and mission structure of the target AMR. The objective is not to modify every Autoware component, but to preserve reusable localization, perception, planning, control, and system-management functions while replacing or extending interfaces that depend on vehicle assumptions. Within the Volume 12 structure, this topic completes the AV software architecture chapter after safety architecture and SOTIF alignment.
The customization process should begin by defining which Autoware functions can remain unchanged and which functions require adaptation. Generic ROS 2 communication, coordinate transforms, diagnostics, localization concepts, perception pipelines, and visualization tools may often be reused. Vehicle interfaces, motion models, planning constraints, map assumptions, and control parameters are more likely to depend on the AMR platform. Establishing this boundary early prevents unnecessary source-code modification and reduces the maintenance burden when upstream Autoware packages evolve.
A dedicated AMR integration workspace can contain platform-specific packages while maintaining upstream Autoware packages separately. The customized workspace may provide sensor drivers, vehicle interfaces, launch configurations, parameter files, calibration data, motion-model adapters, mission interfaces, and safety extensions. ROS 2 package dependencies should remain explicit so that customized functions can be rebuilt and tested independently. This overlay-oriented approach is generally easier to maintain than directly editing large numbers of upstream package sources.
Sensor integration should expose AMR hardware through interfaces expected by the autonomy stack. Camera, LiDAR, GNSS RTK, IMU, wheel encoder, radar, and ultrasonic data may require custom ROS 2 drivers or adapters. Topic names, message types, timestamps, frame identifiers, update frequencies, and QoS policies must be consistent with downstream consumers. When an existing Autoware component expects a particular representation, an adapter can convert platform-specific data rather than forcing the perception or localization algorithm to understand the original hardware protocol.
Coordinate-frame configuration is one of the most important customization tasks. The AMR should define stable relationships among sensor frames, the robot or vehicle base frame, odometry, local map, and global map. Static transformations can describe sensor mounting positions, while localization provides dynamic transformations associated with vehicle motion. Calibration data should remain outside algorithm source code whenever possible. This allows sensor positions or platform dimensions to change without requiring modifications throughout the autonomy software.
Localization customization depends strongly on the AMR operating environment. An outdoor AMR may combine GNSS RTK, IMU, wheel odometry, LiDAR localization, and map matching, while indoor or GNSS-denied operation may require a different source combination. Existing localization packages can be reused where their assumptions remain valid, while fusion logic and configuration can be adapted to the available sensors. The customized interface should continue to provide consistent pose, velocity, uncertainty, status, and coordinate-frame information to downstream modules.
Perception customization should focus on the objects and terrain conditions relevant to AMR operation. A road-oriented detector may emphasize passenger vehicles, lanes, and traffic participants, whereas an outdoor AMR may additionally require curbs, vegetation, industrial equipment, small obstacles, uneven terrain, slopes, and traversable surfaces. Existing point-cloud and image-processing packages can remain useful, but model weights, classes, thresholds, regions of interest, fusion logic, or output adapters may require modification. The interface to planning should remain stable despite these internal changes.
Planning often requires deeper customization because AMR missions and motion constraints can differ significantly from passenger-vehicle assumptions. Route planning may need to support patrol points, inspection locations, docking stations, charging areas, restricted zones, or repeated industrial routes. Behavior planning can add AMR-specific states such as mission waiting, docking approach, obstacle bypass, remote intervention, or return to base. These extensions should preferably be introduced through clearly defined packages and interfaces instead of embedding mission logic directly into generic trajectory algorithms.
Motion planning must reflect the actual kinematic and dynamic model of the AMR. An Ackermann-steered outdoor platform may reuse more road-vehicle planning concepts, while differential-drive, skid-steer, multi-axle, or independently driven platforms may require different feasibility checks and trajectory representations. Vehicle dimensions, minimum turning radius, steering limits, velocity limits, acceleration, braking capability, ground clearance, payload, and terrain constraints should be represented through configuration or dedicated model interfaces rather than scattered constants.
Control packages must also be matched to the physical vehicle. Existing longitudinal and lateral controllers can be retained when their underlying vehicle assumptions are compatible with the AMR, but controller gains, limits, feedback signals, and command representations require platform-specific tuning. A different drive architecture may require a custom control backend that converts a planned trajectory into wheel velocity, steering, torque, or braking commands. The rest of the autonomy stack should remain isolated from these actuator-level differences.
The vehicle interface package forms the principal boundary between Autoware and the AMR embedded system. It translates generic autonomy commands into CAN, Ethernet, EtherCAT, serial, or proprietary controller messages and converts vehicle feedback into standardized state information. Steering state, velocity, wheel speed, brake state, battery condition, drive mode, emergency-stop status, and faults can be exposed through this interface. Keeping protocol-specific implementation here prevents hardware details from spreading throughout higher-level packages.
ROS 2 launch and parameter architecture should support multiple AMR configurations without duplicating the complete software stack. Platform-specific launch files can select sensors, localization sources, perception models, planners, controllers, and vehicle adapters. Parameter sets can define dimensions, limits, QoS settings, topic remapping, map locations, and algorithm thresholds. Configuration inheritance or layered parameter files can separate common autonomy settings from platform-specific values, simplifying deployment across several robot variants.
Lifecycle management can improve startup, shutdown, and recovery behavior. Drivers and autonomy components should not become active before required calibration, transforms, maps, and vehicle interfaces are ready. Managed transitions can coordinate configuration, activation, deactivation, error handling, and restart. This is especially useful for AMRs that repeatedly power on in operational environments or recover from communication and sensor faults. System state should therefore be coordinated rather than relying only on arbitrary process startup order.
Safety customization should preserve the architecture developed around monitoring, fallback, and SOTIF-related limitations. AMR-specific safety nodes can monitor sensor availability, localization confidence, traversability validity, trajectory freshness, control timing, communication state, and actuator feedback. Speed reduction, restricted operation, mission cancellation, controlled stopping, or emergency stopping can be triggered according to defined conditions. Safety logic should remain sufficiently independent from high-compute perception and planning to remain available during overload or partial failure.
Testing should accompany every customization layer. Individual ROS 2 packages can be verified with unit and interface tests, while recorded sensor data can support perception and localization regression testing. Simulation can validate planning, control, and mission behavior before deployment, and hardware-in-the-loop testing can exercise actual controllers and communication interfaces. Field tests should then verify timing, calibration, vehicle dynamics, fallback behavior, and complete sensor-to-actuation performance under representative outdoor AMR conditions.
Long-term maintenance requires minimizing divergence from upstream Autoware. Custom packages should use stable interfaces where possible, document every modified assumption, and isolate unavoidable patches. Changes to message definitions, package APIs, dependencies, or ROS 2 versions should be evaluated through automated builds and regression tests. Maintaining a clear boundary between upstream software and AMR-specific extensions makes it easier to adopt future improvements without repeatedly reconstructing the customized autonomy stack.
The resulting system should therefore be viewed as an Autoware-based AMR software platform rather than a completely rewritten AV stack. Reusable Autoware and ROS 2 capabilities provide the foundation, while AMR-specific packages adapt sensors, coordinate frames, localization, perception, missions, motion models, control, vehicle interfaces, configuration, and safety behavior. Careful package boundaries allow the same architecture to support multiple AMR platforms while preserving portability, testability, maintainability, and a practical path for continued integration with the evolving Autoware ecosystem.
Author: Richard Minsoo Go
LinkedIn: Richard Minsoo Go










Top comments (0)