STATERA Library · Series 08 · Volume 05 · Chapter 05 · EN
Source: https://github.com/Statera-Library/08_Vol_05_Robot_Middleware_and_ROS2/blob/main/Chapter_05_ROS2_Real_Time/Chapter_05_ROS2_Real_Time_EN.md
Volume 05 Robot Middleware and ROS2
05. ROS2 Real-Time
05.01 ROS2 Real-Time Limitations and Realistic Goals
ROS 2 is frequently described as suitable for real-time robotics, but this statement requires careful interpretation. ROS 2 provides architectural mechanisms that can support deterministic execution, yet a standard ROS 2 application running on a general-purpose Linux system is not automatically a hard real-time system. The practical objective is therefore to identify which parts of a robot require bounded timing and engineer those paths separately from functions that tolerate variable latency.
Real-time behavior is fundamentally different from simply achieving high computational speed. A system may execute a control operation in hundreds of microseconds on average and still be unsuitable for a critical control loop if occasional delays extend into several milliseconds. Real-time engineering focuses on predictability, bounded worst-case latency, deadline satisfaction, and controlled jitter rather than average throughput alone. This distinction establishes the correct performance goal for ROS 2 systems.
Several software layers influence ROS 2 timing simultaneously. An application callback passes through rclcpp or rclpy, the ROS client library, the rmw abstraction, a DDS implementation, operating-system scheduling, memory management, device drivers, and potentially a network stack. Each layer can introduce scheduling delays or resource contention. Consequently, changing only the DDS QoS configuration or increasing processor performance cannot guarantee deterministic end-to-end behavior.
The ROS 2 executor is one important source of timing variability. Executors determine when subscriptions, timers, services, and other callbacks become executable and when they actually receive CPU time. If multiple callbacks share an executor, a long-running callback can delay a time-sensitive one. Multi-threading can improve concurrency but may also introduce contention, synchronization overhead, cache interference, and less predictable scheduling behavior unless callback groups and priorities are carefully designed.
Dynamic memory allocation is another obstacle to deterministic execution. Operations such as heap allocation, message construction, container resizing, serialization buffers, and object destruction may exhibit variable execution times. Page faults can produce much larger disturbances when memory pages are accessed unexpectedly. Real-time-oriented ROS 2 software therefore attempts to preallocate frequently used objects, reuse buffers, lock required memory, and eliminate unnecessary allocation from critical execution paths.
The operating system establishes another boundary on achievable determinism. Standard Linux is optimized for throughput, fairness, and broad application compatibility rather than strict latency bounds. Kernel activity, interrupts, background services, scheduler decisions, power-management transitions, and device drivers can interrupt robot workloads. PREEMPT_RT significantly improves Linux scheduling behavior, but it should be regarded as an enabling technology whose performance must still be measured under representative system load.
DDS also provides real-time-oriented communication capabilities, but DDS QoS policies should not be confused with hard timing guarantees. Reliability, history depth, durability, deadline, lifespan, and transport configuration influence communication behavior, resource usage, and failure handling. A poorly selected reliable configuration can increase blocking or retransmission effects, while excessive queue depth can preserve stale data. QoS must therefore reflect the semantic and temporal requirements of each data flow.
Network communication makes strict determinism substantially more difficult. Ethernet congestion, switch buffering, retransmissions, packet fragmentation, competing traffic, and clock differences can all affect end-to-end latency. A ROS 2 topic that appears deterministic between processes on one computer may behave differently across machines. Realistic validation must therefore measure complete paths from data production through middleware and networking to the consuming callback rather than benchmarking isolated API operations.
Robot architecture should distinguish hard real-time, firm or soft real-time, and non-real-time workloads. Motor current regulation, torque control, and some safety functions may require tightly bounded execution and are commonly placed on dedicated controllers, MCUs, FPGAs, or specialized real-time environments. Higher-level trajectory generation, navigation, perception, diagnostics, visualization, and fleet coordination generally tolerate progressively larger timing variation and are better suited to conventional ROS 2 execution.
This separation leads to a practical architectural principle: ROS 2 should often coordinate real-time subsystems rather than replace every low-level real-time mechanism. A servo controller can execute a deterministic inner loop locally while ROS 2 supplies targets, trajectories, configuration, state reporting, and supervisory commands. ros2_control and micro-ROS support variations of this layered approach, allowing strict device-level timing requirements to coexist with flexible distributed robot software.
A realistic goal for ROS 2 is therefore bounded and measurable latency for selected critical paths rather than universal hard real-time behavior across the entire software graph. Engineers should define deadlines for individual functions, acceptable jitter, maximum callback execution time, message age limits, and overload behavior. These requirements should be derived from robot dynamics and safety needs instead of adopting an arbitrary target such as "1 ms real-time" for every subsystem.
Average latency is insufficient for validating such requirements. Measurements should include worst observed latency, high-percentile latency, jitter distributions, deadline misses, scheduling delays, and behavior during CPU, memory, disk, GPU, and network stress. Long-duration tests are particularly important because rare latency spikes may remain invisible in short benchmarks. Tracing facilities can then correlate abnormal timing with executor activity, callback execution, DDS communication, or operating-system scheduling.
CPU affinity, thread priority, interrupt placement, memory locking, preallocation, executor configuration, and isolation of critical workloads can progressively improve predictability. However, these techniques should be applied as part of a measured optimization process rather than as independent tuning tricks. A configuration that improves one benchmark may degrade another because CPU isolation, networking, device interrupts, and application concurrency interact at the complete-system level.
Modern Physical AI robots make this distinction even more important. GPU inference for vision, foundation models, world models, or learned policies may require tens or hundreds of milliseconds and exhibit workload-dependent latency, while stabilization or actuator loops may operate at hundreds or thousands of hertz. Attempting to place these functions under one timing model is unrealistic. The architecture should instead connect asynchronous intelligence with deterministic control through explicitly defined interfaces.
Safety requirements must also remain independent of assumptions about ROS 2 timing performance. Emergency stopping, torque limitation, protective monitoring, and other safety-related mechanisms may require certified hardware or safety controllers whose guarantees cannot be replaced merely by assigning high-priority ROS 2 threads. ROS 2 can exchange status and supervisory information with such systems, but architectural boundaries must preserve the required independence and failure response.
The practical objective of ROS 2 real-time engineering is therefore not zero jitter or identical execution time on every cycle. It is controlled temporal behavior: critical deadlines are identified, latency sources are understood, execution paths are isolated where necessary, and deadline violations can be detected and handled safely. This makes determinism an engineering property demonstrated through architecture, configuration, measurement, and validation rather than a feature obtained simply by selecting ROS 2.
Within the broader robot software architecture, ROS 2 real-time capability should be treated as a continuum between conventional distributed middleware and dedicated deterministic control. This positioning complements separate real-time operating-system and control-software layers while allowing ROS 2 to provide standardized communication and integration across them. The most successful systems therefore assign each workload to the timing domain appropriate to its physical consequence.
A mature ROS 2 real-time design ultimately begins with realistic expectations. ROS 2 can support demanding robotic control when combined with appropriate operating systems, middleware configuration, executor design, memory discipline, hardware partitioning, and systematic benchmarking. It should not be expected to transform arbitrary Linux applications into hard real-time software. Defining this boundary clearly provides the foundation for the subsequent techniques used to optimize ROS 2 executors, memory behavior, PREEMPT_RT integration, and control-loop performance.
05.02 SingleThreadedExecutor Real-Time Tuning [w/Code]
The SingleThreadedExecutor in ROS 2 provides one of the simplest execution models for understanding and controlling callback scheduling. It processes executable entities through a single execution thread, preventing callbacks assigned to that executor from running concurrently. For real-time-oriented systems, this simplicity can be valuable because it eliminates many synchronization and thread-contention effects introduced by concurrent execution, although it does not by itself guarantee deterministic timing.
During execution, the executor monitors subscriptions, timers, services, clients, guard conditions, and other entities associated with its nodes. When work becomes available, the executor selects an executable callback and runs it before proceeding to another callback. Because only one callback executes at a time, callback execution becomes serialized. This makes timing interactions easier to analyze, but a long-running callback can directly delay every other callback waiting for the same executor.
The most important tuning principle is therefore to keep callbacks short, predictable, and non-blocking. A callback that performs lengthy computation, waits for network communication, accesses slow storage, or invokes an unpredictable external library can occupy the executor thread for an excessive period. Time-critical callbacks should perform only bounded operations and move non-critical processing to separate execution contexts when necessary, preserving the responsiveness of the primary executor.
Timer-driven control loops require particular attention because timer readiness does not mean immediate execution. A timer may become ready while another callback is already running, causing scheduling latency before the timer callback begins. The effective control-loop period therefore consists not only of the configured timer interval but also of executor waiting time, callback execution time, operating-system scheduling delay, and interference from other entities handled by the same executor.
Callback execution time should be treated as part of a timing budget. If a 1 kHz control loop has a nominal period of 1 ms, every operation sharing its execution path must fit within a substantially constrained budget while leaving margin for scheduling and middleware overhead. When callback execution approaches the available period, even small disturbances can produce deadline misses. Real-time tuning therefore requires worst-case-oriented measurements rather than optimization based solely on average execution time.
Node composition also affects SingleThreadedExecutor behavior. Placing many nodes in one executor simplifies deployment but increases the number of callbacks competing for the same execution thread. A diagnostic subscription, parameter callback, logging operation, or high-bandwidth sensor callback can unexpectedly interfere with a control timer. Critical and non-critical functionality should therefore be partitioned according to timing requirements instead of placing every ROS 2 node into one execution context for convenience.
A useful architecture is to dedicate a SingleThreadedExecutor to a small set of closely related real-time-oriented callbacks while moving perception, logging, visualization, diagnostics, and other variable-latency functions elsewhere. This does not require the entire robot software stack to use the same executor model. Different processes or execution contexts can be assigned according to workload characteristics, allowing the critical path to remain simple while higher-level software retains greater concurrency.
Memory behavior inside callbacks must also be controlled. Dynamic allocation, container resizing, message copying, string manipulation, and creation of temporary objects can introduce variable execution times. Frequently used buffers and messages should be allocated before entering the critical execution phase whenever possible. Reusing objects and maintaining fixed-capacity data structures reduces allocator activity and makes callback execution time more stable under sustained operation.
Blocking synchronization primitives should be minimized on the critical executor thread. Waiting for a mutex, condition variable, service response, file operation, or another thread can destroy the timing advantages of serialized execution. Even when a mutex normally becomes available immediately, contention under unusual load may create substantial latency. Real-time-oriented callback paths should therefore minimize shared mutable state and avoid dependencies whose completion time cannot be bounded.
ROS 2 communication settings also influence executor behavior indirectly. A subscription receiving data faster than its callback can process it may accumulate pending work according to its QoS history configuration. Reliable communication can introduce additional transport behavior, while unnecessarily deep queues may allow stale messages to remain available for processing. QoS depth, reliability, deadline, and message rates should therefore be selected together with the executor\'s processing capacity.
For sensor and control applications, processing the newest meaningful state can be more valuable than processing every historical sample. A high-rate sensor stream that overwhelms a SingleThreadedExecutor may cause increasing data age even if no individual callback is particularly slow. System designers should distinguish information that must never be lost from state data for which freshness is more important than completeness, and configure communication and processing policies accordingly.
Operating-system scheduling becomes increasingly important after application-level sources of variability have been reduced. The executor thread can be assigned an appropriate real-time scheduling policy and priority on systems configured for such operation. CPU affinity can restrict the thread to selected processor cores, while isolation strategies can reduce interference from unrelated workloads. PREEMPT_RT can further reduce kernel scheduling latency when Linux is used for demanding control applications.
CPU affinity alone does not provide isolation because interrupts, kernel threads, device activity, and other processes may still execute on the same processor. Effective tuning considers the complete CPU environment, including interrupt affinity, background services, power-management behavior, frequency scaling, and competing real-time threads. Assigning the highest possible priority to a ROS 2 executor without understanding these interactions can create starvation or system-level instability rather than improved determinism.
Memory locking is another important technique when the operating system supports it. Locking process memory can reduce the risk of page faults during critical execution, while prefaulting stack and heap regions can ensure that required pages are resident before real-time operation begins. These techniques are most effective when combined with preallocation because locking memory does not eliminate unpredictable behavior caused by dynamic allocation or application-level resource contention.
Instrumentation is essential because executor tuning should be based on observed timing behavior. Useful measurements include callback start latency, callback execution duration, timer period error, message age, deadline misses, and end-to-end latency between data production and control output. ros2_tracing and operating-system tracing tools can help distinguish whether a delay originated in callback execution, executor scheduling, middleware processing, kernel scheduling, or another subsystem.
Testing should reproduce realistic worst-case conditions rather than an idle development computer. CPU-intensive perception, GPU inference, network traffic, sensor streams, logging, storage activity, and background services can expose interference that remains invisible during simple tests. Long-duration measurements are especially valuable because rare scheduling disturbances may occur only after millions of callback cycles. Percentile distributions and maximum observed latency should accompany average values.
The SingleThreadedExecutor is particularly useful when execution order simplicity and absence of callback concurrency are more important than maximum throughput. It is less suitable when several computationally expensive callbacks must execute simultaneously or when one blocking operation cannot be removed. In those cases, separating workloads across processes or executors may provide better timing isolation than simply replacing the design with unrestricted multithreaded execution.
For robot control, a practical design often combines a dedicated ROS 2 executor with a lower-level deterministic controller. The ROS 2 executor may handle command exchange, state updates, trajectory references, and supervisory logic, while an MCU, dedicated controller, or tightly controlled real-time thread executes the fastest servo loop. This separation prevents middleware scheduling from becoming the sole timing mechanism for control functions requiring stronger guarantees.
Real-time tuning should therefore proceed from architecture to measurement rather than from isolated parameter changes. First minimize the critical callback set, then eliminate blocking operations and unnecessary allocation, establish suitable QoS and message rates, configure operating-system scheduling and CPU resources, and finally validate the complete path under stress. Each optimization should be justified by measured improvement in latency, jitter, or deadline behavior.
The realistic objective of SingleThreadedExecutor tuning is not to make every ROS 2 callback execute at an identical instant. Its strength is that a deliberately small and controlled callback set can produce an execution model that is easier to reason about, profile, and validate. Combined with disciplined callback design, memory control, PREEMPT_RT, CPU isolation, and systematic benchmarking, it can form a predictable ROS 2 execution layer for many real-time-oriented robotic workloads.
05.03 rclcpp Real-Time Node Design Patterns [w/Code]
The design of a real-time-oriented ROS 2 node begins with recognizing that rclcpp provides communication and execution abstractions, not automatic timing guarantees. A node becomes predictable only when its callbacks, memory behavior, communication paths, synchronization mechanisms, and execution environment are deliberately constrained. The objective is to create a small critical path whose execution time and interference sources can be measured and controlled.
A useful rclcpp design pattern separates initialization from real-time execution. Publishers, subscriptions, timers, parameters, buffers, and internal data structures should be created before the critical operating phase begins. Initialization may perform dynamic allocation and configuration because it occurs outside the timing-critical path. Once real-time execution starts, the node should transition toward stable resource usage and avoid operations whose duration cannot be bounded.
Callback design is the center of real-time node architecture. A critical callback should perform only the operations required to transform its input into the next control or state output. Expensive perception, logging, file access, network requests, parameter reconfiguration, and other variable-duration work should not be inserted into the same callback path. Short callbacks reduce blocking of other entities and make worst-case execution time easier to characterize.
Dynamic memory allocation should be minimized after initialization. Creating temporary vectors, resizing containers, repeatedly constructing messages, manipulating large strings, or allocating new objects inside high-frequency callbacks can introduce allocator-dependent latency. A preferable pattern is to allocate fixed or bounded buffers beforehand, reserve container capacity, reuse message objects, and maintain stable data structures throughout the real-time operating period.
Message ownership and copying also influence timing. Large sensor or state messages may generate significant overhead when repeatedly copied between callbacks and publishers. rclcpp applications should therefore consider message reuse, move semantics, loaned-message capabilities where supported, and intra-process communication when components operate in the same process. The appropriate mechanism depends on middleware and message characteristics, so its timing benefit should be validated rather than assumed.
Synchronization between callbacks requires particular discipline. A mutex protecting shared state may appear inexpensive under normal operation, but its acquisition time becomes unpredictable when another thread holds the lock. Critical data exchange should minimize shared mutable state and keep protected sections extremely short. Where appropriate, bounded queues, atomic variables, double-buffering, or carefully designed lock-free communication can reduce blocking between real-time and non-real-time execution domains.
A common pattern is to divide the node or system into real-time and non-real-time paths. The real-time side performs control calculations, state updates, and time-critical publication, while the non-real-time side handles diagnostics, logging, configuration, visualization, and computationally variable tasks. Data crossing this boundary should use explicitly designed interfaces so that slow auxiliary functions cannot block or acquire resources required by the critical path.
Timers are frequently used to implement periodic behavior in rclcpp, but a timer interval should not be interpreted as a guaranteed execution period. The timer indicates that work is ready; the executor and operating system still determine when the callback actually starts. Periodic nodes should therefore measure actual wake-up time, callback start latency, execution duration, and cycle-to-cycle jitter instead of assuming that a configured period directly represents physical control-loop timing.
Callback groups provide an architectural mechanism for controlling execution relationships. Mutually exclusive groups can prevent selected callbacks from executing simultaneously, while reentrant groups permit concurrency when the executor provides multiple threads. For real-time-oriented nodes, callback-group assignment should express intentional timing and synchronization relationships rather than simply follow software functionality. Critical callbacks should not accidentally share execution resources with unpredictable work.
The executor should be selected together with the node design. A carefully constructed rclcpp node cannot achieve predictable behavior if its callbacks are placed in an executor containing many unrelated, long-running entities. A dedicated SingleThreadedExecutor can provide a simple serialized model for a small critical callback set, while separate executors or processes can handle variable workloads. Execution topology is therefore part of node architecture rather than merely a deployment detail.
Publisher behavior should also be designed around bounded resource use. A control callback that publishes at high frequency can encounter serialization, middleware, transport, or queue-related overhead. Message size and publication rate should match the physical requirements of the subsystem. Large diagnostic information should not share the same critical communication path as compact control-state data when their timing characteristics and priorities are fundamentally different.
Subscription design must consider data freshness as well as reliability. If messages arrive faster than the callback can consume them, queued data may become stale even when no messages are technically lost. For state estimation and control, processing the newest available state may be more valuable than preserving a long history. QoS history depth, reliability, deadline, and publication frequency should therefore reflect the temporal meaning of the data.
Services and synchronous interactions require caution in real-time nodes because waiting for a response introduces a dependency on another execution context. A critical callback should generally avoid synchronous service calls, blocking futures, filesystem operations, and external process communication. Configuration and supervisory commands can often be handled outside the periodic control path, allowing asynchronous events to modify prepared state without suspending time-critical execution.
Logging is another easily overlooked source of timing variation. Formatting strings, acquiring internal locks, writing to consoles, or forwarding logs to storage can introduce unpredictable delays. Detailed logging is valuable during development, but high-frequency critical callbacks should minimize synchronous logging. Timing information can instead be collected through lightweight tracing, counters, or preallocated telemetry mechanisms and processed outside the real-time execution domain.
Parameter updates should similarly be separated from deterministic computation. Runtime parameters are useful for robot configuration, but arbitrary parameter changes can require validation, memory updates, or reconstruction of internal resources. A robust node can receive configuration changes in a non-critical callback, validate them, construct the required state, and then expose the prepared configuration to the real-time path through a bounded handoff mechanism.
Error handling must be designed without introducing uncontrolled recovery work into the critical callback. Exceptions, resource reconstruction, repeated allocation, network reconnection, and extensive diagnostic generation can produce large latency variations. The real-time path should preferably detect abnormal conditions quickly, record compact fault information, transition to a defined safe or degraded state, and delegate complex recovery operations to a supervisory non-real-time component.
Lifecycle concepts complement this pattern because initialization, activation, deactivation, cleanup, and recovery can be separated from normal periodic execution. Resources needed by critical callbacks can be prepared before activation, while cleanup and reconfiguration can occur after execution has been stopped. This separation reduces the amount of unpredictable resource management that must occur while the node is performing time-sensitive robot functions.
At the operating-system boundary, the thread executing critical rclcpp callbacks may require suitable scheduling priority, CPU affinity, memory locking, and protection from unrelated workloads. PREEMPT_RT can reduce Linux scheduling latency, but application design remains essential. A poorly structured node containing blocking operations and dynamic allocation will not become deterministic merely because it executes on a real-time-enabled kernel.
Real-time node validation should measure the complete behavior rather than only callback computation time. Important observations include callback release-to-start latency, execution duration, publication latency, message age, timer jitter, deadline misses, and end-to-end response time. Measurements should be collected under realistic CPU, network, sensor, storage, and accelerator workloads because interference frequently appears only when the integrated robot system is stressed.
For Physical AI systems, rclcpp nodes often connect deterministic control with computationally variable AI pipelines. GPU inference, world models, perception networks, and learned policies should generally be treated as asynchronous producers whose outputs carry timestamps and validity information. The control path can consume the latest acceptable result while maintaining its own deterministic cycle, rather than blocking until every AI inference operation completes.
The resulting design pattern is a layered rclcpp node architecture in which critical execution is deliberately small, bounded, and isolated while flexible ROS 2 functionality surrounds it. Preallocated memory, short callbacks, controlled synchronization, suitable QoS, intentional executor placement, and measured operating-system behavior collectively determine predictability. Real-time performance is therefore an emergent property of the complete node design rather than a single rclcpp API feature.
A mature rclcpp real-time node should remain understandable enough that engineers can identify every operation occurring on its critical path and explain the timing consequence of each dependency. When execution boundaries, resource ownership, communication semantics, and failure behavior are explicit, the node becomes easier to profile and validate. This disciplined design provides the foundation for integrating ROS 2 communication with demanding robot control without confusing middleware flexibility with hard real-time guarantees.
05.04 Memory Locking / Stack Pre-allocation in ROS2 [w/Code]
Memory behavior is one of the most important sources of latency variation in real-time-oriented ROS 2 systems. Even when callback computation is fast, page faults, heap allocation, memory reclamation, and unexpected stack growth can interrupt execution for unpredictable periods. Memory locking and pre-allocation therefore aim to ensure that the memory required by critical ROS 2 callbacks is already resident, allocated, and accessible before deterministic operation begins.
Virtual memory allows Linux processes to operate with an address space larger and more flexible than directly mapped physical memory. However, a virtual page may not be resident in RAM when a real-time thread accesses it. The operating system must then resolve a page fault and potentially perform memory-management work before execution continues. Although this behavior is acceptable for general applications, an unexpected page fault can create unacceptable latency in a high-frequency robot control loop.
Memory locking prevents selected process memory from being paged out or otherwise becoming unavailable during critical execution. On Linux, mechanisms such as mlock() and mlockall() can request that memory remain resident in physical RAM. A real-time ROS 2 process commonly locks its current and future mappings before entering its critical operating phase, reducing the probability that callback execution will be interrupted by major or minor page-fault handling.
Calling memory-locking functions alone is not sufficient because memory pages may still be allocated lazily. Operating systems often map virtual address ranges without immediately establishing all required physical pages. A real-time process should therefore touch or prefault memory regions before critical execution. By deliberately accessing stack, heap, and preallocated buffers during initialization, the application forces required pages to become resident before timing-sensitive callbacks begin.
Stack prefaulting is particularly important because thread stacks can grow as functions are called and local variables are created. If a real-time callback reaches a stack page that has never previously been touched, the operating system may need to map that page during execution. A common strategy is to reserve an adequate stack size and access representative stack memory before entering the real-time loop, ensuring that expected stack pages are committed in advance.
Stack size should be selected with sufficient safety margin but should not be treated as unlimited. Deep call chains, recursion, large local arrays, and complex library functions can increase stack consumption unexpectedly. Real-time-oriented ROS 2 code should prefer bounded call depth and predictable local storage. Large buffers are generally better allocated explicitly during initialization rather than created as large automatic variables inside high-frequency callbacks.
Heap allocation introduces a different class of timing variability. Calls to new, malloc(), container growth, shared-pointer construction, message allocation, and internal library operations may interact with memory allocators whose execution time depends on previous allocation history and synchronization. The real-time path should therefore avoid repeated allocation and deallocation whenever possible, particularly when the callback deadline is short or execution frequency is high.
Pre-allocation moves this work into the initialization phase. Vectors can reserve their maximum expected capacity, message buffers can be created beforehand, queues can use fixed or bounded storage, and reusable working memory can be assigned to algorithms before activation. Once the node enters real-time operation, callbacks operate on these existing resources rather than requesting new memory. This transforms memory consumption from an event-driven behavior into a planned resource model.
ROS 2 messages deserve special attention because apparently simple publish and subscribe operations can involve message construction, serialization, copying, middleware buffers, and transport resources. Reusing message objects can reduce application-level allocation, while intra-process communication and loaned-message mechanisms may reduce copies in appropriate configurations. However, actual behavior depends on the rclcpp, rmw, DDS, and transport implementation, so allocation behavior should be measured on the deployed stack.
Standard C++ containers are useful in ROS 2 software but require disciplined use in critical paths. std::vector may reallocate when capacity is exceeded, std::string can allocate as content grows, and associative containers can allocate nodes dynamically. Reserving capacity or replacing unbounded structures with fixed-capacity alternatives can improve predictability. The critical requirement is that runtime data cannot unexpectedly exceed the storage assumptions established during initialization.
Custom allocators can provide additional control when standard allocation behavior is insufficient. rclcpp and C++ interfaces can support allocator-aware designs in selected areas, allowing applications to use preallocated pools or deterministic allocation strategies. Nevertheless, introducing a custom allocator does not automatically create real-time behavior. Middleware, logging, synchronization, and application libraries may still allocate memory, so the complete execution path must be examined.
Memory pools are useful when objects must be acquired and released during operation but their maximum number can be bounded. Instead of requesting arbitrary heap memory, the application reserves a fixed set of objects during initialization and recycles them at runtime. Pool exhaustion must have explicitly defined behavior, because silently falling back to dynamic allocation would defeat the deterministic design. Failure should instead be detected and handled according to system requirements.
Real-time and non-real-time memory domains can also be separated architecturally. A control callback may operate only on fixed-size state structures and bounded buffers, while perception, diagnostics, configuration, and logging use conventional dynamic memory outside the critical path. Data exchanged between the domains can pass through preallocated queues, double buffers, or bounded shared structures, preventing flexible high-level software from disturbing deterministic control memory behavior.
Memory locking interacts with operating-system resource limits. Linux commonly restricts the amount of memory a process may lock, and insufficient privileges or configuration can cause locking operations to fail. A production ROS 2 application should check return values rather than assuming memory locking succeeded. System configuration, process limits, capabilities, and deployment settings must therefore be treated as part of the real-time software architecture.
Locking excessive amounts of memory can also harm the overall system. If a process pins more physical memory than necessary, other applications and kernel activities may experience memory pressure. Real-time engineering should therefore identify the working set required by critical execution and provision adequate physical RAM. Memory locking is a resource-management technique, not a reason to indiscriminately lock every available memory region on the robot computer.
The correct sequence of operations is important. A node typically creates required ROS 2 entities, allocates buffers, reserves container capacity, establishes communication resources, configures memory locking, and prefaults required pages before beginning critical periodic execution. After this transition, the real-time path should avoid operations that create new mappings or unpredictable allocations. Lifecycle-based initialization can provide a convenient architectural boundary for this preparation.
Thread creation also deserves consideration because every real-time thread has its own stack and scheduling context. Creating threads after memory preparation can introduce new stacks and resources that were not previously prefaulted. Critical worker threads should therefore be created before the deterministic phase, configured with suitable stack sizes, and exercised sufficiently to establish required memory pages before high-priority operation begins.
Monitoring page faults provides direct evidence about whether memory preparation is effective. Real-time validation can record minor and major page-fault counts before and during critical execution while simultaneously measuring callback latency and jitter. Ideally, the steady-state critical phase should show no unexpected faults associated with the real-time thread. Tracing and operating-system statistics can help correlate latency spikes with memory-management activity.
Stress testing remains necessary because allocation behavior may change under integrated workloads. High-rate sensor traffic, DDS communication, GPU pipelines, logging, and diagnostic activity can increase memory pressure even when the control callback itself performs no allocation. Tests should therefore reproduce realistic robot workloads and run long enough to expose rare events. Memory stability must be validated as a system property rather than only through inspection of one callback.
Physical AI systems make memory-domain separation especially valuable. AI inference pipelines may allocate large CPU and GPU buffers, change tensor shapes, load models, or perform asynchronous transfers, while low-level control requires stable timing. These workloads should not be assumed to share identical memory requirements. Deterministic control memory can remain fixed and resident while AI components use larger flexible memory regions through explicitly controlled interfaces.
Memory locking and pre-allocation should ultimately be viewed as complementary techniques. Locking keeps required pages resident, prefaulting ensures those pages are actually established before use, and pre-allocation prevents the critical path from repeatedly requesting new storage. None of these techniques alone guarantees real-time performance, but together they remove major sources of memory-related latency variation from a properly structured ROS 2 execution path.
A mature ROS 2 real-time memory design therefore establishes a clear transition from flexible initialization to controlled execution. Required memory is allocated, sized, locked, touched, and validated before critical callbacks begin, while runtime data structures remain bounded and reusable. Combined with disciplined rclcpp node design, executor isolation, PREEMPT_RT, and latency measurement, this approach creates a substantially more predictable memory environment for demanding robotic control workloads.
05.05 Real-Time Publisher Latency Measurement Automation [w/Code]
Real-time publisher latency measurement in ROS 2 is intended to determine how long information takes to travel from a defined publication event to a corresponding observation point and how consistently that delay remains bounded. A useful measurement system must capture more than average latency. It should expose jitter, percentile distributions, maximum observed delay, deadline violations, and rare latency spikes that can affect control behavior even when typical performance appears excellent.
The first requirement is to define the measurement boundary precisely. Publisher latency may refer to time spent inside publish(), middleware processing time, process-to-process delivery, host-to-host transport, or complete application-level latency from message generation to subscriber callback execution. These measurements describe different phenomena. Automated benchmarks should therefore record explicit measurement points so that results from different configurations remain meaningful and comparable.
A common application-level method places a timestamp and sequence identifier in each test message immediately before publication. The subscriber records another timestamp when the corresponding message reaches the selected observation point. The difference represents latency across the defined path, while the sequence number enables detection of lost, duplicated, reordered, or delayed samples. Additional timestamps can separate publisher-side processing, middleware transport, and callback scheduling effects when deeper analysis is required.
Clock selection is critical because measurement accuracy cannot exceed the quality of the time source. For communication within one computer, a monotonic clock is generally preferable because it is not affected by wall-clock corrections. For latency measurements across separate computers, clocks must be synchronized sufficiently well for the required resolution. PTP, hardware timestamping, or other synchronization mechanisms may be necessary when microsecond-level network measurements are expected.
Round-trip measurement provides an alternative when synchronized clocks are unavailable. A publisher sends a timestamped message to a responder, which returns it to the originating process, allowing round-trip time to be calculated using one local clock. This avoids direct clock-offset errors but measures two communication paths plus responder processing. Dividing round-trip time by two is only an approximation because forward and reverse delays are not necessarily symmetric.
Measurement instrumentation should introduce as little disturbance as possible. Printing every sample to the console, dynamically allocating result objects, writing each measurement immediately to disk, or performing complex statistics inside the subscriber callback can alter the latency being measured. A better pattern stores compact timestamps and sequence data in preallocated memory during the test and performs aggregation, formatting, visualization, and file export after the measurement interval.
Automating the benchmark requires reproducible control of message size, publication frequency, test duration, warm-up period, QoS configuration, executor model, and transport environment. A test controller can generate a matrix of configurations and execute each case repeatedly. Metadata should record ROS 2 distribution, rmw implementation, DDS configuration, operating-system kernel, CPU configuration, network topology, and relevant real-time settings so that results can be interpreted later.
A warm-up phase is important because startup behavior may differ substantially from steady-state operation. DDS discovery, endpoint matching, memory allocation, cache population, thread initialization, and network setup can produce temporary latency that does not represent sustained execution. Automated tests should therefore distinguish initialization latency from steady-state latency, while still preserving startup measurements when system activation time is itself an engineering requirement.
Publication rate should be varied systematically because latency often changes when the system approaches saturation. A publisher operating at 10 Hz may appear perfectly deterministic while the same path at hundreds or thousands of messages per second develops queue buildup and increasing message age. Rate-sweep automation can identify the region where throughput remains stable and the point where latency, loss, or deadline misses begin to increase rapidly.
Message size is another essential test dimension. Small control-state messages exercise scheduling and middleware overhead differently from camera frames, point clouds, tensors, or other large payloads. Automated tests should sweep representative payload sizes rather than extrapolating from a single message type. Large messages can introduce serialization cost, fragmentation, memory copying, transport buffering, and network congestion that are nearly invisible in small-message benchmarks.
QoS settings must be included in the experiment definition. Reliable and best-effort communication can exhibit different latency behavior under congestion or packet loss, while history depth affects queue accumulation. Deadline, durability, and other policies may also influence resource usage. Benchmark automation should preserve QoS configuration with every result so that latency values are never separated from the communication semantics under which they were measured.
Executor configuration can strongly influence subscriber-observed latency. A message may already be available in middleware while its callback waits behind another executable entity. Tests should therefore compare relevant executor configurations and callback workloads when application-level response time is important. Measuring only network or DDS transport latency can hide scheduling delays that dominate the actual behavior experienced by the robot control application.
Real-time operating-system configuration should also become an explicit benchmark variable. PREEMPT_RT, scheduling policy, thread priority, CPU affinity, CPU isolation, interrupt placement, memory locking, and power-management settings can each influence latency distributions. Automation makes it practical to compare baseline Linux against progressively tuned configurations while maintaining identical message workloads and statistical analysis.
Background load should be intentionally introduced because idle-system measurements provide limited evidence about production behavior. CPU stress, memory pressure, storage I/O, network traffic, sensor processing, logging, and GPU workloads can reproduce interference found in integrated robots. Each load profile should be defined and repeatable so that latency degradation can be attributed to known operating conditions rather than uncontrolled differences between test runs.
Latency statistics should preserve the distribution rather than report only a mean. Useful outputs include minimum, median, mean, standard deviation, high percentiles such as p95, p99, and p99.9, maximum observed latency, jitter, message loss, and deadline-miss counts. For real-time-oriented systems, tail behavior is particularly important because a small number of extreme delays can violate a control deadline even when millions of other samples perform normally.
Outliers should not automatically be removed because they may represent precisely the behavior the benchmark is intended to discover. A latency spike can originate from scheduling, page faults, interrupt storms, DDS retransmission, network congestion, or competing callbacks. Automated analysis should preserve anomalous samples and correlate them with tracing or operating-system events. Statistical filtering may be useful for separate descriptive analysis, but raw timing evidence should remain available.
Long-duration testing is required to characterize rare events. A benchmark lasting several seconds may collect many samples at high frequency yet still miss disturbances occurring only once every few minutes or hours. Automation enables extended tests without manual intervention and can divide results into time windows to detect drift, periodic disturbances, thermal effects, or workload-dependent changes in latency behavior.
Tracing can complement timestamp-based measurement by explaining where latency originates. ros2_tracing and operating-system tracing can expose callback scheduling, executor activity, middleware events, and kernel scheduling behavior. A practical automated framework can first detect abnormal latency samples using lightweight instrumentation and then use detailed traces in targeted experiments to identify the mechanism responsible for the observed delay.
Regression testing converts latency measurement from a one-time experiment into an engineering process. A known benchmark configuration can be executed after changes to ROS 2 packages, DDS versions, kernel settings, hardware, or application code. Results can be compared against stored baselines and predefined limits. This allows performance regressions to be detected before deployment instead of being discovered later during integrated robot testing.
Pass and fail criteria should be derived from application timing requirements rather than arbitrary benchmark numbers. A system may define a maximum acceptable latency, percentile threshold, jitter limit, permitted message-loss rate, or maximum number of deadline misses over a specified test duration. Automated evaluation can compare measured results with these requirements and generate a machine-readable verdict together with the complete statistical evidence.
The resulting framework should retain raw measurements, summarized statistics, configuration metadata, and test identifiers as one reproducible experiment record. CSV, structured logs, or other analysis-friendly formats can support subsequent comparison and visualization. Naming conventions and timestamps should allow engineers to trace every graph or report back to the exact software, hardware, QoS, workload, and real-time configuration that produced it.
For Physical AI robots, publisher latency tests should include interactions between deterministic control and variable AI workloads. GPU inference, perception pipelines, world models, and learned policies may create CPU, memory, and communication interference even when they execute asynchronously. Automated load scenarios can determine whether control-state communication remains within its timing limits while the complete AI workload is operating.
Real-time publisher latency automation therefore combines precise timestamping, controlled experiment generation, workload reproduction, statistical analysis, tracing, and regression criteria into a repeatable validation pipeline. Its purpose is not simply to produce the smallest latency number, but to demonstrate whether ROS 2 communication remains predictable under realistic conditions and to identify the configurations and interference sources that determine its timing behavior.
05.06 ROS2 with PREEMPT RT: Integration Optimization [w/Code]
PREEMPT_RT transforms Linux toward a more deterministic execution environment by making a larger portion of kernel activity preemptible and reducing the time that high-priority threads can be delayed by lower-priority work. For ROS 2, this does not create real-time guarantees automatically, but it provides an operating-system foundation on which carefully designed executors, callbacks, communication paths, and control loops can achieve substantially lower and more predictable scheduling latency.
The central objective of ROS 2 and PREEMPT_RT integration is not maximum computational throughput but bounded temporal behavior. A robot may have sufficient CPU performance while still experiencing occasional scheduling delays that violate control deadlines. PREEMPT_RT reduces important sources of kernel-induced latency, allowing engineers to focus more effectively on application-level interference, executor scheduling, memory behavior, middleware communication, and device-related timing.
Kernel configuration should be verified before application optimization begins. Engineers should confirm that the running kernel actually includes the required real-time preemption support and that the deployed system boots into the intended kernel. Distribution packages may simplify installation, while custom kernels provide greater configuration control. Regardless of installation method, kernel identity and configuration should become part of the recorded ROS 2 deployment and benchmark metadata.
PREEMPT_RT changes interrupt and locking behavior so that more kernel activities can be scheduled with real-time priorities. Many interrupt handlers become threaded interrupts, enabling their scheduling relationships with application threads to be controlled more explicitly. This improves latency control but also introduces configuration responsibility. Poorly assigned interrupt priorities or CPU placement can still interfere with a high-priority ROS 2 control thread.
Real-time scheduling policies such as SCHED_FIFO and SCHED_RR allow critical ROS 2 threads to receive stronger scheduling priority than conventional SCHED_OTHER tasks. SCHED_FIFO is frequently considered for deterministic control because a runnable higher-priority thread can preempt lower-priority work. However, incorrectly configured high-priority threads can monopolize CPU resources. Priorities must therefore reflect system timing dependencies rather than simply assigning the maximum value to every critical component.
Thread priority should be designed across the complete control chain. A high-priority executor thread provides little benefit if the data it requires depends on a lower-priority communication or device thread that cannot execute promptly. Conversely, middleware, interrupt, and application threads configured without a coherent priority model may create priority inversion or starvation. Priority assignment should therefore represent the actual producer-consumer and control dependencies of the robot.
CPU affinity can reduce migration and interference by binding critical threads to selected processor cores. A ROS 2 executor responsible for a time-sensitive control path may be pinned to a dedicated CPU while perception, logging, visualization, and background processing use other cores. This can improve cache locality and scheduling predictability, but affinity alone does not prevent interrupts, kernel threads, or unrelated processes from executing on the same CPU.
CPU isolation extends this concept by reducing general-purpose scheduler activity on selected cores. Kernel boot parameters and system configuration can reserve CPUs for critical workloads, while housekeeping tasks remain on other processors. Effective isolation requires a system-level view because timers, interrupts, work queues, and operating-system services may still generate interference. The goal is a controlled CPU domain rather than merely a thread pinned to an otherwise busy core.
Interrupt affinity is particularly important for networked and sensor-driven ROS 2 systems. Network interface controllers, storage devices, cameras, and other hardware generate interrupts that can compete with control threads. Engineers can distribute interrupts across CPUs so that high-volume devices do not disturb isolated control cores. However, interrupts associated with latency-critical devices may need deliberate placement near the thread consuming their data to minimize communication and scheduling overhead.
Memory locking complements PREEMPT_RT because reducing kernel scheduling latency does not eliminate page faults. Critical ROS 2 processes can lock required memory, prefault stack and heap pages, and preallocate buffers before entering deterministic execution. Without these measures, a high-priority thread may still encounter unpredictable memory-management delays. Real-time kernel tuning and real-time memory design should therefore be treated as parts of the same execution strategy.
Dynamic allocation inside critical callbacks should remain limited even on PREEMPT_RT. Creating messages, resizing containers, formatting strings, or repeatedly allocating temporary objects can introduce variable latency unrelated to kernel preemption. ROS 2 nodes should reuse buffers and messages, reserve container capacity, and separate flexible initialization from controlled execution. PREEMPT_RT improves the operating-system layer but cannot compensate for an unbounded application execution path.
Executor architecture remains equally important. A high-priority SingleThreadedExecutor can provide predictable serialized execution when it contains only a small number of bounded callbacks. If the same executor handles expensive perception, diagnostics, logging, and control functions, PREEMPT_RT cannot prevent one application callback from blocking another. Critical and non-critical ROS 2 workloads should therefore be separated through dedicated executors, processes, or carefully designed callback groups.
DDS and rmw behavior must also be considered because communication introduces threads, queues, serialization, memory operations, and network activity outside the immediate application callback. Different DDS implementations and transport configurations may produce different scheduling characteristics. QoS settings such as reliability and history depth can also affect queue behavior. PREEMPT_RT optimization should therefore measure the complete ROS 2 communication stack rather than focusing exclusively on executor threads.
Power management can create substantial latency variation. Dynamic frequency scaling, deep CPU idle states, and aggressive energy-saving policies may increase wake-up or execution latency when a critical thread becomes runnable. Systems requiring strict timing may use performance-oriented CPU-frequency policies and restrict selected idle states. These choices increase power consumption and thermal load, so they should be applied according to measured timing requirements rather than universally.
Simultaneous multithreading can also influence predictability because logical CPUs may share execution resources on the same physical core. A supposedly isolated real-time thread can experience interference when its sibling logical processor executes a computationally intensive workload. Critical deployments should understand the processor topology and, when necessary, avoid placing unrelated workloads on sibling threads associated with the same physical core.
Network timing requires additional attention when ROS 2 communication crosses machines. PREEMPT_RT can reduce host scheduling latency but cannot independently control switch queues, Ethernet congestion, packet retransmission, or clock offset. PTP or related synchronization mechanisms may be required for accurate one-way latency measurement, while traffic engineering and hardware timestamping can improve network timing analysis. Host real-time tuning is therefore only one part of distributed determinism.
Validation should begin with operating-system latency tests before evaluating the complete ROS 2 application. Tools designed to measure scheduling latency can establish whether the PREEMPT_RT platform itself behaves acceptably under CPU, memory, I/O, and interrupt stress. ROS 2 publisher-subscriber and control-loop benchmarks can then determine how much additional latency is introduced by executors, middleware, application callbacks, and communication paths.
Measurements should emphasize maximum and tail latency rather than average values alone. PREEMPT_RT optimization is successful when rare scheduling disturbances are reduced sufficiently to satisfy application deadlines, not merely when mean latency improves. Long-duration tests should record high percentiles, worst observed latency, jitter, deadline misses, and system load. Rare outliers should be investigated rather than automatically removed from the results.
Tracing is valuable when a latency spike remains unexplained. ros2_tracing can reveal executor and callback activity, while Linux tracing mechanisms can expose scheduling transitions, interrupts, wake-ups, and kernel execution. Correlating these observations helps determine whether a deadline miss originated in the ROS 2 application, DDS middleware, kernel scheduler, device interrupt activity, or another workload competing for CPU resources.
A reproducible optimization process should change one controlled group of parameters at a time. Engineers can establish a baseline with the standard configuration, enable PREEMPT_RT, apply scheduling priorities, introduce CPU affinity and isolation, tune interrupts, lock memory, and adjust power management while repeating the same workload. This progressive approach identifies which modifications actually improve latency and avoids attributing improvements to unrelated configuration changes.
Safety-critical functions require an architectural distinction between improved Linux real-time behavior and certified deterministic safety control. PREEMPT_RT can support demanding robot-control workloads, but it does not by itself convert ROS 2 and Linux into a certified safety controller. Emergency stopping, drive safety, protective monitoring, and other safety functions may require dedicated safety hardware or certified control environments independent of ROS 2 scheduling.
Physical AI robots particularly benefit from workload separation. GPU inference, world models, perception networks, mapping, and learned policies can generate substantial CPU, memory, PCIe, and network activity. Critical ROS 2 control execution should be isolated from these variable workloads where necessary. PREEMPT_RT, CPU partitioning, bounded communication, and asynchronous AI interfaces can allow deterministic control to coexist with computationally intensive intelligence.
The optimized architecture therefore combines several layers rather than relying on one kernel option. PREEMPT_RT reduces operating-system scheduling latency; CPU and interrupt placement reduce interference; memory locking and preallocation reduce memory-related disturbances; executor and callback design control application scheduling; and DDS configuration governs communication behavior. Determinism emerges from the coordinated behavior of all these layers.
Successful ROS 2 and PREEMPT_RT integration ultimately requires measurement-driven engineering. The target is not to make Linux theoretically free of latency, but to demonstrate that critical robot functions satisfy defined timing requirements under representative worst-case workloads. When kernel configuration, scheduling, CPU topology, memory, middleware, executors, and application design are tuned and validated together, ROS 2 can provide a substantially more predictable platform for demanding real-time-oriented robotic control.
05.07 micro.ROS: ROS2 on MCU [w/Code]
micro-ROS extends the ROS 2 communication model toward microcontrollers and deeply embedded devices that cannot practically run a full Linux-based ROS 2 stack. It allows resource-constrained controllers to participate in a ROS 2 system using familiar concepts such as nodes, publishers, subscriptions, services, and actions. Its purpose is not to reproduce every desktop ROS 2 capability on an MCU, but to connect deterministic embedded control with higher-level distributed robotics software.
A typical micro-ROS architecture separates the microcontroller application from the conventional ROS 2 network through the micro-ROS Agent. The MCU runs the micro-ROS Client, while the Agent usually executes on a Linux computer, embedded processor, or companion computer. The Agent bridges communication between the MCU-side middleware and the DDS-based ROS 2 domain, allowing embedded devices to appear as participants in the wider ROS 2 computational graph.
At the middleware level, micro-ROS commonly relies on the DDS-XRCE model designed for resource-constrained devices. Instead of requiring a microcontroller to execute a complete DDS implementation, the client communicates with an Agent that represents its entities within the DDS environment. This architecture significantly reduces memory, processing, and networking requirements on the MCU while preserving interoperability with ROS 2 applications operating on more capable computers.
The micro-ROS software stack follows the layered philosophy of ROS 2 while adapting it to embedded constraints. Application code can use the rclc interface, which provides C-based APIs suitable for microcontrollers and real-time applications. Beneath this layer are ROS client support, middleware adaptation, Micro XRCE-DDS communication, transport mechanisms, and the underlying RTOS or embedded platform. Each layer must be configured according to the MCU\'s available memory, timing requirements, and communication resources.
The rclc Executor is particularly important for real-time-oriented micro-ROS applications because it provides structured control over callback execution. Embedded developers often require explicit execution ordering and bounded behavior rather than unrestricted concurrency. Timers, subscriptions, and other callbacks can be organized around the control requirements of the device. The execution model should remain sufficiently simple that engineers can analyze when sensor acquisition, state processing, communication, and actuator commands occur.
Microcontrollers usually operate with far less memory than Linux-based ROS 2 computers, making memory discipline fundamental. Dynamic allocation during critical execution should be minimized or eliminated where practical. Message storage, communication buffers, executor resources, and application state should be prepared during initialization. Fixed-size or bounded structures are preferable when maximum resource requirements can be established in advance, reducing runtime allocation uncertainty and memory fragmentation.
This initialization-oriented design creates a useful transition between flexible setup and controlled runtime execution. The MCU can initialize its transport, create ROS entities, configure publishers and subscriptions, allocate required resources, and establish communication with the Agent before entering the operational control state. Once activated, the application should reuse previously prepared resources and avoid unnecessary creation or destruction of communication entities during high-frequency operation.
Transport selection depends strongly on the robot architecture. micro-ROS can operate over transports appropriate to embedded systems, including serial links and UDP-based networks, depending on platform support and deployment requirements. A directly connected MCU may communicate with a companion computer through a serial interface, while distributed controllers may use Ethernet or another network path. Bandwidth, latency, reliability, electrical environment, and failure behavior should guide transport selection.
Communication with the Agent introduces an important architectural boundary. Even if the MCU executes a highly deterministic local control loop, communication through the Agent and the broader ROS 2 network can experience variable latency. Therefore, control functions that require strict cycle-by-cycle timing should not depend on receiving a new ROS 2 message for every control iteration. The embedded controller should maintain essential local control behavior even when higher-level communication is temporarily delayed.
This principle naturally produces a hierarchical control architecture. Fast motor current, torque, velocity, or stabilization loops can remain on the MCU, while a Linux ROS 2 computer provides trajectory references, navigation commands, mission objectives, configuration, and supervision. The MCU becomes responsible for immediate physical interaction, whereas ROS 2 coordinates slower and computationally richer functions. Such separation aligns control frequency with the computational platform best suited to each task.
For example, an MCU may execute a motor-control loop at kilohertz frequencies while receiving velocity or trajectory references from ROS 2 at a substantially lower rate. The local controller interpolates, limits, validates, and applies these references according to its own deterministic timing. Sensor feedback required for immediate stabilization remains local, while selected state information is published upward for estimation, monitoring, planning, and diagnostics.
Timestamping is important when MCU sensor data participates in state estimation or synchronized perception. A timestamp should represent the time associated with the physical measurement as accurately as the architecture requires, rather than merely the time when a message eventually reaches a ROS 2 subscriber. Clock relationships between the MCU and host must therefore be understood. Systems requiring precise temporal alignment may need explicit synchronization or hardware-assisted timing mechanisms.
The distinction between communication time and physical event time becomes especially important in distributed robots. Encoder measurements, IMU samples, actuator states, and trigger events may travel through several software and transport layers before reaching the host. If timestamps are assigned too late, communication jitter becomes mixed with sensor timing. A robust design preserves timestamp provenance so higher-level ROS 2 components can reason correctly about when the underlying physical event actually occurred.
Quality of Service should reflect the semantics of embedded data rather than being selected uniformly. High-rate state data may prioritize freshness and bounded queue depth, while configuration or important command information may require stronger delivery behavior. Excessively deep queues can cause stale embedded data to arrive after it is useful. Conversely, inappropriate loss tolerance can be dangerous for commands whose semantics require confirmed handling. QoS should therefore be designed per communication path.
Agent availability must also be treated as a runtime condition rather than an assumption. The MCU should define what happens if communication with the micro-ROS Agent is interrupted, delayed, or restored. Depending on the robot, the embedded controller may continue local stabilization, hold the last valid reference for a bounded interval, transition to a predefined degraded mode, or initiate a safe stop. Reconnection should not produce uncontrolled command discontinuities or invalidate local safety logic.
Safety-related behavior should remain architecturally distinct from ordinary ROS 2 communication. micro-ROS can carry commands, state information, and diagnostic data, but communication availability alone should not be treated as a certified safety mechanism. Emergency stopping, drive enable, torque limits, watchdogs, and other safety functions may require dedicated hardware, safety-rated controllers, or independent communication paths according to the system\'s safety requirements.
Watchdogs are especially useful at the MCU boundary. A local watchdog can supervise control execution, command freshness, communication health, and internal software state. If expected references stop arriving, the MCU can apply a predefined fallback independently of the host. Similarly, the host can monitor MCU heartbeat or state information. This bidirectional supervision prevents either side from assuming indefinitely that the other component remains healthy.
Real-time operating systems provide a natural execution environment for many micro-ROS targets. An RTOS can schedule sensor acquisition, control, communication, and diagnostic tasks with defined priorities and periods. However, adding an RTOS does not automatically make the entire micro-ROS application deterministic. Shared resources, communication stacks, interrupt handling, callback execution, and application algorithms must still be analyzed to ensure that critical tasks meet their deadlines.
Interrupt design requires particular care because embedded control frequently depends on hardware timers, encoders, ADCs, communication peripherals, and sensor interrupts. Very high-frequency control should not be overloaded with ROS communication processing inside interrupt context. A better architecture typically keeps interrupt service routines short, captures necessary data or signals a task, and performs bounded control or communication work in appropriately scheduled execution contexts.
Performance measurement should include both local MCU timing and end-to-end ROS 2 communication. Local measurements can characterize control-loop period, execution time, interrupt latency, callback timing, and deadline misses. End-to-end measurements can characterize the path from physical sampling or command generation through the MCU, transport, Agent, DDS, host executor, and destination application. Separating these measurements helps locate the actual source of timing variation.
Resource monitoring is equally important because embedded systems operate close to fixed limits. Stack high-water marks, static and dynamic memory consumption, communication-buffer utilization, CPU load, queue occupancy, transport errors, and missed deadlines should be measured during representative operation. Testing only a minimal demonstration can hide resource exhaustion that appears when the full sensor, actuator, diagnostic, and communication workload runs simultaneously.
For Physical AI robots, micro-ROS provides a useful boundary between fast physical control and computationally intensive intelligence. AI perception, world models, planning, and learned policies can execute on Jetson, GPU, or on-premise computing resources, while MCUs maintain deterministic actuator and sensor interaction. High-level intelligence sends bounded references or goals rather than becoming directly responsible for every microsecond-scale hardware action.
This architecture also supports graceful degradation. If AI inference becomes temporarily slow, network traffic increases, or the ROS 2 host is restarted, the MCU can preserve essential control behavior according to locally defined rules. The physical system therefore does not need to inherit every timing variation of the AI pipeline. Explicit command validity, timeout policies, local limits, and state machines create a controlled boundary between asynchronous intelligence and deterministic physical execution.
A mature micro-ROS design consequently treats the MCU not as a miniature Linux ROS 2 computer but as a specialized real-time participant in a larger robotic architecture. The MCU owns bounded low-level behavior, the micro-ROS Agent provides integration with the DDS domain, and conventional ROS 2 nodes perform higher-level computation and coordination. Clear timing, memory, transport, synchronization, and failure boundaries allow these layers to cooperate without confusing connectivity with deterministic control.
The practical goal of micro-ROS is therefore to extend ROS 2 concepts down to resource-constrained embedded controllers while preserving the architectural strengths of each computing domain. When local control remains autonomous where necessary, resources are bounded, communication is explicitly designed, failures are handled locally, and end-to-end timing is measured, micro-ROS becomes a powerful bridge between deterministic MCU control and distributed ROS 2 intelligence.
05.08 ros2.control Real-Time Hardware Interface [w/Code]
ros2_control provides a standardized framework for connecting ROS 2 software to robot hardware while separating controller algorithms from device-specific implementation. Its real-time hardware interface is the boundary where joint commands, sensor states, actuator feedback, and physical devices enter the control architecture. The objective is to keep hardware access predictable and bounded so that controllers can execute at stable frequencies without inheriting unnecessary latency from higher-level ROS 2 processing.
The central ros2_control architecture is organized around the Controller Manager, controllers, hardware components, and exported command and state interfaces. The Controller Manager coordinates the control cycle and manages controller lifecycle and switching. Controllers consume state interfaces and produce command values, while hardware components translate those abstract interfaces into communication with motors, drives, encoders, force sensors, or other devices. This separation allows control logic to remain largely independent of hardware protocol details.
A typical real-time control cycle follows a read-update-write sequence. During read(), the hardware interface obtains the latest actuator and sensor states and exposes them to controllers. The Controller Manager then executes controller update() functions using those states and generates new command values. Finally, write() transfers the commands to the physical hardware. Maintaining bounded execution time across all three stages is essential because excessive delay in any stage extends the complete control-loop period.
The hardware abstraction is commonly represented through system, actuator, or sensor components according to the physical device architecture. A system interface can represent a multi-joint robot or integrated drive subsystem, while actuator interfaces can model individual controllable devices and sensor interfaces represent measurement-only hardware. The selected abstraction should reflect ownership of communication resources and synchronization requirements rather than simply mirroring the mechanical structure of the robot.
State interfaces expose measurements such as position, velocity, effort, temperature, force, or device-specific feedback to controllers. Command interfaces represent values that controllers can request, such as target position, velocity, effort, or other supported commands. These interfaces form a contract between controller and hardware implementations. Their semantics, units, update timing, valid ranges, and failure behavior should be explicitly defined so that the real-time control path does not depend on ambiguous interpretation.
Hardware initialization should be separated from cyclic real-time execution. Device discovery, memory allocation, communication setup, calibration loading, configuration parsing, and diagnostic initialization can involve unpredictable operations and should normally occur before the active control loop begins. Once activated, the hardware interface should operate using prepared resources and established communication channels, avoiding repeated construction of objects or dynamic reconfiguration within read() and write().
Lifecycle management provides a useful mechanism for controlling these transitions. Hardware components can move through initialization, configuration, activation, deactivation, and cleanup states so that resources are prepared before command execution becomes possible. Activation should verify that the device is ready for cyclic operation, while deactivation should transition outputs into an appropriate controlled condition. Lifecycle transitions themselves should remain outside the high-frequency deterministic path whenever possible.
Dynamic memory allocation should be minimized in the real-time hardware path. Repeated vector resizing, message construction, string processing, container growth, or temporary buffer allocation can introduce variable execution time and memory fragmentation. Buffers for device communication, joint state storage, and command data should preferably be allocated during initialization and reused. Fixed-size or bounded data structures are particularly useful when the maximum number of joints and communication frames is known.
Blocking operations are another major threat to predictable control. A read() call that waits indefinitely for a network packet or a write() operation that blocks until a slow device responds can stall every controller sharing the update loop. Hardware communication should therefore use bounded timeouts, non-blocking mechanisms, asynchronous buffering, or deterministic fieldbus behavior as appropriate. A missing device response should become an explicit error condition rather than an unlimited wait.
Many robot devices communicate asynchronously while the control loop operates periodically. In such cases, a dedicated communication thread can receive device data independently and place the newest complete state into a bounded shared buffer. The real-time read() function can then consume the latest valid snapshot without waiting for the physical transport. Commands can similarly be transferred from the control loop to a communication thread through a bounded real-time-safe exchange mechanism when synchronous device access would create unacceptable latency.
Synchronization between real-time and non-real-time threads must be designed carefully. Conventional mutexes can introduce priority inversion or unbounded waiting when a low-priority thread holds a resource needed by the control loop. Short bounded critical sections, double buffering, atomic state exchange, real-time buffers, or appropriately designed lock-free structures can reduce this risk. The chosen method should preserve data consistency without making the control thread dependent on unpredictable background execution.
The Controller Manager update rate defines an important timing target but does not by itself guarantee the actual loop period. Scheduler delays, controller execution time, hardware communication, page faults, interrupts, and other workloads can produce jitter. The real execution period should therefore be measured directly. Controllers should use the provided timing information correctly and avoid assuming that every update occurs at an ideal mathematical interval.
PREEMPT_RT can improve ros2_control timing by reducing Linux scheduling latency and making high-priority execution more predictable. The control thread may be assigned an appropriate real-time scheduling policy and priority, combined with CPU affinity or isolation where required. However, these operating-system techniques cannot compensate for blocking hardware drivers, excessive controller computation, dynamic allocation, or poorly structured communication. Determinism must be preserved across the entire read-update-write chain.
Hardware communication technology strongly affects achievable timing. EtherCAT, CAN/CAN-FD, serial protocols, industrial Ethernet, shared memory, PCIe devices, and vendor-specific APIs have different latency and synchronization properties. A real-time hardware interface should account for bus cycle timing, packet scheduling, driver behavior, buffering, and device response characteristics. The ROS 2 abstraction should not hide timing properties that are essential to safe and stable control.
For distributed drives, synchronized sampling and command application may be as important as communication latency. Multiple joints should ideally represent states from a coherent physical time when coordinated motion depends on them. Hardware clocks, distributed clocks, trigger signals, or synchronized fieldbus cycles can provide stronger temporal alignment than software arrival timestamps alone. Timestamp provenance should identify whether time represents physical sampling, driver reception, or ROS-side processing.
Command validation should occur before values reach the physical actuator. Position, velocity, effort, current, acceleration, and other limits can be enforced according to the robot design and controller architecture. Invalid numerical values, stale commands, discontinuities, or commands outside supported operating ranges should be detected predictably. The response may involve clamping, rejection, degraded operation, or transition toward a safe state depending on the hardware and safety requirements.
Command freshness is especially important when higher-level components generate references asynchronously. The hardware layer or controller architecture should know whether a command remains valid and what to do when updates stop. Holding the last command indefinitely may be inappropriate for a mobile base or manipulator. Timeouts, watchdogs, reference validity periods, and defined fallback behavior can prevent communication failures from becoming uncontrolled physical behavior.
Hardware errors should be propagated without performing expensive recovery operations inside the real-time cycle. read() or write() can detect communication loss, invalid state, device faults, or unavailable hardware and report an appropriate status. Complex reconnection, logging, device reconfiguration, or operator interaction should normally occur outside the critical loop. The immediate real-time responsibility is to detect the fault quickly and move the system toward a defined controlled response.
Controller switching also requires timing awareness because multiple controllers may claim overlapping command interfaces. The Controller Manager coordinates activation and deactivation so that ownership of hardware resources remains consistent. Switching should avoid intervals in which conflicting controllers command the same actuator or no valid controller owns a required interface. For demanding robots, the transition behavior and resulting actuator commands should be explicitly tested under motion.
Diagnostics and logging must not disturb the control path. High-frequency logging, string formatting, file access, or synchronous diagnostic publication inside read(), update(), or write() can produce significant jitter. The real-time side should record compact status values or counters into bounded storage, while non-real-time threads publish diagnostics and detailed logs asynchronously. This preserves observability without placing monitoring overhead directly inside the deterministic execution chain.
Performance validation should separately measure read time, controller update time, write time, and total cycle duration. Useful metrics include minimum, average, high-percentile, and maximum execution time, period jitter, deadline misses, communication errors, and stale-state occurrences. Long-duration testing under CPU, network, sensor, storage, and AI workloads is necessary because rare interference may not appear during short unloaded demonstrations.
Tracing can help locate the source of missed cycles. ros2_tracing and operating-system tracing can reveal controller execution, scheduling delays, middleware activity, interrupts, and competing threads. Hardware-specific timestamps can additionally expose bus or device latency. Correlating these layers makes it possible to distinguish a slow controller algorithm from a delayed hardware response, kernel scheduling event, communication interruption, or non-real-time software interference.
Safety functions should remain clearly separated from ordinary ros2_control timing guarantees. ros2_control can enforce operational limits and coordinate controlled behavior, but it should not automatically be considered a certified functional-safety mechanism. Emergency stop, safe torque off, protective limits, and safety-rated monitoring may require independent hardware, safety PLCs, certified drives, or safety communication according to the robot\'s required safety architecture.
In Physical AI robots, ros2_control forms a valuable deterministic boundary beneath perception, world models, planning, and learned policies. AI components can generate trajectories, references, or goals asynchronously, while the controller and hardware interface maintain a stable high-frequency physical loop. This prevents variable GPU inference or network timing from directly controlling every actuator cycle and allows intelligent behavior to coexist with bounded low-level execution.
A mature real-time ros2_control hardware interface therefore combines clear interface semantics, initialization-time resource preparation, bounded read-update-write execution, real-time-safe synchronization, deterministic hardware communication, explicit fault handling, and measurement-based validation. When these principles are coordinated with PREEMPT_RT, appropriate CPU scheduling, synchronized devices, and independent safety mechanisms, ros2_control becomes a robust bridge between ROS 2 software intelligence and predictable physical robot actuation.
05.09 Real-Time Robot Control Loop and ROS2 Integration Benchmark
A real-time robot control-loop benchmark should evaluate whether the complete sensing, computation, communication, and actuation chain can repeatedly satisfy defined timing requirements. The objective is not simply to measure ROS 2 message speed or controller execution time independently, but to characterize end-to-end temporal behavior. A useful benchmark therefore measures control period, execution latency, jitter, deadline misses, data age, and response consistency under realistic operating conditions.
The benchmark boundary must be defined before measurements begin. A control loop may start when a sensor physically acquires a sample, when a driver exposes that sample, when a ROS 2 callback receives it, or when the controller begins computation. Similarly, the endpoint may be command publication, hardware-interface write(), fieldbus transmission, or actual actuator application. Different boundaries produce different latency values, so every reported result should explicitly identify the measured path.
A representative integrated loop can be expressed as sensing, state acquisition, ROS 2 communication, controller execution, command generation, hardware transfer, and actuation. In ros2_control systems, this often corresponds to the read-update-write sequence, while distributed architectures may add DDS, network transport, micro-ROS, or external drive communication. Benchmarking should preserve these architectural stages so that total latency can later be decomposed into meaningful components.
Control-loop period is one of the primary measurements. If a controller is configured for 1 kHz operation, its nominal period is 1 ms, but actual activation intervals may vary because of scheduler activity, interrupts, communication delays, or competing workloads. The benchmark should timestamp consecutive control-cycle activations and calculate the observed period distribution. Mean frequency alone can hide timing instability, so minimum, maximum, percentile values, and period jitter should also be retained.
Execution time must be distinguished from loop period. A controller may execute its algorithm in only 100 microseconds while still being activated late because of operating-system scheduling or executor interference. Conversely, activation may be precise while controller computation occasionally exceeds its available budget. Separately measuring wake-up latency, read time, controller update time, write time, and total cycle execution makes it possible to identify which portion consumes the timing margin.
Deadline analysis converts raw timing measurements into control-relevant evidence. For a 1 ms control period, the architecture must define how much of that interval may be consumed by sensing, communication, computation, and command delivery. A deadline miss occurs when the required operation is not completed within its defined timing boundary. The benchmark should count misses, preserve their magnitude, and identify whether isolated extreme events or sustained overload caused them.
Jitter describes variation in timing rather than absolute latency and is especially important for feedback control. Even when average latency is acceptable, irregular sampling and command intervals can affect estimation, filtering, trajectory tracking, and stability. The benchmark should therefore measure both period jitter and end-to-end latency jitter. These quantities should not be combined because a loop can maintain a stable activation period while receiving sensor data with highly variable age.
Data age is another critical metric in ROS 2 integration. A controller may execute exactly on schedule but operate on a sensor sample that has spent significant time in queues or communication layers. Each measurement should therefore preserve the physical or acquisition timestamp when available and compare it with the time the data is consumed. This reveals stale-data problems that ordinary callback execution measurements may completely miss.
Timestamp provenance must be documented. Software timestamps taken after driver processing do not represent the same physical event as hardware timestamps captured at sensor acquisition. Distributed systems may also contain clock offsets between MCU, host, sensor, and drive computers. Accurate one-way latency analysis may require PTP, synchronized device clocks, hardware timestamping, or explicit clock-offset characterization. Timing precision should match the requirements of the control system being evaluated.
ROS 2 executor behavior should be included in the benchmark because callback scheduling can directly affect control timing. A dedicated executor containing only critical callbacks can be compared with an executor shared by perception, diagnostics, services, and other workloads. Single-threaded and multi-threaded configurations may exhibit different latency distributions. The purpose is not to declare one executor universally superior, but to determine how the selected execution architecture behaves for the target control workload.
DDS and QoS configuration should remain fixed and recorded for each benchmark case. Reliability, history depth, queue behavior, transport selection, and middleware implementation can change latency and data freshness. Reliable delivery may introduce retransmission-related delays under network disturbance, while deep histories can allow stale samples to accumulate. Benchmark results are meaningful only when the communication semantics associated with each measurement are preserved.
The operating-system configuration should also be part of the experiment definition. Standard Linux and PREEMPT_RT can be compared using identical control workloads, while scheduling policy, thread priority, CPU affinity, CPU isolation, interrupt placement, memory locking, and power management are progressively controlled. This creates a measurable optimization path and prevents a result obtained from a heavily tuned platform from being presented as a generic property of ROS 2.
Hardware communication must be represented realistically. EtherCAT, CAN/CAN-FD, industrial Ethernet, serial links, shared memory, and vendor APIs introduce different bus cycles and response characteristics. If the benchmark stops at the ROS 2 publisher or hardware-interface write() call, it may exclude an important portion of physical response latency. Where possible, measurements should extend toward the device boundary and identify when the actuator actually receives or applies the command.
A closed-loop benchmark provides stronger evidence than an isolated communication test. A known reference can be injected into the controller while the resulting actuator state is measured and returned through the sensing path. This allows command-to-response and sensor-to-command timing to be characterized together. Hardware-in-the-loop or controlled physical tests can expose synchronization, driver, and device behavior that software-only publisher-subscriber benchmarks cannot reproduce.
Background workload should represent the integrated robot rather than an idle laboratory computer. CPU stress, network traffic, storage activity, sensor processing, logging, visualization, GPU inference, mapping, and perception can all interfere with control execution. Several repeatable load profiles can be defined, ranging from idle baseline to expected operational load and intentionally severe stress. Timing degradation can then be related to measurable system interference.
Physical AI workloads deserve explicit benchmark scenarios because their timing behavior differs from traditional periodic control. GPU inference, learned policies, world models, and large perception pipelines may execute asynchronously and consume CPU, memory bandwidth, PCIe, and network resources. The control benchmark should verify that the deterministic path remains within its timing limits while these workloads execute concurrently rather than evaluating AI and control independently.
Long-duration testing is necessary because rare timing failures may not appear in short runs. A system that performs correctly for ten seconds can still experience scheduler stalls, thermal effects, network retransmissions, memory disturbances, or periodic background activity during hours of operation. Automated tests should therefore support both high-sample-rate short characterization and extended endurance runs while preserving every deadline miss and significant latency outlier.
Statistical reporting should emphasize the distribution and tail rather than a single average value. Minimum, median, mean, standard deviation, p95, p99, p99.9, maximum latency, jitter, deadline-miss count, and stale-data events provide a more complete description. For real-time-oriented control, the worst observed behavior and high percentiles often matter more than small improvements in average execution time because rare outliers can violate physical control requirements.
Tracing should be used to investigate abnormal benchmark samples. ros2_tracing can expose executor and callback activity, while Linux tracing can reveal scheduling transitions, wake-ups, interrupts, and kernel delays. Hardware or fieldbus traces can add device-level evidence. Correlating these timelines allows an observed control-loop spike to be attributed to application computation, ROS 2 scheduling, DDS communication, kernel behavior, hardware transport, or competing workloads.
Benchmark automation improves reproducibility by controlling configuration, workload, duration, repetitions, and result collection. Each run should preserve software versions, ROS 2 distribution, middleware implementation, kernel configuration, CPU topology, scheduling parameters, QoS, control frequency, hardware configuration, and load profile. Raw timing samples should remain available alongside summarized statistics so later analysis can verify conclusions and investigate unexpected behavior.
Pass and fail criteria should originate from robot requirements rather than arbitrary benchmark rankings. A mobile robot, manipulator, quadruped, or UAV may have very different acceptable control periods, maximum latency, jitter, and failure responses. The benchmark should compare measured behavior against explicitly defined limits and report the number and severity of violations. A configuration is acceptable only in relation to the timing requirements of the function being tested.
Safety behavior should be tested separately from ordinary performance. Communication loss, stale commands, delayed sensor data, controller overruns, and hardware faults can be deliberately introduced to verify watchdogs, command timeouts, degraded modes, and safe transitions. However, successful ROS 2 timing tests do not establish functional-safety certification. Safety-rated stopping and protective functions may still require independent hardware and certified control mechanisms.
A useful comparison methodology begins with a baseline and introduces changes systematically. PREEMPT_RT, CPU isolation, executor separation, memory locking, DDS tuning, hardware-interface optimization, and workload partitioning can each be evaluated using the same reference scenario. This makes the effect of each optimization visible and avoids drawing conclusions from configurations in which many variables changed simultaneously.
The final benchmark should connect timing measurements to robot-level behavior. Control-loop jitter matters because it can influence trajectory error, stabilization, actuator smoothness, or estimation quality. Communication latency matters because it increases the age of information used for decisions. By correlating software timing with physical performance, engineers can determine whether a measured latency improvement produces a meaningful improvement in the actual robotic system.
A mature ROS 2 real-time integration benchmark therefore combines control-loop timing, end-to-end communication, hardware response, synchronized timestamps, realistic workloads, statistical tail analysis, tracing, fault injection, and repeatable configuration management. Its purpose is to demonstrate that the complete robot control architecture remains predictably within defined temporal limits, providing measurable evidence that ROS 2, real-time Linux, controllers, middleware, and physical hardware operate as one coordinated system.
05.10 ROS2 Real-Time Monitoring Dashboard [w/Code]
A ROS 2 real-time monitoring dashboard should provide continuous visibility into whether critical robotic software is meeting its timing requirements during actual operation. Its purpose is not merely to display CPU usage or message frequency, but to expose the temporal health of the complete control architecture. The dashboard should combine control-loop periods, execution latency, jitter, deadline misses, data age, communication status, and resource utilization into an operational view that engineers can interpret quickly.
Monitoring begins with clearly defined timing signals. Critical nodes, executors, controllers, hardware interfaces, and communication paths should expose measurements that represent meaningful events rather than arbitrary software activity. Examples include control-loop activation, callback start and completion, sensor acquisition, command generation, hardware read and write, and actuator update. Consistent event definitions allow dashboard metrics to correspond directly to the robot\'s real-time architecture.
Control-loop monitoring should show both the requested period and the measured period. A controller configured for 1 kHz nominal operation should therefore report whether actual cycles remain close to the expected 1 ms interval. Current period, average period, maximum period, high-percentile values, and jitter can be displayed together. A frequency indicator alone is insufficient because an average of 1 kHz can still contain individual cycles that significantly violate the required timing boundary.
Execution-time monitoring should decompose the critical cycle whenever possible. For ros2_control, separate measurements for read(), controller update(), write(), and total cycle duration can identify where timing budget is consumed. For conventional ROS 2 nodes, callback execution, executor waiting, middleware transfer, and application processing may be monitored separately. This decomposition allows an operator to distinguish computational overload from communication or scheduling delays.
Deadline monitoring converts timing information into an immediately understandable operational condition. Each critical task or communication path can define a maximum acceptable completion time, after which a deadline miss is recorded. The dashboard should show recent misses, cumulative counts, miss rate, and severity. Repeated minor violations and a single extreme violation may have different engineering implications, so the magnitude and temporal distribution of misses should remain available.
Jitter should be visualized as a distribution rather than only a single value. Period jitter, callback latency jitter, and end-to-end communication jitter describe different properties and should remain distinguishable. Trend plots can reveal whether variability increases under load, while percentile and maximum indicators expose tail behavior. For feedback control systems, changes in timing variance may become significant before average execution time shows a noticeable problem.
Data-age monitoring is essential because timely execution does not guarantee timely information. A controller can run exactly on schedule while consuming sensor data that has remained in middleware or queues for too long. The dashboard should compare acquisition timestamps with consumption timestamps where possible and report current and maximum data age. Stale-data counters or age thresholds provide a direct indication that communication remains active but information is no longer sufficiently fresh.
Timestamp provenance should accompany timing metrics when multiple clocks are involved. Measurements originating from sensor hardware, MCU firmware, Linux hosts, ROS 2 nodes, or distributed drives may represent different clock domains. The monitoring architecture should identify whether timestamps originate from hardware sampling, driver reception, ROS callback execution, or another stage. PTP synchronization status and clock-offset information can be monitored when precise distributed timing is required.
ROS 2 communication health should include topic rates, message loss, queue behavior, DDS status, and QoS-related events. A dashboard can compare expected and observed publication rates while tracking delayed or missing samples. Reliable and best-effort paths may require different interpretations. Queue depth or stale-message indicators are especially valuable because a communication channel can maintain throughput while accumulating latency that makes the delivered data operationally useless.
Executor monitoring helps reveal scheduling interference inside ROS 2 processes. Critical callbacks can be monitored for activation delay, execution duration, and waiting time, while executor utilization indicates how heavily callback processing is loaded. When critical and non-critical callbacks share an executor, correlated latency spikes can reveal interference. Dedicated real-time executors can similarly be observed to verify that separation actually produces the expected timing behavior.
Operating-system metrics provide context for application-level timing. CPU utilization, per-core load, scheduling latency, real-time thread state, context switches, interrupt activity, memory faults, and CPU frequency can help explain abnormal control behavior. On PREEMPT_RT systems, the dashboard may additionally track real-time thread priorities and CPU affinity. These measurements should support diagnosis without overwhelming the primary view with low-level implementation details.
Memory monitoring should focus on events that can threaten deterministic behavior. Page faults, unexpected allocation growth, swap activity, memory pressure, and buffer exhaustion can produce latency spikes even when average memory usage appears acceptable. Real-time applications that use memory locking or preallocation can expose status indicators confirming that required resources remain available. The objective is to detect memory behavior that changes timing rather than simply display total RAM consumption.
Hardware-interface monitoring should extend observability toward the physical device boundary. ros2_control read and write timing, fieldbus cycle duration, communication errors, device response latency, stale actuator states, and missed hardware updates can be displayed alongside controller metrics. EtherCAT, CAN/CAN-FD, serial, or vendor-specific interfaces may require different measurements, but the dashboard should preserve a common distinction between software timing and physical communication timing.
The dashboard should distinguish real-time critical signals from general diagnostics. A small set of primary indicators can represent control health, such as loop deadline status, maximum latency, jitter, data freshness, hardware communication, and command validity. Secondary views can provide CPU, DDS, executor, memory, network, and trace details. This hierarchy prevents a large number of diagnostic values from obscuring the few conditions that determine whether the robot is operating within its timing envelope.
Thresholds should be derived from system requirements rather than arbitrary dashboard conventions. A 1 ms limit may be critical for one control loop and irrelevant for another process running at 10 Hz. Warning and fault thresholds should therefore be associated with specific functions, frequencies, and timing budgets. The dashboard can represent normal, warning, and violation states, but the numerical limits should remain configurable and traceable to engineering requirements.
Trend visualization is important because many real-time problems develop gradually. Increasing maximum latency, rising data age, growing queue occupancy, or repeated deadline misses can indicate approaching saturation before complete failure occurs. Time-series views allow engineers to correlate these changes with robot activities such as acceleration, mapping, sensor activation, logging, or AI inference. Historical context transforms individual alarms into interpretable system behavior.
Statistical windows should support both short-term operation and long-term characterization. A rolling window can calculate median, mean, p95, p99, p99.9, and maximum latency over recent samples, while cumulative statistics preserve rare events across longer sessions. Resettable experiment windows are useful during tuning and benchmarking. The dashboard should make clear whether a displayed statistic represents the last few seconds, a complete mission, or the entire runtime.
Alerting should focus on actionable timing conditions. Examples include repeated deadline misses, excessive data age, communication loss, controller overrun, clock synchronization failure, hardware timeout, or persistent CPU saturation. Alerts should include enough context to identify the affected component and timing requirement. Excessive alert generation should be avoided because alarm flooding can make genuinely critical real-time failures difficult to identify during robot operation.
Monitoring itself must not become a source of real-time interference. Critical threads should avoid synchronous dashboard updates, heavy serialization, database writes, or high-frequency logging. Instead, real-time code can publish compact counters, timestamps, or snapshots through bounded buffers, while non-real-time monitoring threads perform aggregation and visualization. Sampling rates should be selected so that observability does not materially alter the system being observed.
A practical monitoring architecture therefore separates data acquisition from dashboard presentation. Instrumented ROS 2 nodes, controllers, hardware interfaces, operating-system probes, and DDS diagnostics can feed a monitoring collector. The collector aggregates metrics, calculates statistics, evaluates thresholds, and stores selected history. A visualization layer then presents current status and trends. This separation allows dashboard technology to change without modifying the deterministic control implementation.
ros2_tracing can complement continuous metrics when deeper investigation is required. The dashboard may identify the time of an abnormal event and provide a trigger or reference for detailed traces. Trace data can then reconstruct executor scheduling, callback execution, DDS activity, thread scheduling, interrupts, and other events around the anomaly. Continuous lightweight monitoring and event-driven tracing provide greater diagnostic value than attempting to record every low-level event permanently.
Benchmark results can also be integrated into the monitoring system as reference envelopes. Expected latency distributions, maximum acceptable jitter, and normal resource ranges obtained during validated testing can become operational baselines. Live measurements can then be compared with these envelopes to identify degradation. This connects laboratory real-time benchmarking with deployed robot monitoring and helps detect changes introduced by software, configuration, hardware, or environmental conditions.
Configuration metadata should accompany stored monitoring data. ROS 2 distribution, middleware implementation, kernel version, PREEMPT_RT status, executor configuration, QoS settings, controller frequency, CPU affinity, hardware version, and software build identifiers can strongly influence timing. Without this information, historical graphs may show a performance change without revealing that the underlying execution environment changed at the same time.
For Physical AI systems, the dashboard should explicitly correlate deterministic control metrics with asynchronous intelligence workloads. GPU utilization, inference activity, perception pipelines, world-model updates, learned-policy execution, and network transfers can be observed alongside control-loop timing. The purpose is not to force AI workloads into deterministic schedules, but to verify that their resource consumption does not cause the critical physical-control path to exceed its defined temporal limits.
Fleet and multi-robot systems require an additional aggregation level. Each robot can maintain its own local real-time health metrics while a fleet dashboard summarizes deadline violations, communication health, control degradation, and hardware faults across many units. Local monitoring should continue even when cloud or fleet connectivity is unavailable. Higher-level aggregation provides operational awareness, but it should not become a dependency for immediate low-level control.
Safety-related indicators should remain clearly distinguished from ordinary performance monitoring. The dashboard may display emergency-stop state, watchdog status, command timeout, safe-stop transitions, or safety-controller communication, but visualization itself is not a safety mechanism. Certified safety functions must continue to operate independently according to the robot\'s safety architecture even if the ROS 2 monitoring system, network, or graphical interface fails.
A mature ROS 2 real-time monitoring dashboard therefore combines lightweight instrumentation, timing distributions, deadline analysis, data freshness, executor behavior, DDS communication, operating-system metrics, hardware-interface status, historical trends, alerts, tracing, and configuration metadata. By correlating these signals with Physical AI workloads and robot behavior, it transforms real-time performance from an offline benchmark result into continuously observable operational evidence that the robotic system remains within its designed timing envelope.
Author: Richard Minsoo Go
LinkedIn: Richard Minsoo Go










Top comments (0)
Some comments may only be visible to logged-in visitors. Sign in to view all comments.