Real Time Operating Systems — Ch.04: RT Linux PREEMPT RT [EN]

STATERA Library · Series 08 · Volume 03 · Chapter 04 · EN
Source: https://github.com/Statera-Library/08_Vol_03_Real_Time_Operating_Systems/blob/main/Chapter_04_RT_Linux_PREEMPT_RT/Chapter_04_RT_Linux_PREEMPT_RT_EN.md

Volume 03 Real Time Operating Systems

04. RT Linux / PREEMPT RT

04.01 Linux RT Patch History: PREEMPT RT Evolution

The PREEMPT_RT project emerged from the long-standing effort to make general-purpose Linux suitable for applications that require bounded and predictable response times. Traditional Linux was designed primarily for throughput, fairness, and broad hardware support rather than strict timing guarantees. Early real-time users therefore relied on external patches, dual-kernel approaches, or specialized operating systems when deterministic behavior was required.

During the 1990s and early 2000s, several projects attempted to overcome Linux latency limitations. RTLinux and RTAI introduced a small real-time execution layer beneath the normal Linux kernel, allowing critical tasks to run independently of Linux scheduling. These approaches achieved very low latency, but they created a conceptual separation between the real-time environment and ordinary Linux applications, increasing development and integration complexity.

The Linux 2.6 kernel represented an important transition toward improved native responsiveness. Kernel preemption, the O(1) scheduler, high-resolution timers, improved interrupt handling, and other infrastructure gradually reduced scheduling latency. The PREEMPT_RT effort extended this direction by attempting to transform Linux itself into a highly preemptible operating system rather than placing a separate real-time kernel underneath it.

A fundamental objective of PREEMPT_RT was to remove long sections of kernel execution during which high-priority tasks could not run. Many traditional spinlocks were therefore converted into sleeping locks based on real-time mutex mechanisms. This transformation allowed higher-priority threads to preempt kernel activities that would previously have blocked execution, substantially improving worst-case scheduling latency under system load.

Threaded interrupt handling became another central element of the PREEMPT_RT architecture. Conventional Linux executes much interrupt processing in interrupt context, where normal task scheduling cannot occur. PREEMPT_RT moves most interrupt handlers into schedulable kernel threads. Administrators can consequently assign priorities and CPU affinities to interrupt processing and coordinate device activity with application-level real-time threads.

Priority inheritance became increasingly important as more kernel synchronization mechanisms became preemptible. Without it, a low-priority task holding a required lock could indirectly block a high-priority task while medium-priority work continued executing. Real-time mutexes introduced priority inheritance behavior that temporarily raises the effective priority of the lock owner, reducing unbounded priority inversion within critical kernel paths.

High-resolution timers and dynamic tick mechanisms also contributed to the evolution of real-time Linux. Earlier kernels were strongly influenced by periodic timer ticks, limiting timer precision and introducing unnecessary scheduling activity. High-resolution timers enabled much finer wake-up timing, while tickless operation reduced periodic interruptions. Together, these mechanisms improved the ability to execute periodic control tasks with lower timing variation.

For many years PREEMPT_RT remained a maintained patch set rather than a completely integrated Linux configuration. Developers continuously moved individual mechanisms upstream, including mutex infrastructure, high-resolution timing, interrupt threading support, locking changes, and scheduler improvements. This incremental strategy reduced the difference between the standard kernel and the real-time patch while allowing each subsystem change to undergo broader Linux community review.

The gradual upstreaming process was technically difficult because kernel synchronization assumptions existed throughout device drivers, networking, memory management, filesystems, and architecture-specific code. A change that improved preemptibility could expose race conditions or invalid locking behavior that had remained hidden in conventional kernels. Consequently, PREEMPT_RT evolution involved not only scheduler optimization but extensive restructuring of Linux concurrency mechanisms.

By the Linux 5.x and 6.x generations, the gap between mainline Linux and PREEMPT_RT had become substantially smaller. Many technologies originally required by the real-time patch had already entered the upstream kernel, and remaining changes increasingly concentrated on specialized locking and execution paths. This maturation made real-time Linux easier to maintain and increasingly practical for industrial automation, robotics, audio, telecommunications, and embedded control.

A major historical milestone arrived when the remaining core PREEMPT_RT functionality was accepted into mainline Linux for the Linux 6.12 development cycle. This represented the culmination of more than two decades of engineering and upstream integration. Real-time capability was no longer best understood as an externally maintained transformation of Linux, but as functionality incorporated into the mainstream kernel development process.

Mainline integration does not mean that every Linux installation automatically behaves as a deterministic real-time system. Appropriate kernel configuration, real-time scheduling policies, memory locking, CPU isolation, interrupt affinity, power-management control, driver selection, and application architecture remain essential. PREEMPT_RT improves the kernel\'s preemption characteristics, but complete system determinism depends on the behavior of the entire hardware and software stack.

For robotics, this evolution is particularly important because modern robots combine workloads with very different computational characteristics. Control loops may require sub-millisecond or millisecond-level timing, while perception, SLAM, ROS 2 communication, logging, networking, and AI inference benefit from the rich Linux ecosystem. PREEMPT_RT allows many of these workloads to coexist on a common Linux platform while providing stronger scheduling control for time-critical threads.

The resulting architecture occupies an important position between small MCU-oriented RTOS platforms and conventional general-purpose Linux. FreeRTOS or Zephyr may remain preferable for tightly bounded low-level motor and safety control, whereas PREEMPT_RT provides Linux APIs, networking, filesystems, device support, containers, ROS 2, and accelerator integration together with significantly improved real-time behavior. This makes it especially attractive for robot edge computers.

In a practical robot architecture, PREEMPT_RT should therefore be viewed as an enabling foundation rather than a complete real-time solution. Developers must characterize worst-case latency with tools such as cyclictest, analyze scheduling and interrupt behavior, eliminate page faults from critical paths, control CPU contention, and validate timing under realistic stress. These engineering practices convert kernel-level preemptibility into measurable application-level determinism.

The historical progression from external real-time layers, through the PREEMPT_RT patch series, to extensive upstream integration illustrates a broader change in Linux engineering. Real-time capability gradually evolved from a specialized modification into a mainstream kernel concern. This evolution enables modern robotic systems to combine deterministic control requirements with the scalability and software ecosystem of Linux on increasingly powerful multicore edge platforms.

04.02 PREEMPT RT Kernel Compile and Configuration [w/Code]

Compiling a PREEMPT_RT kernel begins with selecting a Linux kernel version that provides the required real-time capabilities and matches the target platform. Modern kernels increasingly integrate PREEMPT_RT functionality directly into mainline Linux, while older versions may require a corresponding RT patch set. Version compatibility among the kernel source, RT patch, architecture, compiler, and device drivers must be verified before configuration begins.

The kernel source is normally obtained from the official Linux kernel tree or from a distribution or hardware vendor that maintains an appropriate real-time branch. When an external PREEMPT_RT patch is required, its version should closely match the selected kernel release. Applying mismatched patches can produce rejected hunks, compilation failures, or subtle runtime problems because synchronization and interrupt-handling code changes significantly between kernel versions.

A practical build environment requires the compiler toolchain, development libraries, configuration utilities, and packaging tools appropriate for the target architecture. Native compilation is straightforward when the build host and robot computer use the same processor architecture, while cross-compilation is commonly used for ARM-based embedded platforms. The build environment should remain reproducible so that deployed kernels can later be rebuilt, audited, and validated.

Kernel configuration usually starts from an existing distribution or vendor configuration rather than from an empty configuration. This preserves essential platform support for storage controllers, networking devices, GPU interfaces, USB devices, CAN interfaces, and other robot hardware. The configuration can then be modified using tools such as menuconfig, allowing real-time functionality to be enabled while unnecessary components are removed or converted into modules when appropriate.

The central configuration decision is the kernel preemption model. A PREEMPT_RT-enabled build selects the real-time preemption option so that kernel execution becomes highly preemptible. This affects locking behavior, interrupt execution, and scheduling responsiveness throughout the system. The objective is not simply to make the average execution time faster, but to reduce long non-preemptible intervals that produce unpredictable worst-case latency.

High-resolution timer support is another important configuration element because periodic robot control tasks often require wake-up intervals much finer than traditional kernel timer ticks. High-resolution timers provide precise timing events for real-time threads and complement PREEMPT_RT scheduling behavior. Tickless kernel features can further reduce unnecessary periodic timer activity, particularly on CPUs reserved for deterministic control workloads.

Real-time scheduling support must also be available for applications using policies such as SCHED_FIFO and SCHED_RR. PREEMPT_RT improves kernel responsiveness, while these scheduling classes determine how high-priority application threads compete for CPU execution. Priority assignment must therefore be designed carefully because an incorrectly configured high-priority thread can monopolize a processor and prevent important system services or lower-priority tasks from running.

Interrupt configuration has a major influence on the effectiveness of the compiled real-time kernel. PREEMPT_RT converts most interrupt handling into schedulable threaded interrupt contexts, making IRQ priorities and processor placement manageable. Robot systems can use this capability to place motor communication, EtherCAT, CAN, sensor acquisition, or timing-related interrupts on selected CPUs while separating them from computationally intensive perception and AI workloads.

CPU architecture and symmetric multiprocessing options must be configured consistently with the target robot computer. Multicore systems provide opportunities to separate real-time control from general workloads, but kernel configuration alone does not establish this separation. Runtime mechanisms such as CPU affinity, IRQ affinity, CPU isolation, and scheduler affinity are normally combined with the compiled kernel to create dedicated execution resources for critical control threads.

Memory-related configuration also influences deterministic operation. Virtual memory is a fundamental Linux capability, but page faults and unpredictable allocation paths can introduce latency into critical loops. The kernel must support memory-locking mechanisms that allow real-time applications to keep important pages resident in physical memory. Applications can then combine memory locking with heap pre-allocation and controlled memory access to reduce timing disturbances after initialization.

Device-driver compatibility must be evaluated carefully because a real-time kernel cannot provide deterministic behavior when a driver contains long non-preemptible sections or generates excessive interrupt activity. Drivers for network interfaces, CAN adapters, storage devices, GPU systems, and specialized robot hardware should be tested under PREEMPT_RT. Vendor-specific kernel modules may require rebuilding whenever the kernel version or configuration changes.

After configuration, the kernel and required modules are compiled using the selected toolchain. Parallel compilation can significantly reduce build time on multicore development systems, but build success alone does not demonstrate real-time suitability. The generated kernel image, modules, configuration file, and related metadata should be retained together so that the exact software configuration deployed on a robot can be reproduced during debugging or certification activities.

Installation normally places the new kernel, modules, and boot-related files alongside the existing kernel rather than immediately replacing the known working configuration. Maintaining a fallback kernel is important because an incorrect configuration can prevent storage, networking, graphics, or other essential hardware from operating. The bootloader should therefore provide a method for selecting either the PREEMPT_RT kernel or a previously validated standard kernel.

After rebooting, the running kernel version and preemption mode should be verified before real-time testing begins. Kernel logs should also be inspected for driver failures, interrupt problems, firmware errors, or unexpected warnings. At this stage, the objective is first to confirm functional stability. Real-time tuning should only proceed after the system reliably boots and all hardware required by the robot operates correctly.

Runtime tuning transforms the compiled PREEMPT_RT kernel into a practical real-time platform. CPU frequency scaling, deep power-saving states, automatic IRQ balancing, background services, and uncontrolled kernel activity can increase latency variation. Depending on system requirements, selected CPUs may be isolated, interrupts explicitly assigned, performance-oriented CPU policies applied, and unnecessary services restricted to non-real-time processor cores.

Real-time performance must finally be measured under realistic system load rather than inferred from configuration settings. Tools such as cyclictest can characterize scheduling latency while CPU, memory, storage, networking, and application workloads stress the system. Tracing tools can then identify latency sources when unexpected peaks occur. Maximum observed latency and jitter are generally more significant than average latency for deterministic control applications.

For robotics, the resulting kernel configuration should reflect the actual workload architecture. A PREEMPT_RT edge computer may execute high-priority control and communication threads together with ROS 2, SLAM, perception, networking, logging, and AI inference. CPU partitioning and priority design can protect control execution while allowing computationally intensive workloads to use remaining resources, creating a practical bridge between real-time control and general Linux computing.

Kernel compilation and configuration should therefore be treated as part of a complete real-time engineering process rather than as a one-time software installation task. Source and configuration management, reproducible builds, driver validation, boot recovery, scheduling design, CPU and IRQ tuning, memory preparation, and latency measurement must operate together. Only after these elements are validated can PREEMPT_RT provide the predictable execution behavior required by demanding robotic control systems.

04.03 Real-Time Thread Design: SCHED.FIFO / SCHED.RR [w/Code]

Real-time thread design in Linux is based on assigning execution priorities and scheduling policies that allow timing-critical work to preempt less important activity. Under PREEMPT_RT, application threads can use POSIX real-time scheduling classes such as SCHED_FIFO and SCHED_RR. These policies operate above normal time-sharing tasks and are therefore fundamental mechanisms for implementing predictable robot control, communication, and acquisition loops.

SCHED_FIFO implements fixed-priority first-in-first-out scheduling. When a runnable SCHED_FIFO thread has a higher priority than the currently executing thread, it immediately becomes eligible to preempt that thread. Unlike conventional Linux scheduling, a running SCHED_FIFO thread does not receive a normal time slice. It can continue executing until it blocks, voluntarily yields the processor, terminates, or is preempted by a higher-priority real-time thread.

This behavior makes SCHED_FIFO particularly suitable for periodic control loops that must execute with minimum scheduling interference. A motor-control or EtherCAT communication thread can wake at a predefined period, perform bounded computation, transmit or receive data, and then block until its next activation. If execution time is carefully controlled, the thread receives highly predictable processor access while leaving sufficient CPU capacity for other system activities.

The absence of automatic time slicing also makes SCHED_FIFO potentially dangerous when incorrectly designed. A high-priority thread containing an infinite loop, excessive computation, blocking error, or missing sleep operation can monopolize a CPU. Lower-priority threads may then experience severe starvation. Real-time applications must therefore ensure that every high-priority execution path has bounded computation and predictable blocking or synchronization behavior.

SCHED_RR extends the fixed-priority real-time model with round-robin time slicing among runnable threads at the same priority. A SCHED_RR thread can still preempt lower-priority threads immediately, but when multiple threads share its priority, each receives CPU execution for a defined quantum before another equal-priority thread can run. This prevents one equal-priority thread from continuously occupying the processor.

The practical distinction between SCHED_FIFO and SCHED_RR therefore appears mainly when multiple runnable threads have identical priorities. SCHED_FIFO preserves execution until the running thread blocks, yields, or is preempted by a higher priority, whereas SCHED_RR periodically rotates execution among equal-priority threads. For tightly controlled periodic tasks, SCHED_FIFO is often preferred, while SCHED_RR can be useful when several equivalent real-time activities must share processor time.

Priority design is more important than simply selecting a scheduling policy. Linux real-time priorities form an ordered range in which higher numerical real-time priorities represent stronger scheduling precedence. Designers should map priorities to timing criticality rather than assigning high values indiscriminately. Emergency handling, hard control deadlines, deterministic communication, sensor acquisition, planning, logging, and background processing should receive priorities according to their actual timing constraints.

A typical robot may assign its highest application-level priorities to safety-related monitoring or tightly constrained actuator communication, followed by motor or motion-control loops. Deterministic network processing and time-sensitive sensor acquisition can occupy intermediate real-time priorities. ROS 2 processing, perception, logging, user interfaces, and AI inference may operate at lower priorities or under conventional scheduling when their deadlines are less restrictive.

Periodic real-time threads should be designed around absolute timing rather than repeated relative delays whenever possible. If a control thread simply sleeps for its desired period after completing each iteration, execution time becomes added to the sleep interval and timing drift accumulates. An absolute wake-up schedule references successive predetermined activation times, preventing normal execution-time variation from continuously shifting the control-loop phase.

For example, a 1 kHz control thread has a nominal period of one millisecond. Within that interval it may read feedback, update state, calculate a control command, exchange actuator data, and then wait for the next absolute activation time. The complete worst-case execution path must remain safely below the available period. Average execution time alone is insufficient because occasional overruns can violate control deadlines and destabilize timing behavior.

Thread synchronization must also preserve priority relationships. A high-priority control thread can be indirectly delayed when it waits for a mutex owned by a lower-priority thread. If unrelated medium-priority work then preempts the lock owner, priority inversion occurs. Priority-inheritance mutexes allow the lower-priority owner to temporarily inherit the waiting thread\'s priority, helping it release the shared resource quickly and bounding the inversion interval.

Blocking operations require equally careful consideration. File access, uncontrolled network communication, memory allocation, console output, and some device operations can introduce unpredictable delays. Critical threads should avoid such operations during periodic execution whenever possible. Logging and noncritical I/O can instead be transferred through bounded queues or preallocated buffers to lower-priority worker threads that perform potentially nondeterministic processing outside the control path.

Memory behavior must be designed together with scheduling. A correctly prioritized SCHED_FIFO thread can still experience large latency if it triggers a page fault or performs unpredictable dynamic allocation. Real-time processes commonly lock required memory into RAM, prefault stack regions, allocate buffers during initialization, and reuse fixed memory structures. These practices remove sources of timing uncertainty that scheduling policy alone cannot control.

Multicore processors provide another level of real-time isolation. CPU affinity can bind critical threads to selected processor cores, while CPU isolation and IRQ affinity can reduce interference from ordinary applications and unrelated hardware interrupts. A control CPU may execute only selected SCHED_FIFO threads and required device IRQ threads, while other cores handle perception, SLAM, ROS 2 services, networking, storage, and AI inference.

Interrupt threads must be considered part of the priority architecture under PREEMPT_RT. Because many hardware interrupts execute as schedulable threads, an application can be delayed if IRQ priorities are poorly coordinated with control-thread priorities. Conversely, assigning an application priority above an interrupt required to deliver its input data can produce unexpected behavior. Thread and IRQ priorities should therefore be designed as one integrated scheduling hierarchy.

CPU utilization must remain below a level that provides sufficient timing margin. Even when every task is individually bounded, several real-time threads can become runnable simultaneously and create contention. Worst-case execution time, activation period, blocking duration, interrupt workload, and scheduling interference should therefore be considered together. Real-time engineering focuses on whether every critical deadline can be satisfied under worst-case operating conditions.

Permissions and resource limits are also required when applications request real-time scheduling. Linux normally restricts the ability of ordinary processes to assign high real-time priorities because incorrectly configured threads can interfere with system operation. Deployment environments must deliberately configure appropriate capabilities, user limits, or service permissions while avoiding unnecessarily broad privileges for robot applications.

SCHED_FIFO and SCHED_RR should consequently be treated as components of a complete thread architecture rather than performance switches. Deterministic behavior results from bounded execution, correct priority ordering, predictable synchronization, controlled memory usage, CPU and IRQ placement, and careful handling of blocking operations. PREEMPT_RT provides the kernel foundation, while application design determines whether that foundation becomes a reliable real-time system.

For modern robotics, this architecture enables fast control and slower computational workloads to coexist on the same Linux computer. A 1 kHz actuator or communication loop can execute with SCHED_FIFO on an isolated CPU while ROS 2, SLAM, perception, planning, logging, and AI inference execute on other cores at lower priorities. Properly engineered scheduling therefore connects deterministic physical control with the broader Linux and robotics software ecosystem.

04.04 Memory Locking and Heap Pre-allocation Strategy [w/Code]

Real-time memory management is essential because deterministic scheduling alone cannot guarantee predictable execution when memory access introduces page faults, allocation delays, or kernel reclamation activity. A PREEMPT_RT thread may have the highest scheduling priority and still miss its deadline if required memory is not resident in physical RAM. Memory locking and heap pre-allocation therefore remove major sources of nondeterministic latency from critical execution paths.

Linux applications normally operate with virtual memory, where allocated address ranges are mapped to physical pages when they are actually accessed. This demand-paging mechanism is efficient for general-purpose workloads, but the first access to a page can trigger a page fault requiring kernel processing. Major faults may even involve storage access. Such unpredictable delays are unacceptable inside tightly bounded control loops that must repeatedly meet sub-millisecond or millisecond deadlines.

Memory locking prevents selected virtual memory pages from being paged out or otherwise becoming unavailable when a real-time thread requires them. POSIX interfaces such as mlock() can lock specific address ranges, while mlockall() can lock the process address space according to selected flags. Real-time applications commonly lock both existing mappings and future mappings so that important code, data, stacks, and buffers remain resident throughout deterministic operation.

Calling mlockall() alone does not guarantee that every required page has already been physically instantiated. Stack pages, dynamically allocated regions, and other mappings may still produce minor page faults when first touched. Real-time initialization should therefore actively access required memory before entering the critical operating state. Prefaulting forces page mappings to be established while unpredictable latency is still acceptable rather than during the control loop.

Thread stacks require particular attention because Linux can expand stack usage as functions call deeper levels or local variables consume additional space. A real-time thread that suddenly accesses previously untouched stack pages may generate faults at an undesirable moment. Applications can reserve sufficient stack capacity and deliberately touch stack memory during initialization, ensuring that the expected working region is physically backed before deterministic execution begins.

Dynamic heap allocation is another significant source of timing uncertainty. General-purpose allocators such as malloc() and free() optimize memory efficiency and average performance rather than strict worst-case latency. Allocation may involve allocator locks, metadata manipulation, page acquisition, fragmentation handling, or kernel interaction. Concurrent threads can also contend for allocator structures, causing execution time to vary significantly even when allocation sizes appear small.

Heap pre-allocation addresses this problem by obtaining required memory before real-time processing starts. The application estimates the maximum number and size of objects, messages, sensor frames, control structures, and communication buffers required during operation. These resources are allocated during initialization and then reused repeatedly. Once the real-time phase begins, critical threads avoid ordinary dynamic allocation and operate primarily on previously prepared memory.

Memory pools provide a structured implementation of this strategy. A fixed-size pool can contain preconstructed objects or blocks that are acquired and returned through bounded operations. Because the pool capacity is predetermined, runtime behavior becomes easier to analyze and fragmentation can be minimized. Pools are especially useful for robot messages, control commands, telemetry packets, sensor metadata, trajectory elements, and other objects with known maximum concurrency.

Fixed-size allocation is generally easier to make deterministic than variable-size allocation. When objects have predictable dimensions, a pool can maintain free elements through an indexed array, bitmap, ring, or bounded free list. Allocation then becomes selection of an available element rather than searching or extending a fragmented heap. The design must nevertheless define behavior for pool exhaustion instead of silently falling back to an unpredictable general-purpose allocator.

Pre-allocation should include communication structures as well as application objects. Queues, ring buffers, shared-memory regions, DMA buffers, network packet structures, and middleware message storage can all influence real-time execution. In a robotic control path, memory required for sensor input, state estimation, actuator commands, and inter-thread communication should ideally be established before the system transitions from initialization into deterministic operation.

ROS 2 and other middleware environments require special consideration because seemingly simple publish or subscribe operations may internally allocate memory. Real-time designs can use bounded message sizes, reusable message instances, preallocated middleware resources, loaned-message mechanisms, or carefully selected executors and communication paths. The objective is to prevent hidden allocation activity from appearing inside a control loop whose timing was otherwise designed to be deterministic.

Deallocation can also introduce nondeterministic behavior and should not automatically be considered safer than allocation. free() may modify allocator metadata, acquire internal locks, merge blocks, or eventually interact with memory-management mechanisms. Critical threads should therefore avoid frequent allocation-deallocation cycles. Reusing persistent buffers and returning objects to deterministic pools usually produces more stable timing behavior than repeatedly creating and destroying memory.

Multithreaded systems add allocator contention to the problem. A high-priority control thread may be delayed by a lower-priority thread that holds an allocator lock, producing a form of indirect blocking. Separating real-time memory resources from general-purpose application heaps reduces this dependency. Dedicated per-thread or per-function pools can further limit contention and make ownership relationships easier to analyze during worst-case latency evaluation.

Memory locking must be applied carefully because locked pages consume physical RAM that the operating system cannot freely reclaim. Locking excessive amounts of memory can reduce resources available to ROS 2, perception, SLAM, AI inference, logging, networking, and other applications. A robot edge computer should therefore reserve sufficient memory for deterministic workloads while maintaining enough capacity for the broader Linux system to operate reliably under maximum expected load.

Operating-system permissions and resource limits also affect memory locking. Linux restricts how much memory an ordinary process can lock, so deployment configuration may require appropriate RLIMIT_MEMLOCK settings or capabilities. These permissions should be deliberately assigned to the real-time service rather than broadly granted. Startup procedures should detect failed locking operations because continuing silently without the intended memory guarantees can invalidate real-time assumptions.

A robust initialization sequence separates preparation from deterministic execution. The process starts, loads libraries, creates threads, allocates pools and buffers, establishes communication resources, locks memory, prefaults stacks and working regions, initializes devices, and verifies resource availability. Only after these potentially nondeterministic operations are completed does the system raise real-time priorities and enter its periodic control state.

For a 1 kHz robot control loop, this separation is especially important because each iteration has only one millisecond available. Sensor and actuator buffers, state vectors, control matrices, communication objects, and diagnostic records needed by the loop can be allocated and touched during startup. Runtime execution then reads and updates these persistent structures without requesting new heap memory, significantly reducing latency variation caused by the virtual-memory subsystem.

Memory behavior must ultimately be validated rather than assumed. Page-fault counters, tracing facilities, performance tools, and application instrumentation can confirm whether faults or allocations occur after the real-time phase begins. Stress testing should combine memory pressure with CPU, network, storage, perception, and AI workloads. Unexpected latency peaks can then be correlated with memory events and eliminated through further pre-allocation or architectural separation.

Memory locking and heap pre-allocation therefore complement PREEMPT_RT scheduling, CPU isolation, IRQ affinity, and priority design. Scheduling determines when a real-time thread may execute, while deterministic memory design ensures that the thread does not unexpectedly wait for memory-management activity after it starts. Together, these mechanisms transform a fast Linux application into a more predictable real-time execution environment suitable for demanding robotic control.

04.05 CPU Isolation: isolcpus and IRQ Affinity [w/Code]

CPU isolation is a fundamental technique for reducing scheduling interference in PREEMPT_RT systems. Even when a real-time thread uses SCHED_FIFO with a high priority, Linux may execute ordinary tasks, kernel workers, timers, interrupts, or housekeeping activity on the same processor. Isolating selected CPU cores creates a more controlled execution environment in which timing-critical robot workloads experience fewer unpredictable interruptions.

Modern multicore processors make this strategy practical because different classes of workloads can be assigned to different cores. A robot computer may dedicate one or more CPUs to deterministic control while other CPUs execute ROS 2, perception, SLAM, planning, networking, logging, and AI inference. The objective is not necessarily to maximize processor utilization, but to reserve predictable computational capacity for tasks with strict deadlines.

The isolcpus kernel boot parameter has historically been used to remove selected logical CPUs from normal scheduler load balancing. When a CPU is isolated, ordinary tasks are less likely to be automatically migrated onto it by the scheduler. A real-time application can then explicitly bind its critical threads to that CPU using CPU affinity, creating a processor environment with substantially less interference from general-purpose workloads.

CPU isolation should not be interpreted as complete physical separation. The isolcpus mechanism primarily influences scheduler behavior and does not automatically remove interrupts, kernel threads, timers, workqueues, or every form of operating-system housekeeping. A supposedly isolated CPU can therefore still experience latency spikes unless other sources of interference are separately controlled. Effective real-time isolation requires several mechanisms to operate together.

CPU affinity determines which processors a particular process or thread may execute on. Real-time control threads can be pinned to dedicated CPUs so that the scheduler cannot migrate them among arbitrary cores. Preventing migration improves cache locality and makes execution behavior easier to analyze. Affinity is commonly configured through Linux APIs, command-line utilities, service configuration, or application-level thread initialization.

Normal application tasks should simultaneously be kept away from the real-time CPUs. If only the control thread is pinned but other workloads remain free to execute everywhere, background processes may still compete for processor resources. A complementary affinity policy can place ROS 2 services, logging, graphical interfaces, storage operations, and AI workloads on housekeeping CPUs, leaving selected cores primarily available for deterministic processing.

Hardware interrupts are another major source of interference. Network adapters, CAN interfaces, storage controllers, USB devices, GPUs, sensors, and other hardware generate IRQs that require CPU processing. If these interrupts execute on a real-time CPU without deliberate planning, they can interrupt control execution and increase jitter. IRQ affinity therefore determines which CPUs are allowed to service particular hardware interrupt sources.

PREEMPT_RT makes IRQ placement especially important because most interrupt handlers are transformed into schedulable threaded interrupts. These IRQ threads can be assigned CPU affinities and priorities, allowing hardware processing to participate in the overall real-time scheduling architecture. A control-related interrupt may intentionally share a real-time CPU with its application thread, while unrelated storage or network interrupts can be moved to housekeeping CPUs.

The relationship between a real-time application and its required IRQs must be analyzed rather than applying a rule that all interrupts should leave isolated CPUs. An EtherCAT, CAN, or sensor interface providing time-critical data may need its interrupt processing close to the associated control thread. Moving that IRQ elsewhere can introduce cross-core communication and scheduling delays. Isolation should therefore reflect data flow and deadline requirements.

Automatic IRQ balancing can conflict with deterministic IRQ placement. Linux systems often use irqbalance to distribute interrupts dynamically across processors for throughput and load distribution. This behavior is useful for general computing but may move interrupts onto CPUs reserved for real-time work. Real-time deployments commonly restrict or disable automatic balancing for selected IRQs and explicitly establish stable processor assignments.

Kernel housekeeping activity must also be considered. Timer callbacks, scheduler functions, read-copy-update processing, workqueues, and other kernel operations can generate disturbances even when normal tasks and device IRQs are controlled. Additional boot and runtime mechanisms may therefore be used to move appropriate housekeeping activity away from dedicated real-time CPUs, depending on the kernel version and required level of isolation.

The nohz_full mechanism can reduce periodic scheduler ticks on selected CPUs when conditions allow. Removing unnecessary ticks reduces repeated interruptions of long-running real-time threads and can improve jitter characteristics. However, full tickless operation has constraints and does not eliminate every timer or kernel event. It should be combined with workload design and measurement rather than assumed to provide deterministic behavior automatically.

RCU callback processing can similarly generate kernel activity that interferes with isolated CPUs. Real-time configurations may offload suitable RCU callbacks to housekeeping processors so that CPUs dedicated to control perform fewer unrelated kernel operations. As with other tuning mechanisms, the exact configuration depends on kernel capabilities, platform architecture, and application requirements and must be validated under realistic load.

Processor topology influences the effectiveness of isolation. Logical CPUs may share physical cores, caches, memory controllers, or simultaneous multithreading resources. Assigning a real-time task to one hardware thread while a computationally intensive task executes on its sibling can still create resource contention. CPU selection should therefore consider physical core boundaries, shared cache topology, NUMA organization, and simultaneous multithreading behavior.

Shared hardware resources cannot be completely controlled through scheduler affinity. Memory bandwidth, last-level cache, PCIe traffic, DMA, and interconnect activity from perception or AI accelerators can affect control workloads even when CPU cores are separated. A powerful AI inference task may therefore increase latency indirectly without ever executing on the real-time CPU. System-level stress testing is necessary to reveal these interference paths.

A practical robot architecture can divide CPUs into real-time and housekeeping groups. Real-time CPUs execute selected SCHED_FIFO control threads and the IRQ threads directly required by those control paths. Housekeeping CPUs handle normal Linux processes, ROS 2 noncritical nodes, networking services, storage, diagnostics, logging, and system management. Compute-intensive perception and AI processing can be assigned to additional CPU cores and GPU resources.

For a 1 kHz control loop, CPU isolation provides a protected one-millisecond execution window rather than additional computation time. The control thread must still have bounded worst-case execution time, preallocated memory, appropriate synchronization, and correctly prioritized device interrupts. Isolation reduces external interference, but it cannot compensate for an application that occasionally performs excessive computation or unpredictable blocking operations.

CPU isolation should be validated by measurement rather than configuration inspection alone. cyclictest and application-specific timing instrumentation can measure latency and jitter while stress workloads exercise CPUs, memory, networking, storage, and accelerators. Tracing facilities can identify unexpected IRQs, migrations, kernel threads, or scheduling events occurring on isolated CPUs and guide further tuning.

The final configuration should be documented as part of the robot software platform because CPU numbering, IRQ assignments, kernel command-line parameters, and hardware topology may change between computers. Automated deployment should reproduce affinity and isolation policies consistently. Monitoring should also detect configuration drift that could silently return interrupts or background workloads to processors intended for deterministic control.

CPU isolation, isolcpus, CPU affinity, and IRQ affinity therefore form a coordinated resource-partitioning strategy rather than independent performance options. Combined with PREEMPT_RT, real-time scheduling, memory locking, heap pre-allocation, and bounded application design, they reduce scheduling interference and improve worst-case latency. This enables Linux robot computers to support deterministic control while simultaneously running the rich perception, communication, and AI workloads required by modern Physical AI systems.

04.06 Real-Time Jitter Measurement: cyclictest Interpretation [w/Code]

Real-time jitter measurement determines how consistently a system can wake and execute a time-critical thread relative to its intended schedule. In robotics, average execution speed is less important than knowing whether occasional latency peaks can violate control deadlines. cyclictest is widely used with PREEMPT_RT Linux because it measures scheduler wake-up latency repeatedly and exposes minimum, average, and worst observed timing behavior.

cyclictest operates by creating one or more real-time threads that sleep until predetermined timer deadlines. When a thread becomes runnable, the tool compares the expected wake-up time with the actual time at which the thread executes. The difference represents scheduling latency. Repeating this measurement thousands or millions of times creates a statistical view of how reliably the operating system responds to periodic real-time events.

The measurement should not be interpreted as the complete execution time of a robot control loop. cyclictest primarily measures the delay between a requested wake-up instant and actual thread execution. Application computation, sensor communication, actuator transmission, synchronization, middleware processing, and device response introduce additional delays. The result therefore characterizes an important part of real-time behavior rather than the complete end-to-end control latency.

Typical cyclictest output includes minimum latency, average latency, and maximum latency for each measurement thread. Minimum latency shows the best observed scheduling response, while average latency describes typical behavior during the test. Maximum latency is usually the most important value for deterministic engineering because a single large scheduling delay can cause a periodic control task to miss its deadline even when the average latency appears excellent.

Jitter describes variation in timing rather than simply the absolute latency value. A system that repeatedly wakes with nearly identical latency has low jitter, while one that usually responds quickly but occasionally produces large spikes has poor temporal consistency. For robot control, stable timing can be more valuable than slightly lower average latency because controllers, communication cycles, and state estimators often assume relatively regular sampling and actuation intervals.

A useful cyclictest experiment should specify scheduling policy, priority, measurement interval, duration, and CPU placement deliberately. The test thread can use a real-time policy with a priority representative of the intended application. Its interval should reflect the target control period when practical. A 1 kHz control architecture, for example, has a nominal one-millisecond period, making microsecond-scale scheduling disturbances directly relevant to the available timing budget.

Short tests performed on an idle computer are insufficient for evaluating worst-case behavior. Real robots simultaneously execute perception, ROS 2 communication, networking, storage, logging, sensor drivers, and AI inference. cyclictest should therefore run for extended periods while representative stress workloads exercise CPU cores, memory, network interfaces, storage devices, and accelerators. The objective is to discover rare latency spikes that appear only under contention.

CPU affinity strongly affects measurement interpretation. If cyclictest moves among processors, results combine the behavior of several CPU cores and include migration effects. Pinning a measurement thread to the CPU intended for real-time control produces a more meaningful characterization. When CPU isolation is used, tests should compare isolated and housekeeping processors to verify whether the isolation strategy actually reduces maximum latency and jitter.

IRQ placement should also be preserved during testing because interrupt activity is part of the real operating environment. A result obtained after disabling nearly every device may look excellent but may not represent the deployed robot. Time-critical EtherCAT, CAN, encoder, or sensor IRQs should be configured as intended, while unrelated interrupts can be assigned to housekeeping CPUs according to the final architecture.

Histograms provide more information than a single maximum value. A latency histogram shows how frequently wake-up delays fall into different ranges and makes long-tail behavior visible. Most samples may cluster around a small latency while a tiny number occur far outside the normal distribution. These rare outliers are exactly the events that deterministic system design must investigate rather than hiding them inside an average value.

Maximum latency must be interpreted relative to the application deadline. A measured peak of several tens of microseconds may be insignificant for a 10 ms control period but consume a meaningful fraction of a 1 ms period. Scheduling latency is also only one component of the total budget. Worst-case application execution, synchronization, communication, IRQ handling, and safety margin must all fit inside the available period.

For example, a one-millisecond control cycle cannot safely allocate the entire millisecond to application computation. Part of the period must accommodate wake-up latency, sensor and actuator communication, blocking or synchronization effects, and engineering margin. If cyclictest reveals occasional large peaks, the remaining execution budget shrinks accordingly. Real-time design therefore converts measured latency into a timing-budget decision rather than treating it as an isolated benchmark score.

Unexpected latency spikes should trigger root-cause analysis. Linux tracing mechanisms can reveal whether the delay corresponds to an interrupt, scheduler event, kernel thread, timer, migration, driver activity, or other execution path. Tools such as ftrace and trace-cmd can complement cyclictest by showing what occurred around a latency event. Measurement identifies the symptom, while tracing helps locate the subsystem responsible for it.

Power-management behavior can produce significant variation in latency. CPU frequency scaling, deep idle states, thermal management, and transitions between power states may increase wake-up delay. Real-time systems frequently evaluate performance-oriented processor settings and restrict deep sleep states when necessary. However, these changes increase power consumption and thermal load, so the final configuration should balance timing requirements against platform constraints.

Simultaneous multithreading, shared caches, memory bandwidth, DMA traffic, and accelerator workloads can also influence results even when the real-time CPU is isolated. A perception pipeline or GPU workload may generate heavy memory and PCIe activity without executing directly on the control core. cyclictest under combined system stress is therefore valuable for detecting interference that cannot be predicted from CPU scheduling configuration alone.

A baseline comparison is useful during system development. Measurements can first be collected using the standard Linux configuration and then repeated after enabling PREEMPT_RT, CPU affinity, IRQ affinity, memory preparation, CPU isolation, nohz_full, or other tuning mechanisms. Comparing the distributions and maximum values shows which changes actually improve timing behavior and prevents configuration complexity from being accepted without measurable benefit.

Test reproducibility is equally important. Kernel version, PREEMPT_RT configuration, processor topology, CPU frequency policy, BIOS settings, IRQ assignments, cyclictest parameters, background workloads, temperature, and test duration should be recorded. Two latency numbers obtained under different conditions cannot be meaningfully compared without this context. Real-time validation should therefore produce a repeatable test profile rather than a single screenshot of favorable results.

Production validation should extend beyond cyclictest to the actual application. The robot control software can timestamp wake-up, sensor acquisition, computation completion, command transmission, and cycle completion. These measurements reveal whether kernel-level latency improvements translate into predictable end-to-end behavior. A system can perform well in cyclictest yet still suffer application jitter caused by locks, memory allocation, middleware, drivers, or communication.

The final acceptance criterion should consequently be based on worst-case timing requirements rather than an arbitrary target latency. The measured maximum scheduling delay, application worst-case execution time, communication latency, and required safety margin must remain below the control deadline under representative worst-case load. Long-duration testing is particularly important because rare events may not appear during short laboratory measurements.

cyclictest should therefore be understood as a diagnostic and validation instrument within a larger real-time engineering process. PREEMPT_RT provides kernel preemptibility, SCHED_FIFO establishes scheduling precedence, memory locking reduces page-related disturbances, and CPU and IRQ isolation reduce interference. cyclictest then measures whether these mechanisms collectively produce the predictable latency required by the robotic control architecture.

04.07 PREEMPT RT Debugging: ftrace, perf, LTTng [w/Code]

Debugging PREEMPT_RT systems requires a different mindset from conventional functional debugging because the software may produce correct results while occasionally violating timing requirements. A control loop that normally completes within its deadline can still fail because of a rare scheduler delay, interrupt storm, lock contention, or kernel execution path. Real-time debugging therefore focuses on reconstructing what happened immediately before and during abnormal latency.

The debugging process normally begins after latency measurement identifies an unexpected event. cyclictest or application-level timing instrumentation can reveal that a deadline or latency threshold was exceeded, but these measurements rarely explain the cause. Tracing and profiling tools such as ftrace, perf, and LTTng provide complementary views of kernel execution, CPU consumption, scheduling activity, interrupts, and application behavior.

ftrace is a tracing framework integrated directly into the Linux kernel and is particularly valuable for PREEMPT_RT analysis. It can record function execution, scheduler events, interrupts, wake-ups, context switches, and other tracepoints with relatively detailed timing information. Because the tracer operates close to the kernel, it can expose execution paths that are difficult to observe from normal application logs or user-space monitoring tools.

Function tracing allows developers to inspect which kernel functions execute around a latency event. However, tracing every function can generate enormous amounts of data and may itself influence system timing. Real-time debugging therefore benefits from selective tracing. Developers can restrict tracing to relevant CPUs, functions, processes, or event categories so that the captured information remains focused on the suspected source of interference.

Scheduler tracing is especially useful when investigating delayed real-time threads. Events related to thread wake-up, scheduling, migration, and context switching can reveal when a SCHED_FIFO task became runnable and when it actually received CPU time. The interval between these events can then be correlated with another thread, kernel activity, or interrupt processing that occupied the processor during the unexpected delay.

PREEMPT_RT makes interrupt tracing particularly important because most interrupt handlers execute as schedulable IRQ threads. A control thread can therefore interact with hardware processing through the same priority hierarchy used by other real-time threads. Traces can reveal whether an IRQ arrived at an unexpected frequency, executed on the wrong CPU, consumed excessive time, or had a priority relationship that delayed the application control thread.

Specialized latency tracers can help identify sections of execution that prevent timely scheduling. Depending on kernel configuration and version, tracing facilities can investigate interrupt-disabled regions, preemption-disabled regions, or wake-up latency. These measurements help distinguish a scheduling problem from a kernel path that temporarily prevents normal preemption, narrowing the investigation toward synchronization, driver, or architecture-specific behavior.

trace-cmd provides a convenient user-space interface for configuring, recording, and analyzing kernel tracing information associated with ftrace. Instead of manipulating tracing control files manually, developers can select events and CPUs, start a recording session, reproduce the latency problem, and inspect the resulting trace. This workflow is useful when creating repeatable diagnostic procedures for robot platforms.

perf provides a different but complementary perspective. Rather than primarily reconstructing detailed event sequences, perf is commonly used to analyze where CPU time is consumed, which functions execute frequently, and how hardware or software performance events behave. Sampling profiles can identify unexpectedly expensive application functions, kernel paths, driver routines, cache behavior, context switching, and other performance characteristics.

A real-time task may meet its scheduling requirements but still exceed its deadline because its own computation becomes too expensive. In this case, scheduler tracing alone may not reveal the primary problem. perf can show that a control algorithm, middleware component, serialization routine, or system call consumes significantly more CPU time than expected. Profiling therefore separates scheduling latency from excessive execution time inside the workload itself.

Hardware performance counters accessible through perf can also expose resource contention. Cache misses, CPU cycles, instructions, branch behavior, and related events may reveal why execution time changes under system load. For example, an isolated control thread can still experience longer computation when perception or AI workloads compete for shared cache or memory bandwidth, even though no competing task executes directly on its CPU.

LTTng, the Linux Trace Toolkit Next Generation, is designed for efficient tracing across both kernel and user space. It is useful when debugging complex robotic systems where a latency event crosses multiple software layers. A trace can include kernel scheduling and interrupts together with instrumented application events, allowing developers to follow activity through control software, middleware, communication, and operating-system execution.

This cross-layer capability is valuable for ROS 2 and distributed robotic software. A delayed actuator command may originate from scheduling interference, but it may also result from middleware processing, message handling, synchronization, or application logic. User-space tracepoints combined with kernel events can establish a common timeline and show where the delay accumulated instead of attributing every timing failure to PREEMPT_RT itself.

Tracing introduces overhead, so measurement methodology must consider observer effects. Recording too many events, enabling broad function tracing, or continuously writing large traces to storage can change the timing characteristics being investigated. A practical strategy starts with low-overhead measurements and progressively enables more detailed tracing around suspicious components. The diagnostic configuration itself should be documented together with the resulting latency data.

CPU affinity should remain consistent during debugging. If a real-time thread is intended to run on an isolated CPU, traces and profiles should preserve that deployment configuration. Relevant IRQ affinities should also remain realistic. Moving workloads merely to simplify tracing can eliminate the very interference being investigated and produce a technically clean trace that does not represent actual robot operation.

A useful investigation begins by establishing a precise trigger condition. The system can detect when control-cycle latency exceeds a defined threshold and preserve trace data surrounding that event. Ring-buffer tracing is particularly valuable because recent events can remain in memory until an anomaly occurs. The captured window then contains activity immediately before the spike, which is often more informative than tracing continuously for many hours.

The analysis should correlate multiple timing dimensions rather than search for a single suspicious function. Developers may compare thread wake-up time, actual scheduling time, IRQ arrival, context switches, lock acquisition, application execution, and control-cycle completion. This timeline helps determine whether the delay originated before the application started running, while it was executing, or while it was waiting for another subsystem.

Root causes discovered through tracing commonly lead to architectural corrections rather than isolated code fixes. An IRQ may need different affinity, a kernel worker may need to leave the real-time CPU, a mutex may require redesign, dynamic allocation may need removal, or a computational workload may require relocation. Profiling may similarly indicate that an algorithm must be bounded, simplified, divided across threads, or moved away from the deterministic control path.

After each correction, the system should return to measurement rather than assuming that the identified change solved the problem. cyclictest, application timestamps, stress workloads, and tracing can be repeated under the same conditions. Comparing before-and-after latency distributions provides evidence that the modification reduced the maximum delay without introducing another interference mechanism elsewhere in the system.

For robotic PREEMPT_RT platforms, ftrace, perf, and LTTng therefore serve different but complementary roles. ftrace reconstructs detailed kernel and scheduler behavior, perf identifies computational and hardware performance characteristics, and LTTng builds broader kernel-to-application timelines. Used together with latency measurement, they transform an unexplained timing spike into an observable sequence of events that can be systematically analyzed.

Real-time debugging is consequently an iterative engineering cycle of measure, trigger, trace, correlate, correct, and validate. PREEMPT_RT reduces kernel latency, but deterministic behavior still depends on drivers, interrupts, priorities, synchronization, memory, CPU topology, middleware, and application code. Effective tracing makes these interactions visible and allows a Linux robotic system to progress from apparently fast execution toward demonstrably predictable real-time behavior.

04.08 Xenomai / RTAI vs PREEMPT RT Comparison

Xenomai, RTAI, and PREEMPT_RT represent different architectural approaches to providing real-time behavior on Linux systems. All three attempt to reduce timing uncertainty and support deterministic execution, but they differ significantly in how deeply they modify the Linux execution model. Understanding these differences is important when selecting a platform for robotics, industrial automation, motion control, and other systems with strict timing requirements.

PREEMPT_RT transforms the standard Linux kernel into a highly preemptible operating environment rather than placing a separate real-time system underneath Linux. Many kernel locks are converted into preemptible synchronization mechanisms, interrupt handlers are largely threaded, and long non-preemptible regions are reduced. Real-time applications therefore continue to use familiar Linux APIs while benefiting from substantially improved scheduling latency and determinism.

This integrated architecture is one of PREEMPT_RT\'s strongest advantages. Applications can use POSIX threads, SCHED_FIFO, SCHED_RR, memory locking, CPU affinity, and standard Linux device interfaces without crossing between fundamentally different execution domains. ROS 2, networking, storage, graphics, middleware, and AI software can coexist naturally with real-time control threads, making PREEMPT_RT attractive for modern robotic edge computers.

Xenomai historically takes a different approach by providing a real-time execution environment closely coupled with Linux. Traditional Xenomai architectures use a dual-kernel or co-kernel concept in which a real-time core handles timing-critical activities with higher precedence, while the normal Linux kernel executes as a lower-priority domain. This separation can provide strong deterministic behavior even when ordinary Linux experiences significant workload variation.

In a dual-kernel architecture, hardware interrupts and real-time events can first be handled by the real-time layer before Linux receives processing opportunities. Critical tasks can therefore avoid many sources of latency originating from the general-purpose kernel. The tradeoff is architectural complexity because applications, drivers, and communication mechanisms must respect the boundary between the real-time environment and conventional Linux services.

Modern Xenomai development has also included approaches that work more closely with the native Linux real-time model rather than relying exclusively on the traditional dual-kernel structure. Consequently, the term Xenomai can refer to different generations and architectures. When evaluating Xenomai for a project, engineers must identify the specific version, kernel integration method, supported APIs, and hardware requirements rather than treating it as one fixed implementation.

RTAI, the Real-Time Application Interface, is another Linux real-time framework historically associated with a dual-kernel approach. Similar to traditional Xenomai, it places a real-time layer beneath or alongside the ordinary Linux kernel so that critical tasks can execute with precedence over normal Linux activity. This architecture was particularly attractive for demanding control and measurement systems requiring very low and tightly bounded latency.

The fundamental advantage of Xenomai and RTAI-style dual-kernel designs is isolation from ordinary Linux scheduling behavior. A hard real-time task can operate within a dedicated real-time domain while Linux handles noncritical services. Under carefully controlled hardware and software configurations, this approach can provide extremely strong latency characteristics for specialized industrial control, instrumentation, and motion-control applications.

However, stronger separation introduces integration costs. Calling ordinary Linux services directly from a hard real-time context may require transitions between execution domains or special communication mechanisms. Device drivers may need real-time-specific support, and debugging can involve interactions between two execution environments. Software architecture must therefore clearly distinguish deterministic operations from services that belong to the normal Linux domain.

PREEMPT_RT generally offers a simpler programming and maintenance model because there is essentially one Linux kernel environment. Existing Linux applications can often be adapted by assigning real-time scheduling policies, locking memory, preallocating resources, controlling CPU and IRQ affinity, and removing nondeterministic operations from critical paths. This reduces the architectural distance between prototype software and the deployed real-time application.

Kernel maintenance is another major consideration. PREEMPT_RT has progressively moved into the upstream Linux ecosystem, greatly improving its long-term integration with mainstream kernel development. This direction reduces dependence on large external patch sets and makes support by distributions and hardware vendors increasingly practical. For products expected to follow contemporary Linux platforms, this integration can significantly simplify lifecycle management.

Dual-kernel systems may require more specialized kernel combinations, patches, pipelines, or driver adaptations. Hardware enablement can therefore require additional engineering when new processors, network devices, GPUs, or vendor-specific drivers are introduced. This matters in robotics because edge computers often combine real-time control with rapidly evolving GPU, networking, camera, accelerator, and middleware technologies that strongly depend on mainstream Linux compatibility.

Driver compatibility is especially important when the same computer performs both deterministic control and high-level perception. PREEMPT_RT can generally retain the conventional Linux driver ecosystem, although individual drivers must still be evaluated for latency behavior. A dual-kernel environment may provide stronger isolation for dedicated hardware paths but can require specialized real-time drivers or carefully designed communication between real-time and Linux-managed devices.

The meaning of hard real-time must also be considered carefully. PREEMPT_RT can provide excellent deterministic performance, but it does not automatically convert every Linux subsystem, driver, and application into a mathematically bounded hard real-time component. Xenomai or RTAI may be preferable when extremely strict latency guarantees justify a more specialized architecture. The decision should therefore be based on measured deadlines and failure consequences rather than terminology alone.

For many robotic systems, control requirements span several timing domains. Low-level motor controllers may already execute current, velocity, or position loops at several kilohertz on dedicated MCUs or drives. The Linux edge computer may instead coordinate motion, communication, state estimation, safety supervision, and higher-level control at hundreds of hertz or around 1 kHz. In such architectures, PREEMPT_RT often provides a practical balance between determinism and software integration.

Xenomai or RTAI becomes more compelling when the Linux computer itself must execute exceptionally strict hard real-time loops and the consequences of deadline violations justify additional engineering complexity. Examples can include specialized machine control, precision measurement, deterministic data acquisition, or legacy industrial systems built around these frameworks. Their value therefore depends strongly on the application\'s timing envelope and existing software ecosystem.

Performance comparisons should not rely on a single published latency number. Kernel version, processor architecture, BIOS configuration, power management, device interrupts, drivers, CPU isolation, workload, and measurement methodology can change results substantially. PREEMPT_RT, Xenomai, and RTAI should therefore be evaluated on the target hardware using representative workloads and identical deadline-oriented test conditions whenever a platform decision depends on latency.

The debugging ecosystem also differs. PREEMPT_RT benefits directly from standard Linux tools such as ftrace, perf, trace-cmd, cyclictest, and LTTng, allowing kernel and application behavior to be analyzed within a largely unified environment. Dual-kernel architectures may require additional framework-specific tracing and debugging knowledge because timing-critical execution can occur outside the normal Linux scheduling domain.

From a system-engineering perspective, the choice can be viewed as a tradeoff between integration and isolation. PREEMPT_RT moves Linux itself toward deterministic execution, preserving broad compatibility and simplifying application integration. Traditional Xenomai and RTAI architectures isolate hard real-time execution from ordinary Linux more aggressively, potentially achieving stronger timing separation at the cost of greater architectural and maintenance complexity.

For modern Physical AI and robotic edge systems combining ROS 2, SLAM, perception, networking, GPU inference, logging, and deterministic control, PREEMPT_RT is often the natural starting point. If measurement demonstrates that its worst-case latency cannot satisfy the required deadline even after careful CPU, IRQ, memory, and application tuning, a more strongly isolated real-time architecture can then be evaluated based on the actual unresolved timing requirement.

The correct platform is therefore not determined by which technology produces the smallest benchmark value in isolation. It depends on deadline severity, hardware compatibility, driver availability, software integration, maintainability, debugging requirements, and product lifecycle. PREEMPT_RT emphasizes mainstream Linux integration, while Xenomai and RTAI historically emphasize stronger real-time separation. Selecting among them requires balancing deterministic performance against the complexity of the complete deployed system.

04.09 ROS2 with PREEMPT RT: Real-Time Node Design [w/Code]

ROS 2 can operate on PREEMPT_RT Linux as part of a real-time robotic architecture, but installing a real-time kernel alone does not make every ROS 2 node deterministic. Predictable behavior depends on the complete execution path, including thread scheduling, executors, callbacks, memory allocation, middleware, communication, device drivers, and application design. A real-time node must therefore be engineered so that critical operations have bounded and observable timing behavior.

The first design decision is to identify which ROS 2 functions actually require real-time execution. A motor command loop, state-feedback controller, safety monitor, or high-rate hardware interface may have strict deadlines, while visualization, logging, diagnostics, map processing, and parameter services usually do not. Separating these responsibilities prevents noncritical ROS 2 activity from competing directly with the deterministic control path.

PREEMPT_RT provides the kernel foundation by reducing non-preemptible execution and allowing most interrupt processing to participate in priority-based scheduling. Critical ROS 2 threads can then use policies such as SCHED_FIFO and receive higher priority than ordinary Linux tasks. Priority selection must remain systematic because assigning maximum priority indiscriminately can starve middleware, device processing, or lower-priority tasks required for correct system operation.

CPU affinity complements priority scheduling by controlling where real-time threads execute. A control node or its critical executor thread can be pinned to an isolated CPU while perception, logging, user interfaces, and general ROS 2 workloads remain on housekeeping CPUs. Relevant device IRQs can be assigned deliberately so that unrelated interrupts do not disturb the control core while necessary hardware events remain available with appropriate priority.

Memory behavior must also be controlled. Page faults and unpredictable dynamic allocation can introduce latency even when scheduling is correctly configured. Real-time ROS 2 processes can lock memory with mechanisms such as mlockall and prefault required memory before entering the critical phase. Buffers, messages, queues, state vectors, and control data should be allocated during initialization whenever possible rather than repeatedly created inside periodic callbacks.

ROS 2 executors strongly influence callback timing because they determine when subscriptions, timers, services, and other executable entities are dispatched. A general-purpose executor may be suitable for noncritical nodes but can introduce interference when many callbacks share the same execution context. Real-time designs should minimize the number of unrelated callbacks participating in the critical executor and make callback execution paths short, bounded, and predictable.

Callback groups provide an additional mechanism for organizing execution. Mutually exclusive groups can prevent selected callbacks from running concurrently, while reentrant groups permit parallel execution where it is safe. These mechanisms improve concurrency control but do not automatically guarantee real-time behavior. The designer must still analyze blocking relationships, shared resources, execution times, and the priorities of threads responsible for executing each callback.

Periodic control should preferably use a timing model that avoids accumulated drift. A 1 kHz controller has a nominal period of one millisecond, so repeated relative delays can gradually shift execution away from the intended timeline. The control architecture should distinguish the intended release time from the actual callback execution time and monitor lateness explicitly. Stable periodic activation is more important than simply executing callbacks as quickly as possible.

The ROS 2 middleware layer introduces another timing dimension. DDS communication may involve serialization, queues, discovery mechanisms, network transport, internal threads, and resource management. Quality of Service settings such as reliability, history depth, durability, and deadline behavior influence communication characteristics. Real-time design therefore requires QoS policies to match the actual control requirement rather than selecting stronger communication guarantees without considering their timing cost.

Intra-process communication can reduce data movement when publishers and subscribers operate within the same process, but architectural decisions should be based on measured behavior rather than assumptions. Inter-process and network communication introduce additional scheduling and transport effects. For a highly critical control loop, designers may keep tightly coupled operations within a compact deterministic process while exposing slower state and supervisory information through normal ROS 2 interfaces.

Synchronization deserves particular attention because mutex contention can destroy otherwise good scheduler latency. A high-priority control callback waiting for a resource held by a lower-priority thread can experience priority inversion. PREEMPT_RT and appropriate synchronization primitives can provide priority-inheritance mechanisms, but critical sections should still remain short. Shared data can often be redesigned using bounded buffers, ownership transfer, or carefully controlled lock-free patterns.

Blocking operations should generally be removed from the real-time callback path. File access, console output, unbounded network operations, parameter updates, memory allocation, and slow service calls can introduce unpredictable delays. Instead, the control thread can exchange bounded data with lower-priority worker threads responsible for logging, storage, diagnostics, or noncritical communication. This creates a clear boundary between deterministic and best-effort processing.

Hardware interfaces form another important boundary. Sensor acquisition and actuator commands may depend on CAN, CANopen, EtherCAT, serial communication, Ethernet, or vendor-specific drivers. The real-time characteristics of the ROS 2 node cannot exceed those of its underlying driver and communication path. IRQ affinity, driver execution time, buffering, bus cycle timing, and device response should therefore be included in the end-to-end timing analysis.

A practical robotic architecture often uses multiple control rates. Dedicated motor controllers may execute current or velocity loops at several kilohertz, while the PREEMPT_RT computer runs a coordinated motion or state-control loop at hundreds of hertz or approximately 1 kHz. ROS 2 can then provide communication with perception, planning, monitoring, and fleet functions at slower rates. This multi-rate structure keeps the hardest electrical control deadlines away from complex Linux software.

ROS 2 composition can be useful for reducing communication overhead, but combining many components into one process also increases the consequences of interference or failure. Real-time components should therefore be grouped according to timing and fault-containment requirements rather than purely for deployment convenience. A compact control process and separate noncritical processes often provide a clearer architecture than placing every robotic function inside one large executor.

Observability must be designed into the node from the beginning. Useful timestamps include intended release time, actual wake-up or callback start, sensor-data availability, computation completion, command publication, and cycle completion. These measurements distinguish scheduler latency from callback execution time and communication delay. Without such instrumentation, a missed deadline may incorrectly be attributed to PREEMPT_RT when the actual cause lies inside ROS 2 or application logic.

Tracing tools can correlate these application measurements with operating-system behavior. ftrace and trace-cmd can expose scheduling, IRQ, migration, and kernel events, while perf can identify expensive computation or hardware-resource contention. ROS 2 tracing and LTTng-based instrumentation can extend the timeline through callbacks and middleware activity. Together, these tools help locate where latency accumulates across the kernel-to-application path.

Validation should be performed under representative system load rather than on an idle development computer. Perception pipelines, GPU inference, cameras, LiDAR, networking, storage, logging, and other ROS 2 nodes should operate while control latency is measured. Long-duration testing is necessary because rare disturbances may appear only after millions of cycles. Maximum latency and deadline misses are more significant than favorable average execution times.

The timing budget should cover the entire control cycle. Scheduler wake-up latency, sensor acquisition, middleware delivery, callback execution, synchronization, actuator communication, and safety margin must collectively remain within the required deadline. For a one-millisecond cycle, excellent cyclictest results alone are insufficient if ROS 2 processing or communication consumes the remaining budget. End-to-end measurements must therefore determine whether the architecture actually satisfies its control requirement.

A robust ROS 2 real-time node is consequently built by combining PREEMPT_RT with application-level discipline. SCHED_FIFO establishes scheduling precedence, CPU and IRQ affinity reduce interference, memory locking and preallocation reduce allocation-related disturbances, bounded callbacks control execution time, and carefully designed ROS 2 communication limits middleware uncertainty. Each mechanism addresses a different source of nondeterminism rather than acting as an independent solution.

The final objective is not to make every ROS 2 node real-time. It is to create a clearly defined deterministic control island inside a larger robotic software system. PREEMPT_RT provides the real-time Linux foundation, while ROS 2 connects that control island to perception, planning, diagnostics, AI, and distributed services. This separation allows modern robotic systems to combine strict control timing with the flexibility and scalability of the broader ROS 2 ecosystem.

04.10 1kHz Robot Control Loop Achievement Case Study

Achieving a 1 kHz robot control loop on PREEMPT_RT Linux means completing one deterministic control cycle every millisecond while maintaining sufficient margin for occasional timing variation. The engineering objective is not simply to observe an average frequency near 1 kHz, but to ensure that wake-up latency, computation, communication, synchronization, and actuator command delivery consistently remain inside the one-millisecond deadline under representative system load.

The case-study architecture separates control responsibilities according to their required timing rates. Fast current and inner velocity loops remain inside dedicated motor drives or MCUs, where several-kilohertz execution can be guaranteed independently of Linux. The PREEMPT_RT edge computer executes the coordinated robot control loop at 1 kHz, while perception, planning, visualization, diagnostics, and AI workloads operate at slower rates on separate execution resources.

The first implementation establishes a functional 1 kHz loop without assuming that it is real-time. A control thread reads the latest state, calculates the control command, transmits the result, and waits for the next absolute release time. Initial measurements commonly reveal acceptable average timing but occasional large latency spikes. These outliers demonstrate why average loop frequency alone cannot establish deterministic operation.

PREEMPT_RT provides the kernel foundation for reducing these disturbances. The kernel is configured for real-time preemption, high-resolution timers, and threaded interrupt handling. The control thread uses SCHED_FIFO so that it can preempt ordinary Linux workloads according to a defined priority hierarchy. Priority is selected high enough to protect the control deadline while preserving execution opportunities for essential IRQ and system threads.

Absolute periodic timing is used instead of repeatedly sleeping for one millisecond relative to the previous iteration. Each cycle calculates or advances the next intended release point on a monotonic clock and waits until that absolute instant. If one cycle starts slightly late, the following release remains aligned with the original timeline. This prevents timing error from accumulating into long-term control-loop drift.

CPU isolation provides the next major improvement. One physical CPU core is reserved primarily for deterministic control, while general ROS 2 nodes, logging, networking services, visualization, storage operations, and background Linux tasks remain on housekeeping CPUs. The control thread is explicitly pinned to its assigned core so that scheduler migration does not introduce additional cache disturbances or unpredictable execution delays.

IRQ affinity is coordinated with CPU isolation rather than configured independently. Unrelated storage, graphics, network, USB, and background device interrupts are moved away from the control CPU. Interrupts required by the control communication path are assigned according to their timing relationship with the control thread. Their priorities are selected so that hardware data can arrive promptly without allowing unnecessary interrupt activity to dominate the one-millisecond cycle.

Memory preparation eliminates another class of unpredictable latency. The process locks required memory with mlockall, prefaults relevant pages and stack regions, and allocates control buffers before entering real-time operation. Dynamic allocation, object construction, container resizing, and deallocation are avoided inside the periodic loop. Fixed-size state and command structures keep memory behavior stable across millions of control iterations.

Synchronization is then examined because a high-priority thread can still miss its deadline while waiting for shared data. Long mutex-protected regions are removed from the control path, and priority-inheritance synchronization is used where locking cannot be eliminated. Sensor and supervisory data can be transferred through bounded buffers or carefully designed ownership mechanisms so that lower-priority processing cannot hold the real-time thread indefinitely.

The communication path is treated as part of the control loop rather than an external service. CAN, CANopen, EtherCAT, Ethernet, or another robot interface contributes bus scheduling, driver execution, buffering, IRQ processing, and device response time. A 1 kHz software timer is meaningless if actuator communication requires unpredictable multi-millisecond delays. Communication timing is therefore measured together with application execution.

ROS 2 integration follows the principle of creating a deterministic control island. The critical controller and hardware interface are kept compact, with only necessary callbacks participating in the real-time execution path. Logging, parameter management, diagnostics, visualization, and other noncritical callbacks are moved to lower-priority contexts. QoS, executor organization, and callback groups are selected according to measured timing requirements rather than general convenience.

A practical one-millisecond budget can be divided among several activities instead of assigning the entire period to controller computation. Wake-up latency consumes part of the budget, followed by state acquisition, communication processing, control computation, synchronization, and command transmission. An explicit safety margin remains unused during normal execution so that small variations do not immediately become deadline violations.

For example, the controller may be engineered so that normal computation occupies only a fraction of the available millisecond. The exact allocation depends on hardware and communication architecture, so fixed universal numbers should not be assumed. What matters is that measured worst-case contributions remain below their assigned limits and that their combined worst-case behavior leaves sufficient margin before the next release deadline.

cyclictest is used before and during integration to characterize the underlying scheduler wake-up latency. Tests are performed on the intended real-time CPU with representative priorities, CPU affinity, IRQ configuration, and stress workloads. Long-duration measurements are more valuable than short idle tests because rare latency spikes may appear only when networking, storage, sensors, GPU inference, or other subsystems create contention.

Application instrumentation then measures what cyclictest cannot observe. Each cycle records the intended release time, actual execution start, state acquisition, computation completion, command transmission, and final cycle completion. These timestamps separate scheduler latency from execution and communication delays. The resulting distribution shows whether the complete robot control path, rather than only the kernel scheduler, satisfies the 1 kHz requirement.

When an abnormal cycle is detected, tracing is triggered around the event. ftrace and trace-cmd can reveal scheduler transitions, IRQ execution, kernel threads, migrations, and driver activity. perf can identify unexpectedly expensive computation or shared-resource contention, while LTTng and ROS 2 tracing can connect kernel events with middleware and callback execution. The objective is to convert a rare timing spike into a reconstructable sequence of events.

Suppose tracing reveals that a network IRQ occasionally executes on the control CPU and delays the SCHED_FIFO controller. The corrective action is not to increase control-thread priority blindly, but to determine whether that interrupt belongs on the control core and adjust affinity and priority accordingly. If the spike instead comes from memory allocation, a mutex, driver path, or expensive callback, the corresponding source of nondeterminism is redesigned.

Validation is repeated after every significant change using the same workload and measurement conditions. Improvements are judged by maximum latency, jitter distribution, deadline misses, and end-to-end cycle completion rather than average CPU utilization alone. A tuning change that improves cyclictest but worsens actuator communication is not a successful optimization because the final engineering target is the complete robot control deadline.

Stress validation runs the 1 kHz controller while realistic background workloads remain active. Cameras and LiDAR produce data, ROS 2 nodes exchange messages, network traffic is generated, storage and logging operate, and GPU perception or AI inference executes concurrently. Thermal and power-management behavior should also be represented because a configuration that works only on a cold, idle development system does not demonstrate production-level determinism.

Long-duration testing is essential because a 1 kHz loop executes 3.6 million cycles per hour. A disturbance occurring only once every million cycles can therefore become operationally relevant within a robot deployment. Validation should examine not only whether most cycles complete quickly, but whether rare tail events remain inside the deadline or are handled by a defined safety strategy when an exceptional overrun occurs.

The achieved architecture is therefore a multi-layer result rather than a single PREEMPT_RT configuration. The kernel provides preemptibility, SCHED_FIFO establishes scheduling precedence, CPU and IRQ isolation reduce interference, memory preparation removes page and allocation disturbances, bounded synchronization controls blocking, and communication design preserves predictable hardware interaction. Application architecture must maintain these properties through the complete control path.

A successful 1 kHz case study consequently demonstrates more than a timer repeatedly firing every millisecond. It demonstrates a measured and validated timing chain from periodic release through state acquisition and computation to actuator command delivery under realistic load. PREEMPT_RT makes this architecture practical on Linux, but deterministic performance emerges only when kernel configuration, hardware communication, ROS 2 design, resource isolation, measurement, and repeated validation are engineered as one system.

---

Author: Richard Minsoo Go

LinkedIn: Richard Minsoo Go