STATERA Library · Series 08 · Volume 09 · Chapter 09 · EN
Source: https://github.com/Statera-Library/08_Vol_09_Cloud_and_Edge_Robotics/blob/main/Chapter_09_OTA_Architecture/Chapter_09_OTA_Architecture_EN.md
Volume 09 Cloud and Edge Robotics
09. OTA Architecture
09.01 Robot OTA Architecture: Server, Client, Deploy, Management
Over-the-air deployment for robots is not simply a remote file transfer mechanism. It is a distributed software lifecycle architecture that coordinates software packages, firmware, containers, configuration, AI models, and system images across robots operating under different network and operational conditions. A practical OTA platform therefore combines centralized deployment management with autonomous client-side execution, verification, recovery, and reporting.
The server side acts as the control plane of the OTA architecture. It maintains software versions, deployment packages, device inventories, compatibility metadata, deployment policies, cryptographic signatures, and historical deployment records. Instead of directly manipulating individual robots, the server describes the desired software state of each robot or fleet group and distributes deployment instructions through authenticated communication channels.
A robot-side OTA client forms the execution plane. The client periodically or event-drivenly communicates with the deployment service, authenticates the server, obtains deployment metadata, and determines whether an available package is applicable to its hardware and software configuration. It then downloads the required artifacts, verifies integrity and authenticity, performs installation, executes validation procedures, and reports the resulting state to the server.
Robot OTA differs fundamentally from conventional desktop software updating because a robot continuously interacts with the physical environment. Updating navigation, perception, motor-control interfaces, ROS 2 components, sensor drivers, or AI inference models can directly change physical behavior. Deployment management must therefore consider operational state, battery level, mission status, connectivity, hardware revision, safety conditions, and recovery capability before installation begins.
The deployment server normally maintains a device registry describing each robot\'s identity and configuration. Typical records include robot ID, hardware model, compute platform, operating-system version, firmware versions, application versions, installed AI models, deployment group, and current operational state. This inventory enables the OTA system to determine which package is compatible with which device rather than treating an entire heterogeneous fleet as identical.
Version management provides another essential architectural function. Every deployable artifact should have an identifiable version and associated metadata describing dependencies, hardware requirements, operating-system compatibility, configuration assumptions, and upgrade paths. A release may contain several coordinated components, such as a container image, ROS 2 application package, configuration bundle, perception model, and device firmware, that together represent one validated robot software baseline.
Deployment management converts releases into controlled fleet operations. A deployment specification identifies the target devices, required release, scheduling constraints, installation policy, verification conditions, and failure behavior. Robots can be organized by model, customer site, geographic region, hardware generation, operational role, or software channel. This grouping allows the same OTA infrastructure to manage development robots, validation fleets, pilot systems, and production robots independently.
The communication architecture should assume that robots may experience intermittent or low-bandwidth connectivity. Large packages should therefore support resumable downloads, retry policies, caching, integrity checks, and bandwidth-aware transfer. The OTA client should preserve enough local state to continue or safely restart an interrupted update process. Network disconnection must not leave the robot in an undefined software state or make normal local operation unnecessarily dependent on cloud availability.
Security begins with trusted device identity and authenticated server communication. The robot must determine not only whether a package was downloaded correctly but also whether it was produced and authorized by a trusted software-release process. Cryptographic hashes protect artifact integrity, while digital signatures establish provenance and authorization. Transport encryption, certificate-based identity, access control, credential rotation, and protected key storage complement package-level verification.
An OTA transaction should be modeled as a state machine rather than a single installation command. Typical states include update discovery, eligibility evaluation, download, verification, staging, installation, activation, post-installation validation, completion, and rollback. Persisting these transitions allows both the robot and management server to understand exactly where an interrupted or failed deployment stopped and determine the appropriate recovery operation.
Safe installation requires separation between downloading an artifact and activating it. A robot can often download software while operating normally, but activation may require the robot to stop, dock, reboot, restart containers, or enter a maintenance state. Separating download, staging, and activation reduces operational interruption and allows deployment managers to schedule disruptive operations according to fleet workload rather than network-transfer duration.
Atomicity becomes particularly important when multiple software components depend on one another. Updating a perception model while leaving an incompatible inference runtime, configuration schema, or application interface unchanged can create failures even though each individual artifact is valid. Release manifests should therefore describe compatible component sets, and the OTA client should verify dependencies before changing the active robot configuration.
Rollback provides the primary recovery path when newly deployed software fails validation. Systems may retain a previous container image, software package, model, or complete system partition so that the client can restore a known-good state. A/B partition strategies are particularly useful for operating-system or firmware updates because the new image can be written to an inactive partition before the boot target changes, preserving the previous image for recovery.
Post-deployment validation determines whether installation success actually means operational success. The client can verify process startup, container health, ROS 2 node availability, required interfaces, sensor connectivity, storage capacity, system temperature, GPU availability, and application-specific health indicators. Only after required checks pass should the new release be declared healthy; otherwise the client can initiate rollback or request operator intervention.
Fleet-scale OTA requires progressive deployment rather than simultaneous fleet-wide activation. A release can first reach laboratory devices, then selected validation robots, a small production subset, and finally progressively larger groups. Each stage provides telemetry and failure evidence before exposure expands. This architecture limits the operational impact of defects and creates measurable decision points between software release and full production adoption.
Server and client responsibilities should remain clearly separated. The server determines policy, target selection, release eligibility, scheduling, and fleet-level visibility, while the robot client retains authority over local installation safety and execution. A server may request an update, but the client should reject or postpone it when required local conditions are not satisfied. This separation is particularly important when cloud connectivity and physical robot safety have different availability assumptions.
Observability connects deployment management with fleet operations. Each OTA transaction should generate timestamps, package identifiers, version transitions, download results, verification outcomes, installation logs, reboot information, health-check results, and rollback events. Aggregating these records enables operators to distinguish isolated device failures from release-wide problems and provides an auditable history of the software configuration deployed to every robot.
The architecture should also support several artifact classes because modern robots contain multiple independently evolving software layers. Linux system images, bootloader or controller firmware, ROS 2 packages, Docker containers, configuration files, maps, calibration parameters, AI models, and security certificates have different installation and rollback characteristics. A unified management plane can track them while specialized client handlers perform artifact-specific deployment operations.
OTA management naturally connects cloud, edge, fleet-management, and DevOps systems. CI/CD pipelines can produce validated artifacts and register approved releases, while fleet services provide robot inventory and operational status. Monitoring systems return deployment health data, and identity services control device authentication. This places OTA between software delivery and robot operations rather than treating it as an isolated maintenance subsystem.
A mature robot OTA architecture ultimately maintains a controlled relationship between desired state and actual state. The server records what software each robot should run, while the client reports what is actually installed and operational. Differences trigger reconciliation according to deployment policy, but physical safety and recoverability remain local constraints. This server-client model provides the foundation for scalable staged rollout, automatic rollback, fleet scheduling, CI/CD integration, and secure software supply-chain management addressed by the subsequent OTA architecture topics.
09.02 OTA Deploy Server Design: Version Registry / Sign Server [w/Code]
An OTA deployment server is the central coordination layer that transforms validated software artifacts into controlled robot releases. Its responsibility extends beyond storing update files: it maintains version relationships, device compatibility, release metadata, cryptographic trust, deployment policies, and historical state. In a fleet architecture, the server provides the authoritative source for determining which software release may be delivered to each robot.
A practical deployment server can be divided into several logical services, including a version registry, artifact repository, release manager, signing service, device registry, deployment controller, and audit database. These services may initially run as modules of one application and later become independent services as fleet size increases. Separating their responsibilities makes software releases easier to validate, secure, scale, and trace throughout the OTA lifecycle.
The version registry maintains the identity and relationships of every deployable software component. A version record normally contains the component name, semantic or internal version identifier, build number, creation time, source revision, supported hardware, operating-system requirements, dependencies, and release status. The registry therefore answers more than "what is the latest version"; it determines which version is valid for a particular robot configuration.
Robot releases often contain multiple independently versioned components. A single release may combine an operating-system image, ROS 2 packages, container images, motor-controller firmware, sensor drivers, configuration files, maps, and AI models. The server should represent the complete validated combination through a release manifest rather than assuming that independently current components are mutually compatible. This creates a reproducible software baseline for production robots.
The artifact repository stores the actual binary objects referenced by the version registry. Large operating-system images, firmware binaries, container layers, model files, and compressed application packages should be treated as immutable artifacts once released. Their metadata remains in the registry while their binary content may reside in object storage, an OCI-compatible container registry, or another scalable storage service optimized for large downloads.
Immutability is important because a version identifier must always resolve to exactly the same content. Replacing a binary while preserving its version number destroys reproducibility and makes field failures difficult to investigate. Each artifact should therefore be associated with a cryptographic digest such as a SHA-256 hash. The server and robot client can independently calculate the digest and detect corruption or unauthorized modification before installation.
The signing server establishes trust between the software production pipeline and deployed robots. After an artifact or release manifest passes the required build and validation stages, an authorized signing operation generates a digital signature using a protected private key. Robot clients verify the signature using a trusted public key before accepting the release. This prevents possession of artifact-storage access alone from becoming sufficient authority to deploy executable software.
Signing keys require stronger protection than ordinary application credentials because compromise of a release-signing key can undermine the entire OTA trust model. Private keys should be isolated from general deployment services and preferably protected by a hardware security module, managed key service, or equivalent controlled signing environment. Access should follow least-privilege principles, and signing requests should generate immutable audit records identifying the release and authorization context.
The release manager connects versions, artifacts, and signatures into a deployable unit. It can maintain lifecycle states such as development, candidate, validated, approved, deprecated, and revoked. Only releases reaching an explicitly authorized state should become available to production deployment policies. This separation prevents an automatically generated CI build from becoming equivalent to a production-approved robot release merely because its artifact exists in storage.
Compatibility metadata is especially important for heterogeneous robot fleets. Robots may differ in CPU architecture, Jetson or x86 computing platform, sensor configuration, controller firmware, storage capacity, and hardware revision. The server should associate releases with explicit compatibility constraints and compare them against device inventory before deployment. An update intended for one hardware generation should never depend solely on operators manually selecting the correct robots.
The device registry supplies the server with the information required for this matching process. Each device identity can reference hardware type, serial identity, software baseline, installed component versions, deployment channel, customer or site grouping, and current OTA state. When a client requests an update, the server evaluates both device identity and release policy before returning eligible release metadata rather than simply returning the newest available package.
Deployment policies determine how an approved release moves through the fleet. A policy can specify target groups, allowed software channels, maintenance windows, prerequisite versions, battery or operational constraints, deployment concurrency, and expiration conditions. The deployment controller resolves these policies into device-specific actions while retaining a centralized record of intended state. This creates a clear distinction between defining a release and authorizing its actual installation.
The server should distribute metadata before transferring large artifacts. A robot can first receive a compact manifest describing the release identifier, artifact URLs, sizes, hashes, signatures, dependencies, required disk space, compatibility constraints, and activation requirements. The client evaluates this information locally and downloads binary content only after determining that the update is applicable. This approach reduces unnecessary network traffic and supports safer offline-first behavior.
Artifact delivery itself should be scalable independently of the control server. The deployment API does not need to stream every multi-gigabyte system image through the same service that manages policies and device states. Binary artifacts can be delivered through object storage, content delivery infrastructure, or geographically distributed caches using temporary authenticated access. Separating control-plane traffic from artifact data improves scalability when thousands of robots update concurrently.
The server must also understand update lineage. Some releases can be installed directly from many previous versions, while others require intermediate migrations because of database schemas, firmware interfaces, bootloader changes, or configuration transformations. Upgrade-path metadata allows the deployment controller to determine whether a direct transition is valid. The same information is valuable when determining whether rollback to an earlier baseline remains technically safe.
Revocation is the inverse of release approval and should be designed from the beginning. If a vulnerability, corrupted artifact, signing problem, or severe operational defect is discovered, administrators must be able to prevent further installation of the affected release. Revocation metadata can also instruct connected clients that a previously approved artifact is no longer trusted, while fleet inventory identifies robots that have already installed the affected version.
Auditability ties the entire deployment server together. Important events include artifact registration, metadata modification, signing requests, release approval, policy creation, target selection, deployment initiation, client acknowledgement, installation result, rollback, and revocation. These records should preserve timestamps, identities, release references, and relevant outcomes so that operators can reconstruct exactly how a specific software version reached a particular robot.
The deployment server should expose machine-readable interfaces so that OTA becomes part of the broader software delivery process. CI/CD systems can register artifacts and candidate releases, validation pipelines can update qualification results, fleet-management systems can provide operational eligibility, and monitoring services can return deployment health. Human operators then manage policy and exceptions through the same authoritative backend rather than maintaining separate manual release records.
High availability is important because the deployment server may coordinate a large production fleet, but temporary server failure should not make robots unsafe. Robots should continue operating with their existing validated software when the server is unavailable. After connectivity returns, clients can reconcile their local state with the desired state stored by the server. This preserves the cloud-edge separation in which centralized management controls evolution without becoming a real-time dependency for robot operation.
A well-designed OTA deployment server therefore forms a trusted release authority rather than merely a download server. The version registry defines what exists, the artifact repository preserves immutable content, the signing server establishes authenticity, the release manager determines what is approved, and the deployment controller determines where and when it may be installed. Together, these functions provide the server foundation required for staged rollout, monitoring, rollback, scheduling, CI/CD integration, and secure fleet-scale OTA operations described throughout the OTA architecture chapter.
09.03 Mender Server Self-Hosted Configuration [w/Code]
Mender is a software update platform designed around a client-server architecture in which robot devices communicate with a centrally managed backend. In a self-hosted configuration, the organization operates the Mender Server infrastructure instead of relying on a hosted service. This approach places responsibility for server deployment, storage, networking, certificates, authentication, upgrades, monitoring, and operational maintenance within the organization's own infrastructure. ([Mender Docs][1])
The Mender Server is composed of multiple microservices, each responsible for a defined part of the OTA lifecycle. The architecture includes services for deployments, device authentication, device inventory, device configuration, device connection, reporting, user administration, workflows, and artifact creation, together with supporting components such as an API gateway, object storage, messaging, and search or data-storage services. This service-oriented structure allows the self-hosted installation to separate device communication, management operations, artifact handling, and internal service communication.
For a production self-hosted environment, Mender provides a Kubernetes-based installation approach using a Helm chart. The Kubernetes cluster hosts the Mender backend services and provides the orchestration layer required for service deployment, scaling, restart handling, and configuration management. The production architecture should therefore be treated as an infrastructure platform rather than simply installing a single Mender application on one Linux server. ([Mender Docs][2])
The external entry point is normally an Ingress controller or load balancer that exposes the Mender Server outside the Kubernetes cluster. Production configuration requires HTTPS, and the ingress layer is responsible for receiving authenticated client and management requests and forwarding them to the appropriate backend services. Health checking and TLS certificate management should also be incorporated into this boundary so that the externally visible server endpoint remains controlled and observable. ([Mender Docs][3])
Persistent storage is another fundamental part of the self-hosted architecture. The Mender backend requires database and artifact-storage services, while large update artifacts should be separated from transactional metadata. Mender documentation describes support for S3-compatible object storage and other storage options, while MongoDB is used for persistent application data. For production high availability, Mender recommends MongoDB replica-set operation rather than relying on a single database instance. ([Mender Docs][2])
The artifact-storage layer should be designed around the characteristics of robot software packages. Operating-system images, firmware, containers, and other large artifacts may consume substantially more storage and bandwidth than ordinary API traffic. Keeping these artifacts in dedicated object storage allows the Mender backend to concentrate on deployment control and metadata while the storage layer handles large-object delivery. This separation also provides a foundation for backup, replication, lifecycle management, and scalable artifact distribution.
Mender also uses supporting infrastructure for internal communication and service operation. The documented server architecture includes NATS as a messaging system and OpenSearch for storage and search, while SeaweedFS can provide object storage in the documented architecture. These components are not merely optional application libraries; they participate in the distributed backend architecture and therefore need to be considered when planning resource allocation, persistence, backup, monitoring, and recovery. ([Mender Docs][1])
Device authentication establishes the trust relationship between a robot and the self-hosted Mender Server. A Mender device is identified using identity attributes together with a public key and device tier. Before a device can authenticate and communicate normally, its authentication identity must be authorized by the server. The authentication service then issues authentication tokens that the Mender Client uses when accessing Device APIs. This model separates identifying a physical robot from authorizing its ability to participate in OTA operations. ([Mender Docs][4])
The robot-side configuration must point the Mender Client to the organization's self-hosted server. The Mender Client uses a JSON configuration file, commonly located at /etc/mender/mender.conf\, for settings such as the server URL and certificate information. Persistent configuration can also be maintained under /var/lib/mender\, allowing selected configuration data to survive future software updates. This distinction is important because an OTA system must not accidentally remove the configuration required to contact the OTA server during an update. ([Mender Docs][5])
TLS configuration protects communication between the robot and server. By default, the Mender Client can validate the server through the operating system's trusted Certificate Authorities when the server certificate is issued by a recognized CA. For self-signed certificates, the appropriate certificate must instead be provisioned to the client so that it can verify the self-hosted server. Consequently, certificate provisioning becomes part of the robot image or manufacturing process when an internal certificate authority or self-signed certificate is used. ([Mender Docs][6])
Artifact signing provides a second security layer that is independent of the network connection. A Mender Artifact can be digitally signed before being uploaded to the server, and the Mender Client can verify the signature using its configured public verification key. If signature verification is enabled and the artifact is unsigned or cannot be verified, the client rejects the update rather than installing it. This means that successfully downloading an artifact is not sufficient to authorize its installation. ([Mender Docs][7])
Signing-key management should therefore be separated from ordinary artifact-building operations whenever possible. Mender documentation notes that using the build system directly as the signing system increases security exposure because compromise of the build environment could provide access to the private signing key. A controlled signing environment, potentially incorporating an HSM or another PKCS#11-compatible cryptographic device, provides stronger protection for the private key and establishes a clearer trust boundary between software creation and production release authorization. ([Mender Docs][7])
A self-hosted Mender installation also requires deliberate configuration of server-side secrets and certificates. Internal service communication may use separate cryptographic credentials, while the API gateway exposes the externally accessible HTTPS endpoint. User authentication, device authentication, artifact verification, and service-to-service communication should therefore not be treated as one generic credential system. Each trust relationship should have an explicit purpose, controlled storage, and an appropriate rotation or replacement procedure. ([Mender Docs][6])
The deployment workflow begins when a validated Mender Artifact is made available to the server. The artifact contains the software payload and associated metadata, while the server maintains information about devices, deployments, and their current state. A deployment can then target a defined set of devices according to the organization's fleet-management policy. The robot periodically communicates with the server, determines whether an update is available, downloads the artifact, verifies it, and performs the configured installation procedure.
Self-hosting also changes the operational responsibility compared with a managed OTA service. The organization must maintain the Kubernetes environment, external database or database cluster, artifact storage, networking, TLS certificates, secrets, monitoring, backups, and recovery procedures. Mender documentation explicitly distinguishes supported production configurations from alternative infrastructure that may technically work but is not necessarily covered by the same support model. This makes infrastructure selection and maintenance strategy part of the OTA system design rather than an afterthought. ([Mender Docs][2])
Scaling should be considered according to the number of connected devices and the characteristics of the update workload. Mender documents example resource sizes ranging from small installations for up to approximately 100 devices to larger configurations for much larger fleets. These figures should be treated as documented infrastructure guidance rather than universal sizing rules, because actual resource requirements depend on deployment frequency, concurrent updates, artifact size, device reporting volume, storage architecture, and other workload characteristics. ([Mender Docs][2])
High availability should focus on avoiding a single point of failure across the complete server path. Kubernetes worker nodes, ingress or load-balancing infrastructure, database services, artifact storage, and supporting services all contribute to overall OTA availability. For example, a highly available application layer cannot provide reliable OTA operation if the database or artifact store remains a single failure point. Production architecture should therefore evaluate the complete dependency chain rather than considering only the Mender application containers.
A self-hosted Mender environment is particularly useful when OTA infrastructure must remain under organizational control. An organization can integrate it with its own network, identity infrastructure, artifact pipeline, monitoring platform, storage policies, and fleet-management environment. The resulting system can serve as an internal OTA control plane while robots remain responsible for local installation, verification, activation, and recovery. This preserves the server-client separation established in the preceding OTA architecture discussion.
The complete configuration can therefore be viewed as a chain from software production to robot execution. A CI/CD environment creates and validates an artifact, a controlled signing process establishes its authenticity, the self-hosted Mender Server registers and manages the release, artifact storage provides the binary payload, deployment services select authorized devices, and the Mender Client performs local installation and verification. Device authentication and TLS protect communication, while persistent databases and storage maintain the state required for reliable fleet operation.
In this architecture, self-hosting Mender is not simply a choice to run OTA software locally. It establishes an organization-controlled OTA platform whose reliability depends on the integration of application services, Kubernetes infrastructure, persistent storage, network exposure, device identity, certificate management, artifact signing, and operational monitoring. Once these foundations are correctly established, the same platform can support controlled fleet deployments and provides the basis for the subsequent topics of staged rollout, monitoring, automatic rollback, scheduling, CI/CD integration, and large-scale robot OTA operation.
09.04 Eclipse hawkBit OTA Server Integration [w/Code]
Eclipse hawkBit is an open-source backend framework for managing software updates across connected edge devices, gateways, and robotic systems. In a robotics environment, hawkBit can serve as the OTA control plane between a fleet-management system and robot-side update clients. Its role is to organize software modules, deployment targets, update assignments, and deployment states so that software delivery becomes a controlled operational process rather than an individual remote maintenance task.
The central concept in hawkBit is the separation between software that can be deployed and devices that receive it. Software modules represent deployable components, while targets represent connected devices or device-side controllers. A deployment connects these elements through an update assignment. This separation is useful for robotics because one software release can be assigned to many robots, while a robot can also contain multiple independently managed software components.
A software module can represent firmware, an application package, a container, or another deployable artifact. The module contains version information and references to the actual artifact content. For a robot, this model can represent a motor-controller firmware package, ROS 2 application, perception container, AI model, configuration package, or complete system image. Keeping the module definition separate from the physical robot allows the same release management mechanism to support different robot platforms.
The target represents the robot-side endpoint that participates in the OTA process. hawkBit identifies a target through a controller ID, which represents the actual service or client communicating with the server. This architecture also allows a single physical target to have multiple controllers when different update responsibilities must be separated, such as firmware management and application management. This distinction is valuable for complex robots containing multiple software update domains.
Communication between a target and hawkBit can use the Direct Device Integration (DDI) API, which is based on HTTP and JSON and uses a polling mechanism. The robot-side client periodically contacts the server to determine whether an update task is available. When an assignment exists, the client obtains the required deployment information and can download the referenced artifact. The client then performs the device-specific installation procedure and reports its progress and result back to the server. ([GitHub][1])
hawkBit also provides a Management API for external applications. A fleet-management system can use this interface to create software modules, register targets, organize deployment information, and initiate update operations without requiring an operator to perform every action manually through a user interface. This makes hawkBit suitable as an OTA backend that can be integrated into a larger robot management platform rather than functioning as an isolated update application. ([GitHub][2])
For architectures that require an intermediary gateway or messaging infrastructure, hawkBit provides the Device Management Federation (DMF) API. DMF uses AMQP-based messaging and can connect hawkBit with other device-management systems or gateways. The messaging approach is useful when a robot fleet already contains an edge gateway responsible for communication with multiple devices. In such an architecture, the gateway can participate in device management while hawkBit remains responsible for update orchestration. ([GitHub][3])
A typical robot OTA integration therefore contains a hawkBit server, an artifact repository, a management application, a device-side update client, and the network connecting them. The management application defines the desired deployment, hawkBit maintains the deployment state, and the robot client executes the actual update. The artifact itself can be stored separately from the management service, allowing large update files to be delivered without forcing the management API to become the primary binary-transfer mechanism.
Deployment state management is particularly important for robots because an update is not complete merely because a file has been downloaded. The client must distinguish between states such as update availability, download, installation, execution, successful completion, and failure. hawkBit maintains a target-oriented state model while the DDI interface provides device-facing states and status messages. This allows the server to retain fleet-level deployment information while the robot performs detailed local operations. ([GitHub][1])
Security must be considered across both device communication and software authenticity. hawkBit supports several target authentication approaches, including target security tokens, while authentication and authorization mechanisms can also be integrated with external identity systems. The server can control which targets are permitted to retrieve commands and report feedback. For production robotics, this communication security should be combined with artifact integrity and digital-signature verification performed by the robot-side update mechanism. ([GitHub][4])
hawkBit can be deployed as a monolithic update server or integrated into a more distributed service architecture. The project provides separate modules for functions such as core processing, artifact handling, repository access, DDI, DMF, management APIs, and user interfaces. This allows an initial laboratory deployment to remain relatively simple while providing architectural paths toward larger fleet-management environments. The appropriate deployment structure depends on fleet scale, availability requirements, network topology, and operational responsibility.
For robotics, hawkBit should normally be positioned between CI/CD and the robot fleet. The CI/CD pipeline builds and tests the software, creates the deployable artifact, and establishes the release version. After validation and approval, the artifact and its metadata are registered with hawkBit. A fleet-management operator or automated deployment service then assigns the release to selected targets. The robot client receives the assignment, downloads the artifact, performs installation, and returns the resulting status.
This architecture also makes staged deployment possible. A new robot software version can first be assigned to a small validation group before being expanded to additional targets. The management layer can use hawkBit deployment information together with robot health and operational telemetry to determine whether further deployment should proceed. This creates a useful separation between software release management and physical robot operation, which is essential when an incorrect update could affect navigation, perception, communication, or other safety-relevant functions.
An Eclipse hawkBit integration is therefore best understood as an OTA orchestration layer rather than a complete robot-update implementation by itself. hawkBit provides the server-side concepts for targets, software modules, deployments, APIs, and update states, while the robot-side client remains responsible for platform-specific installation, activation, validation, and recovery. This separation allows the same OTA management infrastructure to support different robot hardware and update technologies while maintaining centralized fleet-level deployment control. ([GitHub][2])
09.05 Staged Rollout: Canary / Blue.Green Deployment [w/Code]
Staged rollout is a deployment strategy that gradually expands a new robot software release from a small controlled population to the broader fleet. Instead of updating every robot immediately, the OTA system defines deployment groups and promotes the release only after each stage provides sufficient operational evidence. This approach reduces the blast radius of software defects and is particularly important for robots because software changes can affect perception, localization, navigation, control, communication, and physical interaction with the environment.
The first stage is normally a validation or internal group containing representative robots. These devices should reflect the hardware, sensor configurations, software versions, and operating conditions expected in production. The purpose is not simply to verify that installation succeeds, but to determine whether the complete robot system continues to behave correctly after activation. Logs, health information, mission results, resource utilization, and application-specific telemetry should be collected before the release advances to a larger population.
Canary deployment is a staged rollout pattern in which a small subset of production robots receives the new release before the remaining fleet. The canary population should be large enough to expose meaningful operational variation but small enough to limit potential impact. Selection can be based on robot model, site, geography, hardware revision, mission type, or another representative characteristic. The server records which devices belong to the canary group and prevents the release from automatically reaching the wider fleet until the defined promotion conditions are satisfied.
A useful canary strategy separates technical success from operational success. Technical checks can verify that the package was downloaded correctly, its signature was accepted, installation completed, required processes started, and system resources remain within expected conditions. Operational checks examine robot-specific behavior such as navigation stability, localization quality, sensor availability, task completion, communication reliability, and safety-related events. A deployment should advance only when both installation health and operational evidence satisfy the release policy.
Promotion criteria should be defined before the canary deployment begins. These criteria can include deployment completion, application health, error rates, crash frequency, resource consumption, mission completion, communication status, and the absence of specified critical events. The exact thresholds should be derived from the robot system\'s validated requirements and operational baseline rather than chosen arbitrarily. A deployment controller can evaluate these conditions automatically and place the rollout on hold when the evidence is insufficient.
Blue-green deployment uses two separately identifiable software environments or system states, commonly called Blue and Green. One environment represents the currently active release while the other contains the candidate release. After the candidate environment is prepared and validated, traffic or execution responsibility can be switched from the active environment to the new one. The fundamental advantage is that the previous environment remains available as a recovery target, although the practicality of this pattern depends strongly on the robot\'s hardware and software architecture.
For robots, blue-green deployment is most naturally implemented at the software or container level when the hardware provides sufficient storage and compute resources. Two application versions can coexist as inactive and active environments, allowing the new version to be prepared without immediately replacing the running system. A similar principle can be applied to operating-system partitions through an A/B system-image architecture. The update mechanism must define which state is active, which state is staged, and how the system determines whether activation succeeded.
Blue-green deployment becomes more difficult when updates modify shared state or hardware interfaces. Database schemas, persistent configuration, firmware protocols, calibration data, and device drivers may not be simultaneously compatible with both software generations. Therefore, maintaining two software environments does not automatically guarantee safe rollback. Release design must consider backward and forward compatibility, state migration, persistent data handling, and hardware-interface changes before selecting blue-green as the deployment mechanism.
Canary and blue-green deployment can also be combined. A candidate release can first be prepared using a blue-green mechanism on selected robots, activated for a canary group, and then progressively promoted across additional groups. If the canary shows unacceptable behavior, the active state can be returned to the previous version where the underlying platform supports rapid switching. This combines population-level risk control from canary deployment with environment-level recovery capability from blue-green deployment.
The deployment controller should therefore maintain an explicit promotion state for every rollout. A release can move through states such as prepared, canary, validating, promoted, paused, failed, or completed. These states should be associated with deployment groups and individual device results. A paused rollout is particularly important because it prevents an uncertain release from continuing automatically while preserving the evidence required for an operator to investigate the situation.
Fleet segmentation is a critical part of staged deployment. A fleet may contain different robot models, compute platforms, sensor packages, firmware generations, customer sites, and operational roles. The first production group should therefore be selected using compatibility information and operational representativeness rather than simply choosing the first available devices. A release that works correctly on one robot configuration does not automatically establish compatibility with every configuration in the fleet.
Deployment waves provide a practical mechanism for expanding the rollout. After laboratory validation, the release can move to a small production canary, followed by progressively larger groups. Each wave has an explicit entry condition, observation period, and promotion decision. The size and composition of each wave should reflect the organization\'s risk tolerance, fleet heterogeneity, and ability to detect and respond to failures. The OTA server should preserve the relationship between each wave and the software release so that deployment history remains auditable.
Observation time is important because some defects do not appear immediately after installation. A robot may start normally while later exhibiting memory growth, sensor communication instability, navigation degradation, increased CPU or GPU utilization, or intermittent application failures. A staged rollout should therefore provide sufficient operational observation before promotion. The required observation period depends on the system and mission profile and should be established from validated operational requirements rather than assumed to be identical for every robot.
Staged rollout also interacts with robot scheduling. Updating an active robot during a mission can introduce unnecessary operational risk even when the software itself is valid. The deployment system should coordinate with fleet-management services to determine whether a robot is idle, charging, available for maintenance, or executing a mission. This allows the software release to be promoted at controlled operational boundaries rather than interrupting physical work.
The OTA server should maintain both desired and observed deployment state. Desired state identifies which release a target is intended to receive, while observed state records whether the robot has downloaded, installed, activated, and successfully operated that release. The difference between these states provides a basis for deployment monitoring. If a robot remains in an intermediate state, the system can distinguish an ordinary delayed update from a failed deployment and apply the appropriate policy.
Failure handling must be integrated into the staged rollout design rather than added afterward. When a canary robot fails installation or exhibits unacceptable behavior, the deployment controller should be able to pause further promotion. Depending on the failure type and recovery architecture, the affected robot can be rolled back, isolated for investigation, or returned to service using the previous validated software. The next OTA topic, status monitoring and automatic rollback, builds directly on this requirement.
Security and release integrity remain active throughout every deployment stage. The same release-signing and artifact-verification mechanisms used for normal OTA installation should apply to canary and production waves. Staging must never become a mechanism for bypassing release authorization. Deployment policies should also ensure that only compatible and approved software can be assigned to a target, while audit records preserve the identity of the release, target group, deployment decision, and resulting status.
Canary, blue-green, and staged rollout are therefore complementary rather than interchangeable concepts. Staged rollout controls how widely a release is distributed over time, canary deployment uses a limited representative population to obtain production evidence, and blue-green deployment maintains separable active and candidate software states to facilitate controlled activation and recovery. A production robot OTA architecture can use one or more of these mechanisms according to the software layer being updated, the robot platform\'s capabilities, and the operational risk associated with the change.
The overall objective is to transform OTA from a simple update operation into a controlled release process. CI/CD produces and validates the software, the OTA server authorizes the release, a staged policy determines the target population, canary robots provide early production evidence, blue-green or A/B mechanisms provide controlled activation where supported, and monitoring determines whether promotion should continue. This structure creates a measurable path from validated software to fleet-wide deployment while limiting the consequences of defects and providing clear transition points for investigation, rollback, and recovery. The chapter structure places this topic immediately before OTA status monitoring and automatic rollback, followed by fleet scheduling, CI/CD integration, security auditing, and large-scale fleet operation.
09.06 OTA Status Monitoring and Auto Rollback [w/Code]
OTA status monitoring is the operational layer that determines whether a software deployment is actually healthy after an update has been delivered to a robot. A successful download or installation does not necessarily mean that the robot is operating correctly. The monitoring system should therefore observe the complete transition from deployment initiation through download, installation, activation, health verification, and normal operation. This information becomes the basis for deciding whether a rollout should continue, pause, or trigger recovery.
The OTA server should maintain a deployment state for every target robot and every release. Typical states include pending, downloading, installing, activating, validating, successful, failed, paused, and rolled back. These states allow the fleet-management system to distinguish a robot that has not yet started an update from one that is actively installing software or one that has completed installation but failed its post-update health checks. State transitions should be persistent so that deployment history remains available after communication interruptions or server restarts.
Robot-side monitoring should provide both software and system-level health information. The OTA client can report whether required processes and containers are running, whether expected software versions are active, and whether installation scripts completed successfully. System indicators such as CPU utilization, memory consumption, storage capacity, temperature, network connectivity, and GPU availability can provide additional evidence. For robots, application health should also include important functions such as sensor availability, localization, navigation, communication, and mission execution.
A useful monitoring architecture separates deployment status from operational telemetry while connecting both through a common robot identity and release identifier. Deployment status answers whether an update was delivered and activated, while operational telemetry shows how the robot behaves afterward. Combining these data sources makes it possible to identify failures that are not visible from installation logs alone. For example, an update may install successfully while subsequently causing increased resource consumption, repeated application crashes, sensor communication failures, or degraded navigation behavior.
Health checks should be defined before a release enters production deployment. Basic checks can confirm that required services start correctly and that the expected software version is active. More advanced checks can evaluate application interfaces, sensor connectivity, communication channels, mission execution, and system resource conditions. The checks should reflect the robot\'s validated operational baseline. A release should not be considered healthy simply because a process is running; it should satisfy the conditions required for the robot to perform its intended function.
Automatic rollback uses these monitoring results to restore a previously validated software state when a deployment produces unacceptable results. The rollback mechanism should have access to a known-good version and should clearly define which conditions can trigger recovery. Depending on the update architecture, rollback may restore a previous container, application package, firmware image, or operating-system partition. The recovery operation should itself be monitored so that the server can confirm whether the robot successfully returned to the previous operational state.
Rollback triggers should distinguish transient conditions from genuine software failures. A temporary network interruption during an update should not automatically be treated as evidence that the release is defective. In contrast, repeated process crashes, failed health checks, incompatible interfaces, critical sensor failures, or persistent resource exhaustion can provide stronger evidence of an unsuccessful deployment. The deployment controller should therefore evaluate multiple signals and, where appropriate, require persistence or repeated failures before initiating automatic recovery.
For fleet-scale operation, monitoring should aggregate results across deployment groups and rollout waves. Operators need to know not only which individual robots failed but also whether failures are concentrated around a particular software version, hardware revision, site, or robot model. A sudden increase in failures among robots receiving the same release can indicate a release-level problem. Conversely, an isolated failure on one hardware configuration may require device-specific investigation rather than stopping the entire rollout.
The monitoring system should support clear rollout gates between deployment stages. When a canary group receives a new release, the OTA controller can evaluate installation success, health-check results, error rates, crashes, resource behavior, communication status, and mission-level results before allowing the next deployment wave. If the required conditions are not satisfied, the rollout can be paused while affected robots are investigated or rolled back. This directly connects the monitoring architecture with the staged rollout and canary deployment strategy described in the preceding section.
Robot operating conditions must also be considered when interpreting OTA status. A robot executing a mission may temporarily show unusual CPU utilization, network traffic, or sensor activity that is unrelated to the software update. Similarly, a disconnected robot cannot immediately report its current state even though its local software may remain healthy. The OTA system should therefore distinguish unavailable telemetry from confirmed failure and preserve the last known state together with its timestamp. This prevents communication loss from being incorrectly interpreted as software failure.
Observability should include structured logs, metrics, events, and deployment history. Each record should be associated with the robot identity, software release, component version, deployment stage, and relevant timestamp. Centralized storage allows operators to correlate installation events with subsequent robot behavior. Dashboards can provide fleet-level deployment progress and health summaries, while alerts can identify abnormal failure rates, repeated rollbacks, or critical post-update events that require immediate investigation.
Security and auditability remain part of the monitoring process. Status information should be authenticated so that the server can distinguish legitimate robot reports from unauthorized messages. Deployment and rollback events should be recorded with sufficient information to reconstruct what happened, when it happened, which release was involved, and which action was taken. Monitoring should not become a mechanism for bypassing release authorization or software-signature verification; instead, it should provide operational evidence for the same controlled release process.
Automatic rollback should also have safeguards because recovery operations can themselves affect robot availability. The system should prevent repeated update-and-rollback cycles from continuing indefinitely and should place a problematic device into an appropriate recovery or investigation state when repeated failures occur. A robot that cannot establish a healthy state after rollback may require operator intervention, maintenance, or a separate recovery procedure. The OTA architecture should therefore treat automatic rollback as one layer of fault recovery rather than assuming that every software failure can be solved by returning to the previous version.
The complete monitoring and rollback cycle can be understood as a closed feedback loop. A release is assigned to a target, the robot downloads and activates the software, the OTA client reports installation state, monitoring services observe system and application health, and the deployment controller evaluates the resulting evidence. Healthy results permit continued rollout, while defined failure conditions can pause deployment or initiate rollback. After recovery, the robot reports its restored state, allowing the fleet system to reconcile desired and actual software states.
For a robotics fleet, this closed-loop design transforms OTA from a one-way software distribution mechanism into a controlled operational system. The objective is not merely to deliver a new version but to maintain a verified relationship between software release, robot state, and operational behavior. This topic therefore forms the bridge between staged rollout and the following fleet OTA scheduling, CI/CD integration, security auditing, and large-scale fleet operation topics defined in the OTA architecture chapter. The chapter structure explicitly places status monitoring and automatic rollback after staged rollout and before fleet scheduling, package-build automation, security auditing, and large-scale OTA operation.
09.07 Fleet OTA Scheduling: Off-Hours / Group Deploy [w/Code]
Fleet OTA scheduling coordinates software deployment with the operational schedules of a robot fleet so that updates occur when they have the least impact on ongoing missions. Unlike a conventional server environment, robots may be continuously performing delivery, cleaning, patrol, inspection, or other physical tasks. An OTA system therefore needs to understand not only which software should be deployed, but also when a particular robot is available for maintenance and when an update can safely interrupt its normal operation.
Off-hours deployment is the basic scheduling pattern for reducing operational disruption. The fleet-management system can define maintenance windows during periods when robots are normally idle, charging, parked, or otherwise unavailable for missions. The OTA controller uses these windows to determine when an eligible robot may download, install, and activate an update. Downloading can often occur before the maintenance window, while activation and reboot operations can be reserved for the controlled period.
The scheduling system should distinguish between download time and activation time because they have different operational characteristics. A large software package may require substantial network bandwidth and storage activity, but downloading it does not necessarily require the robot to stop. Activation, on the other hand, may require application restarts, container replacement, controller resets, or system reboot. Separating these phases allows the OTA system to prepare updates in advance and minimize the time during which a robot is unavailable.
Robot availability should be evaluated before an update is started. The scheduler can consider whether a robot is executing a mission, waiting for a task, charging, docked, in maintenance mode, disconnected, or experiencing an existing fault. An update should normally be deferred when the robot is performing a mission that cannot safely be interrupted. Once the robot reaches an eligible operational state, the scheduler can initiate the prepared deployment without requiring an operator to manually restart the process.
Group deployment allows multiple robots to receive the same release according to a controlled scheduling policy. Robots can be grouped by model, hardware revision, software baseline, customer site, geographic area, operational role, or fleet-management group. The scheduler can then define different deployment windows and concurrency limits for each group. This is important because updating every robot in a site simultaneously could temporarily reduce fleet capacity even when the software release itself is fully validated.
Deployment concurrency should be controlled rather than allowing unlimited simultaneous updates. If too many robots download large artifacts at once, the network may become congested and the OTA server or artifact storage infrastructure may experience unnecessary load. More importantly, excessive simultaneous activation can reduce the number of robots available for operational work. A concurrency policy can therefore specify how many robots or what percentage of a group may be updated at the same time.
Scheduling should also consider fleet capacity requirements. A fleet may have a minimum number of robots that must remain available to satisfy customer operations. For example, if a site requires a certain operational capacity during the night, the OTA scheduler should avoid taking too many robots offline simultaneously. The scheduler can maintain an availability constraint and release robots for OTA only when sufficient capacity remains. This connects OTA scheduling directly with fleet task management rather than treating software maintenance as an independent process.
Priority is another important scheduling dimension. Critical security patches, emergency fault corrections, and safety-related updates may require different scheduling rules from ordinary feature releases. A routine application update can wait for the next maintenance window, while an urgent security release may require an earlier deployment according to the organization\'s emergency policy. The OTA system should therefore associate each release with a deployment priority and allow authorized policies to determine whether normal scheduling constraints may be modified.
Scheduling must remain aware of robot-specific prerequisites. A robot may require sufficient battery charge, available storage, stable network connectivity, a particular software baseline, or a maintenance state before installation can begin. These conditions should be evaluated immediately before activation rather than only when the deployment was originally scheduled. If the prerequisites are not satisfied, the scheduler should postpone the operation and select another eligible time instead of forcing an unsafe or unreliable update.
Network conditions can also influence the scheduling decision. A fleet operating over Wi-Fi, private 5G, LTE, or mixed connectivity may have different bandwidth characteristics during different periods. Large artifact downloads can be scheduled when network utilization is low, while smaller control messages can continue during normal operation. In a distributed fleet architecture, regional or site-level scheduling can further reduce network contention by staggering deployment across locations.
Time synchronization is important when scheduling fleet-wide operations. The server and robots should use consistent time references so that maintenance windows are interpreted correctly. The scheduler should also account for time zones when a fleet spans multiple geographic regions. A maintenance window defined for one site should not unintentionally be interpreted using the local time of another site. Storing deployment schedules with explicit time-zone information makes operational behavior easier to understand and audit.
A robust scheduler should handle missed maintenance windows and disconnected robots gracefully. A robot may be offline when its scheduled deployment window begins, or it may still be executing a mission when the window expires. The OTA system should preserve the deployment intent and reevaluate eligibility during the next appropriate window. This creates a persistent desired state rather than treating a missed schedule as a permanent deployment failure.
Group deployment should also integrate with staged rollout policies. A new release can first be scheduled for an internal validation group, followed by a small production canary group, and then progressively larger fleet groups. Each group can have its own maintenance window and concurrency limit. If monitoring detects unacceptable behavior in an earlier group, subsequent scheduled deployments should automatically pause according to the rollout policy. Scheduling therefore becomes another control mechanism for staged deployment rather than merely a calendar function.
The scheduler should maintain clear deployment states and events. Important information includes when a deployment was scheduled, when a robot became eligible, when the artifact download started, when activation began, whether the robot completed the update, and whether the deployment was deferred or failed. These events should be associated with the robot identity, release version, deployment group, and scheduling policy. Such records provide the operational history required to understand why a robot was or was not updated at a particular time.
Integration with the Fleet Management System is essential because the FMS normally owns the operational state of the robots. The OTA scheduler should consume information such as mission state, task queue, charging status, robot availability, site assignment, and maintenance state. The FMS can then coordinate task allocation around planned OTA operations. After an update completes successfully, the robot can return to the available pool; if the update fails, the FMS can keep the robot out of service while recovery or investigation takes place.
The scheduling architecture should support both centralized policy and local safety decisions. The central OTA service determines the desired deployment schedule and group policy, while the robot-side client verifies local conditions immediately before performing a disruptive operation. This prevents a stale scheduling decision from forcing an update when the physical robot has entered a new operational condition. The server can request deployment, but the robot must retain the ability to defer activation when local prerequisites or safety conditions are not satisfied.
A complete Fleet OTA scheduling cycle therefore connects release management, fleet state, maintenance windows, group policies, network conditions, deployment concurrency, robot availability, and post-update monitoring. A validated release is assigned to a deployment group, the scheduler identifies an appropriate maintenance window, eligible robots prepare the artifact, controlled activation begins, and the resulting deployment state is reported back to the OTA and fleet-management systems. Monitoring then determines whether the next group should proceed. This creates a coordinated operational loop in which software maintenance is performed without unnecessarily disrupting physical robot services.
For a production robot fleet, the objective of OTA scheduling is not simply to find an empty time slot. It is to coordinate software evolution with physical operations while preserving fleet capacity, network stability, deployment safety, and recovery capability. Off-hours deployment reduces disruption, group deployment controls the scope of each operation, concurrency limits protect infrastructure and fleet availability, and integration with the FMS connects software maintenance with real-world mission scheduling. Together, these mechanisms provide the operational foundation for reliable fleet-scale OTA deployment and prepare the architecture for the following topics of OTA package-build automation, CI/CD integration, security auditing, signing-chain management, and large-scale fleet OTA operation. The chapter structure places this scheduling topic after status monitoring and automatic rollback and before OTA package automation and CI/CD integration.
09.08 OTA Package Build Automation: CI/CD Integration [w/Code]
The provided source confirms that this topic belongs to Chapter 09, immediately after fleet OTA scheduling and before OTA security auditing, with the focus explicitly defined as "OTA Package Build Automation" and "CI/CD Integration." Based on that structure, the following text develops the topic around automated artifact creation, validation, signing, registry publication, and controlled OTA release integration.
OTA package build automation connects the software development lifecycle to the robot deployment lifecycle. Instead of manually collecting binaries, configuration files, container images, and firmware into an update package, the build pipeline automatically produces a reproducible OTA artifact from a defined source revision. The resulting package should contain the software payload together with metadata required for identification, compatibility checking, integrity verification, and deployment. This makes the OTA release process repeatable and reduces manual packaging errors.
A robotics CI/CD pipeline typically begins with a source-code commit or an approved release branch. The pipeline retrieves the exact source revision, resolves dependencies, builds the required software components, and executes automated tests. Depending on the robot architecture, the build may produce ROS 2 packages, container images, firmware binaries, operating-system images, configuration bundles, or AI model packages. These outputs should be generated from controlled build environments so that the resulting artifacts can be reproduced and traced to their source.
The build process should distinguish ordinary development builds from OTA release builds. Development builds are useful for rapid testing but should not automatically become production deployment artifacts. An OTA release build should be generated only after the required validation stages have completed and the release configuration has been explicitly defined. This separation provides a controlled transition from continuously changing development software to an immutable artifact that can be distributed across a robot fleet.
Artifact metadata is an essential part of automated package generation. A package should identify the software component, version, build number, source revision, target platform, architecture, dependencies, required runtime environment, and other compatibility information required by the OTA server and robot client. The metadata can also contain a cryptographic digest, creation timestamp, release channel, and references to the tests or validation results associated with the artifact. This information creates traceability between source code, build output, and deployed robot software.
For containerized robots, the CI/CD pipeline can build and publish container images to a controlled container registry. ROS 2 applications, perception services, navigation components, fleet agents, and AI inference services can each be packaged as versioned images. The OTA release process can then reference the exact image digests rather than relying only on mutable tags. This allows a robot to receive the same binary content that was tested and approved during the release process.
Firmware and system-image updates require additional controls because they can affect the underlying robot platform. The pipeline should verify target hardware, processor architecture, bootloader compatibility, partition requirements, and recovery mechanisms before producing a release artifact. A firmware image intended for one controller or hardware revision should not be accidentally packaged as a valid update for another platform. Compatibility checks therefore belong inside the automated release process rather than being left entirely to the operator.
Automated testing should occur before an artifact becomes eligible for OTA deployment. Unit tests can verify individual software components, while integration tests can examine communication between ROS 2 nodes, drivers, services, and hardware interfaces. System-level tests can verify that the complete robot software stack starts correctly and performs representative functions. Where available, simulation, digital-twin, hardware-in-the-loop, and representative robot tests can provide additional validation before a release is promoted toward production deployment.
The pipeline should also perform static analysis and dependency validation. Robotics software frequently combines operating-system packages, ROS 2 dependencies, third-party libraries, container base images, GPU runtimes, firmware components, and AI frameworks. A dependency change can therefore introduce compatibility or security problems even when the application\'s own source code has not changed. Automated dependency checks help identify these changes before the artifact enters the OTA release repository.
Build reproducibility is important for OTA because deployed software must be traceable after installation. The pipeline should record the source revision, build environment, compiler or runtime versions, dependency versions, configuration parameters, and generated artifact identifiers. Where practical, deterministic or reproducible build practices should be used so that the same source and build inputs produce equivalent outputs. This information becomes valuable when investigating field failures or comparing the software state of different robots.
After successful build and validation, the pipeline can generate the OTA package or release manifest. The manifest identifies the complete set of artifacts that together form the robot release. For example, a release may combine a ROS 2 container, navigation configuration, AI inference model, device firmware, and system-level configuration. Treating this combination as a single tested release prevents the OTA server from independently selecting incompatible component versions.
The next stage is artifact integrity and signing preparation. Each generated artifact should receive a cryptographic digest, allowing both the server and robot client to verify that the transferred content has not changed. A release manifest can also contain references to these digests. Digital signing can then establish that the release was produced or approved by an authorized release process. The signing operation should remain controlled and should not expose private signing keys unnecessarily to ordinary build workers.
After signing, the CI/CD pipeline can publish the approved artifact to the OTA infrastructure. Depending on the selected OTA platform, this may involve uploading the package to an artifact repository, registering metadata in a version registry, publishing a container image, or creating a release in an OTA management system such as Mender or Eclipse hawkBit. The important architectural principle is that publication should occur only after the artifact has passed the required automated and authorization gates.
The OTA server then becomes the controlled distribution layer for the release. CI/CD creates and validates the software, while OTA management determines which robots should receive it and when deployment should occur. This separation is important because a software build pipeline should not independently decide whether every production robot is updated. Deployment policy, fleet state, maintenance windows, staged rollout, and robot availability remain responsibilities of the OTA and fleet-management layers.
CI/CD integration should also support release channels. Development, testing, canary, and production channels can reference different deployment policies while using the same underlying artifact-generation process. A build can therefore progress through controlled environments without rebuilding the software unnecessarily. Promoting the same immutable artifact from one channel to another preserves the relationship between what was tested and what is ultimately deployed to production robots.
Automated promotion can connect test results with OTA deployment policies. When all required validation stages pass, the pipeline can mark the release as eligible for a controlled deployment. A deployment controller can then assign the release to a validation or canary group rather than immediately targeting the entire fleet. Monitoring results from those robots can determine whether the release should advance to the next deployment stage. This creates a continuous feedback loop between software engineering, OTA operations, and robot behavior.
The pipeline should retain deployment traceability after publication. A release record should allow operators to determine which source revision produced an artifact, which build generated it, which tests were executed, which signing process authorized it, where the artifact was stored, and which robot groups subsequently received it. When a field problem occurs, this chain allows the engineering and operations teams to connect a robot\'s observed behavior to the exact software release that was installed.
Rollback preparation should also be incorporated into package automation. A new OTA artifact should not be considered operationally complete unless the deployment system knows which previous validated release can be restored if necessary. The build pipeline can preserve immutable artifact versions while the OTA system maintains release lineage and rollback relationships. This allows automatic recovery mechanisms to return a robot to a known-good state without reconstructing an old software package manually.
For large robot fleets, build automation also reduces operational variation. Every release follows the same sequence of source retrieval, dependency resolution, compilation or image creation, testing, packaging, metadata generation, integrity calculation, signing, and publication. Standardized automation makes it easier to identify where a failure occurred and prevents individual engineers from applying undocumented packaging procedures. The resulting process becomes a repeatable software supply pipeline rather than a collection of manual release activities.
CI/CD integration should therefore be designed as a controlled chain rather than a single build command. Source changes generate candidate builds, automated validation determines whether the software is technically acceptable, release packaging creates immutable OTA artifacts, signing establishes release authenticity, and publication makes the artifact available to the OTA management layer. Deployment scheduling and staged rollout then determine how the validated release reaches the physical fleet, while monitoring provides feedback about its behavior after activation.
In a robotics environment, this architecture creates a direct connection between software engineering and physical robot operations. The CI/CD system is responsible for producing a known and tested software artifact, while the OTA system is responsible for distributing that artifact safely according to fleet policy. This separation allows development velocity to increase without allowing uncontrolled software changes to bypass deployment governance. It also provides the foundation for integrating artifact registries, Mender or hawkBit, container registries, automated testing, release signing, fleet management, monitoring, and rollback into one continuous software delivery lifecycle.
The overall OTA package automation flow can therefore be viewed as a closed engineering-to-operation pipeline: source commit, automated build, dependency and security checks, unit and integration testing, system or simulation validation, package and manifest generation, integrity calculation, controlled signing, artifact publication, OTA release registration, staged fleet deployment, operational monitoring, and feedback into the next software iteration. This structure connects the preceding fleet scheduling topic with the following OTA security audit, signing-chain, and supply-chain topics while establishing the CI/CD foundation required for reliable large-scale robot software delivery.
09.09 OTA Security Audit: Signing Chain / Supply Chain
OTA security auditing establishes the evidence required to prove that robot software was produced, approved, distributed, installed, and operated through an authorized process. In a production fleet, security cannot end with encrypted communication between the OTA server and robot. The complete software delivery path must remain traceable from source code and build systems through artifact repositories, signing services, deployment infrastructure, and the software finally activated on each robot.
The software signing chain creates a cryptographic relationship between an authorized release process and the artifact accepted by the robot. After a package is built and validated, a cryptographic digest is calculated and the approved release information is digitally signed. The robot-side update client verifies the signature before installation or activation. This prevents an artifact from being trusted merely because it was downloaded from the expected server or repository.
Signing keys require stronger protection than ordinary build credentials because compromise of a release-signing key can undermine the trust model of the entire OTA system. Private keys should therefore be isolated from normal build workers and protected through controlled signing infrastructure such as a hardware security module, secure key-management service, or dedicated signing server. CI/CD systems should request signing operations through authenticated interfaces rather than directly exposing long-lived private keys inside build environments.
The signing chain should also define separate responsibilities for artifact creation, release approval, and signing authorization. A successful build should not automatically provide unrestricted authority to sign production software. Development and test artifacts may use different trust policies from production releases. For higher-assurance environments, release approval can require an explicit authorization step before the signing service permits an artifact to enter the production OTA channel.
Certificate and key lifecycle management is part of OTA security architecture. Signing credentials require controlled generation, storage, access, rotation, renewal, revocation, and retirement. Robots must have a trustworthy mechanism for recognizing currently authorized signing identities while rejecting revoked or unauthorized credentials. Key rotation should be planned before existing credentials expire so that fleet devices can transition to new trust material without losing the ability to receive legitimate updates.
Artifact integrity and authenticity address different security questions. Integrity verification determines whether the artifact has changed since its expected digest was calculated, while authenticity verification determines whether the artifact was authorized by a trusted signing identity. Both are necessary. A hash alone can detect modification but cannot prove who approved the package, while a signature without appropriate verification and trust management cannot establish a reliable software release chain.
The OTA supply chain extends beyond software written directly by the robotics organization. Robot releases commonly contain operating-system packages, ROS 2 libraries, container base images, GPU runtimes, device drivers, third-party libraries, AI frameworks, firmware, and other external dependencies. A vulnerability or malicious modification in any of these components can propagate into the final robot artifact even when the organization\'s own application source code remains unchanged.
Software Bill of Materials, or SBOM, information provides visibility into these dependencies. An automated build pipeline can generate or associate an SBOM with each release so that operators know which software components and versions are included in a deployed artifact. When a vulnerability is later discovered in a dependency, the organization can use this information to identify affected releases and determine which robots may contain the vulnerable component without manually reconstructing old builds.
Dependency provenance is also important. The pipeline should record where packages, container images, libraries, firmware, and other build inputs originated and which exact versions or digests were used. Mutable or unverified dependencies weaken reproducibility because the same build definition may later retrieve different content. Pinning critical dependencies to known versions or immutable digests helps maintain a verifiable relationship between the validated build and the artifact distributed through OTA.
Build infrastructure itself is part of the trusted computing path. A compromised build worker could alter binaries after source review but before signing, making source-code controls alone insufficient. CI/CD environments should therefore restrict access, isolate sensitive jobs, protect credentials, record build activity, and minimize unnecessary privileges. Production release jobs should execute through controlled pipelines so that arbitrary developer workstations do not become unofficial paths for creating trusted fleet software.
Artifact repositories and registries require similar protection. Access controls should distinguish users or services that can read artifacts from those authorized to publish, modify, promote, or delete them. Production artifacts should preferably be immutable after publication so that a previously approved release identifier cannot silently refer to different binary content. Repository events should be logged because changes to release metadata or stored artifacts can directly affect downstream OTA deployment.
Security auditing connects these controls through persistent records. An audit trail should capture the source revision, build identifier, artifact digest, validation results, signing identity, approval event, repository location, release version, deployment group, target robot, installation result, rollback event, and relevant timestamps. These records allow an organization to reconstruct how a specific software version reached a particular robot and whether every required control was successfully completed.
Audit logs themselves must be protected from unauthorized modification. If an attacker can alter deployment history after compromising a system, the logs cannot provide reliable evidence during investigation. Centralized logging, restricted write permissions, controlled retention, authenticated event sources, and appropriate integrity protection can strengthen the audit trail. Time synchronization is also important because events from CI/CD, signing services, OTA servers, fleet systems, and robots must be correlated accurately.
The robot should enforce trust locally rather than depending entirely on the OTA server. Even if the server instructs a robot to install a package, the robot-side updater should verify the artifact according to its configured trust policy before activation. This creates an additional security boundary between distribution authorization and execution authorization. A compromised network service should not automatically gain the ability to execute arbitrary software on the physical robot.
Anti-rollback policy may also be required for security-sensitive components. Operational rollback is valuable when a new release fails, but returning to an older version can reintroduce a known vulnerability. The OTA architecture must therefore distinguish a known-good operational version from a version that is still permitted by security policy. Firmware counters, minimum permitted versions, release metadata, or platform-specific secure-boot mechanisms can help enforce this distinction where required.
Secure boot and OTA signing can form complementary layers of trust. OTA verification controls which software packages are accepted during an update, while secure boot can verify trusted software during the device startup process. When combined appropriately, the chain can extend from trusted boot components through the operating system and robot applications. The exact implementation depends on the robot compute platform, boot architecture, firmware capabilities, and required assurance level.
Supply-chain security should also include vulnerability assessment before release publication. Dependency scanners, container-image scanners, static analysis, secret detection, and policy checks can be integrated into CI/CD gates. Their results should contribute to release approval rather than existing only as informational reports. When a critical issue violates the defined policy, the artifact should remain in a non-production state until the problem is resolved or an authorized exception is documented.
The same controls must remain effective during staged rollout and fleet deployment. Canary releases should not bypass production signing requirements simply because they target fewer robots. Likewise, emergency releases should follow an explicitly defined emergency authorization process rather than disabling security controls. Deployment urgency may change approval timing, but it should not make artifact identity, integrity, authorization, and auditability optional.
Incident response depends heavily on the information created by this security architecture. When a suspicious artifact, compromised dependency, leaked credential, or vulnerable component is discovered, operators need to identify affected releases and robots quickly. Release metadata, SBOM records, artifact digests, signing history, and deployment records allow the organization to determine the scope of exposure and coordinate replacement packages, credential revocation, deployment suspension, or recovery operations.
A mature OTA security model therefore treats trust as a continuous chain rather than a single signature check. Source control establishes the software revision, CI/CD produces a traceable build, validation provides technical evidence, signing authorizes the release, repositories preserve immutable artifacts, OTA policy controls distribution, the robot independently verifies the package, and monitoring records the resulting operational state. Each stage contributes evidence that can be inspected during normal operations or security investigations.
The complete security flow connects software supply-chain controls with physical fleet governance. Source and dependency provenance establish what entered the build, automated checks evaluate the candidate release, protected signing infrastructure establishes authorization, repositories preserve the approved artifact, OTA services control deployment, and robot-side verification determines whether the software may execute. Audit records connect every transition so that the organization can trace a release from engineering systems to individual robots.
For robot fleets, this architecture is especially important because a compromised software supply chain can ultimately influence physical behavior. OTA security must therefore protect not only confidentiality and network communication but also software provenance, artifact integrity, signing authority, dependency transparency, deployment authorization, device-side verification, and forensic traceability. Together, these mechanisms establish the security foundation required for controlled large-scale OTA operation across heterogeneous production robot fleets.
09.10 Large-Scale Fleet OTA Operation Case Study
Large-scale fleet OTA operation transforms software deployment from a device-maintenance task into a coordinated fleet-management process. When hundreds or thousands of robots operate across multiple customer sites, software releases cannot be treated as independent updates. The OTA platform must coordinate release versions, robot groups, operational schedules, network capacity, deployment health, rollback policies, and audit records while preserving the availability of the physical robot service.
Consider a distributed service-robot fleet operating in warehouses, commercial buildings, factories, campuses, and logistics facilities. Individual sites may contain different robot models, hardware revisions, sensor configurations, compute platforms, and software baselines. A single release therefore cannot automatically target every connected device. Fleet inventory and device metadata become fundamental inputs for determining which robots are technically compatible and operationally eligible for each OTA release.
The operational architecture separates software creation from fleet deployment. CI/CD produces validated and signed OTA artifacts, while the OTA management platform registers releases and controls their distribution. A Fleet Management System provides mission status, charging state, site assignment, robot availability, and maintenance information. Monitoring services collect deployment and runtime health data. Together, these systems form a closed operational loop rather than a simple server-to-device download mechanism.
Before deployment begins, the release is associated with explicit compatibility rules. These rules may specify robot model, processor architecture, hardware revision, operating-system baseline, firmware dependency, container runtime, application version, or required configuration. The OTA platform evaluates fleet inventory against these constraints and creates an eligible target population. Robots that do not satisfy the release requirements remain outside the deployment group rather than being allowed to fail during installation.
A production rollout normally begins with a small validation population. Internal robots or laboratory systems receive the release first so that package installation, reboot behavior, service startup, communication, and basic robot functions can be verified. Successful technical installation alone is insufficient. The validation process should also confirm that navigation, perception, mission execution, charging, fleet communication, and other relevant operational functions remain within expected behavior.
After internal validation, the release can move to a production canary group. The canary population should be large enough to expose realistic operating conditions while remaining small enough to limit the impact of an unexpected defect. Deployment monitoring compares installation success, robot health, application stability, mission completion, resource utilization, and other defined indicators against release criteria. The next rollout stage begins only when the observation period satisfies the required promotion policy.
The fleet is then divided into controlled deployment waves. Groups can be organized by customer site, geographic region, robot model, hardware revision, operational role, or service priority. A deployment wave may contain only a limited number or percentage of eligible robots. This prevents a defective release from simultaneously affecting the entire fleet and gives operations teams time to identify problems before the software reaches a larger population.
Fleet capacity must remain available throughout the rollout. If a facility depends on twenty robots for normal operations, taking all twenty offline for software activation would create an avoidable service interruption. The scheduler therefore coordinates deployment with the FMS and maintains a defined number of available robots. Updates can proceed while robots are idle, charging, docked, or placed in maintenance mode, with additional robots entering the deployment group as updated units return to service.
Package download and activation can be scheduled independently. Large artifacts may be downloaded in advance while robots continue normal work, reducing the amount of data transferred during the maintenance window. Activation is then performed when the robot reaches an eligible state. This approach is particularly useful when fleets depend on shared Wi-Fi, LTE, private 5G, or constrained site connections because network consumption can be distributed over a longer period.
Network-aware rollout becomes increasingly important as fleet size grows. If hundreds of robots request multi-gigabyte artifacts simultaneously, the resulting traffic can overload site networks, cloud gateways, registries, or OTA infrastructure. Concurrency controls limit the number of simultaneous downloads and activations. Regional mirrors, local caches, or site-level artifact distribution can further reduce repeated wide-area transfers when many robots at the same location require identical software.
Every deployment progresses through observable states such as scheduled, downloading, downloaded, installing, activating, verifying, succeeded, deferred, failed, or rolled back. The OTA platform aggregates these states into fleet-level views while retaining individual robot records. Operators can therefore understand both overall rollout progress and the exact condition of a specific robot. Deployment state should remain distinct from runtime telemetry so that installation success is not mistaken for operational health.
Post-update verification determines whether an updated robot can safely return to service. The robot can perform checks for software version, service startup, container health, sensor availability, controller communication, storage condition, network connectivity, and other platform-specific requirements. The FMS should return the robot to the available task pool only after the required checks succeed. A robot that fails verification remains isolated from normal mission allocation until recovery is completed.
Automatic rollback provides a recovery path when a release fails after activation. Depending on the platform, recovery may restore a previous container image, application package, firmware slot, system partition, or complete release manifest. Rollback should return the robot to a previously validated state rather than simply selecting an arbitrary older version. Security policy must also prevent rollback to versions that are no longer permitted because of known vulnerabilities.
Fleet-wide monitoring determines whether rollout should continue. Failure rate, robot health, service crashes, mission abnormalities, communication loss, resource exhaustion, or other release-specific indicators can trigger a deployment pause. When predefined thresholds are exceeded, the OTA controller should stop promotion to subsequent groups while already affected robots are investigated or rolled back. This converts monitoring data into an active deployment-control mechanism rather than a passive dashboard.
Disconnected robots require persistent deployment intent. Some devices may be offline during a scheduled maintenance window because of network conditions, customer shutdowns, transportation, maintenance, or power loss. The OTA system should retain the desired release state and reevaluate the robot when connectivity returns. Before installation, compatibility, battery level, available storage, operational status, and current release policy should be checked again because conditions may have changed during the disconnection.
Large-scale operation also requires clear exception handling. Individual robots may fail because of insufficient storage, corrupted downloads, incompatible local configuration, hardware faults, or unsuccessful restart procedures. Such failures should not automatically stop an otherwise healthy fleet rollout unless the defined policy threshold is reached. The system should classify failures, retry recoverable operations when appropriate, isolate persistent exceptions, and provide operators with enough diagnostic information for targeted intervention.
Security controls remain active throughout every deployment wave. Robots verify artifact integrity and digital signatures before installation, while release metadata identifies approved versions and target platforms. Signing records, artifact digests, SBOM information, approval events, and repository history provide supply-chain traceability. Emergency deployment procedures may accelerate operational approval, but they should preserve artifact verification, signing requirements, authorization controls, and auditability.
Auditability becomes essential when thousands of deployment events occur automatically. For any robot, operators should be able to determine which version was previously installed, which release was assigned, when the artifact was downloaded, which signature was verified, when activation occurred, what validation result was produced, and whether rollback was required. The same records allow fleet-level analysis of deployment performance across sites, models, software versions, and rollout groups.
Operational dashboards should therefore present both deployment progress and service impact. Useful views include the number of eligible robots, completed updates, deferred devices, active installations, failures, rollbacks, disconnected robots, and robots awaiting verification. These indicators should be correlated with fleet availability and operational health. A release that achieves a high installation percentage but causes degraded mission performance should not be considered a successful fleet deployment.
The complete operating cycle connects CI/CD, release signing, artifact storage, OTA management, fleet scheduling, robot-side verification, monitoring, and rollback. A validated artifact enters the OTA platform, compatible robots are identified, a small validation group receives the release, monitoring evaluates the result, and progressively larger deployment waves follow. Each robot returns to service only after verification, while abnormal results can pause promotion or trigger recovery.
At large scale, the central design principle is controlled progression rather than maximum deployment speed. The objective is to move a verified software release through a heterogeneous fleet while maintaining service capacity, limiting failure scope, protecting network resources, preserving security controls, and maintaining complete traceability. Combining staged rollout, group scheduling, concurrency control, health monitoring, automatic rollback, signing, and fleet-management integration turns OTA into a dependable operational capability for production robot fleets.
Author: Richard Minsoo Go
LinkedIn: Richard Minsoo Go










Top comments (0)