STATERA Library · Series 08 · Volume 25 · Chapter 12 · EN
Source: https://github.com/Statera-Library/08_Vol_25_Testing_Validation_and_Benchmarking/blob/main/Chapter_12_Testing_Case_Studies/Chapter_12_Testing_Case_Studies_EN.md
Volume 25. Testing Validation and Benchmarking
Chapter 12. Testing Case Studies
12.01. Indoor AMR Full Test Suite Build and CI Case
An indoor autonomous mobile robot requires a test strategy that treats software quality, navigation performance, hardware behavior, and operational safety as one integrated system rather than independent verification activities. In this case, the engineering team constructed a full test suite spanning unit, integration, simulation, hardware-in-the-loop, system, and field testing, consistent with the broader testing structure used for robotics software. Volume_25_Testing_Validation_an...
Development began with deterministic unit tests for components whose behavior could be verified without a complete robot. Localization utilities, coordinate transformations, velocity limiters, path-processing functions, obstacle filters, state machines, recovery logic, and diagnostic modules were isolated from physical dependencies. Sensor and actuator interfaces were replaced with mocks or stubs so that normal conditions, boundary values, invalid messages, timeouts, and communication failures could be reproduced reliably.
ROS 2 integration testing then verified behavior that emerged only when multiple nodes operated together. Test environments launched selected portions of the production node graph and checked topic connectivity, message timing, lifecycle transitions, service responses, action execution, parameter loading, and failure propagation. Navigation tests injected controlled odometry, map, LiDAR, and obstacle information while recording planner commands and robot states to determine whether subsystem contracts remained valid after software changes.
The next layer used simulation to exercise the complete navigation stack against reproducible indoor scenarios. Virtual warehouses and corridors contained intersections, narrow passages, static obstacles, moving pedestrians, blocked routes, docking stations, and changing traffic conditions. Scenario definitions included initial pose, destination, obstacle configuration, expected behavior, timeout limits, and measurable acceptance criteria. Simulation therefore transformed complex robot behavior into repeatable regression tests rather than relying exclusively on manual demonstrations. Volume_25_Testing_Validation_an...
Navigation validation was organized around measurable mission outcomes. Each scenario recorded task success, completion time, traveled distance, localization error, minimum obstacle clearance, replanning frequency, recovery count, velocity behavior, and collision events. Repeated trials were used where stochastic behavior or dynamic obstacles could influence results. This approach allowed the team to distinguish a functional regression from normal run-to-run variation and supported systematic AMR navigation benchmarking. Volume_25_Testing_Validation_an...
Hardware-in-the-loop testing provided the transition from software-dominated validation to physical-system verification. Real controllers, safety interfaces, communication devices, or selected compute hardware were connected to simulated plant and sensor signals. Fault injection reproduced encoder loss, delayed sensor packets, corrupted measurements, actuator faults, communication interruptions, and abnormal safety inputs without exposing a complete mobile robot or nearby personnel to unnecessary risk. Volume_25_Testing_Validation_an...
Safety functions were deliberately separated from ordinary mission-success criteria. Emergency stop behavior, controlled stopping, speed limitation, obstacle detection, safety scanner zones, command watchdogs, communication-loss responses, and actuator-disable mechanisms were verified through explicit procedures. Measurements focused not only on whether a safe state was reached but also on response latency, stopping behavior, state transitions, diagnostic reporting, and recovery conditions after the initiating fault disappeared.
A dedicated physical test area was then used for system-level AMR testing. Representative corridors, shelves, intersections, docking targets, pedestrian crossings, reflective surfaces, and restricted zones reproduced important operational conditions. Instrumentation captured ROS 2 bags, controller states, localization estimates, safety events, network statistics, system logs, CPU and memory utilization, and mission-level KPIs so that every significant failure could be reconstructed rather than classified only from operator observation. Volume_25_Testing_Validation_an...
Endurance testing extended the suite beyond individual navigation missions. The AMR repeatedly executed routes, charging cycles, docking operations, task transitions, and recovery procedures over long durations. Engineers monitored memory growth, process restarts, localization drift, thermal behavior, communication degradation, storage consumption, sensor instability, and accumulated timing errors. Defects that were almost invisible during short acceptance tests became reproducible when the same workload was executed hundreds or thousands of times.
The central engineering challenge was integrating these layers into continuous integration rather than maintaining them as separate laboratory procedures. Every source-code change triggered fast static checks and unit tests first. Successful builds progressed to software integration tests and selected simulation scenarios. More expensive regression simulations and hardware-dependent tests were scheduled for appropriate branches, release candidates, or dedicated runners, following the volume\'s explicit emphasis on CI-integrated testing and simulation regression pipelines. Volume_25_Testing_Validation_an... Volume_25_Testing_Validation_an...
Test selection was risk-based because executing the entire AMR validation campaign for every commit would have produced excessive feedback latency. Changes affecting isolated utilities could stop after fast software tests, while modifications to localization, planning, control, safety, or sensor interfaces triggered progressively broader suites. Dependency metadata and historical failure information helped determine which scenarios were relevant, allowing CI resources to concentrate on areas where a change could realistically alter robot behavior.
A practical pipeline also required disciplined test-data management. Maps, robot configurations, simulation worlds, sensor recordings, expected outputs, calibration files, and scenario definitions were versioned alongside compatible software releases. Random seeds and environment parameters were recorded where reproducibility mattered. Test artifacts were retained with build identifiers so that an engineer investigating a regression could reconstruct the exact software, configuration, environment, input data, and acceptance thresholds used during the failed execution.
Pass-or-fail decisions were derived from machine-readable criteria rather than visual inspection wherever possible. A mission could require successful arrival within a defined time, zero collisions, acceptable localization error, valid safety-state transitions, bounded command latency, and minimum clearance from obstacles. The pipeline aggregated these results into reports that distinguished functional failures, performance regressions, infrastructure errors, and flaky tests, preventing unreliable automation from gradually becoming accepted as normal behavior.
Field failures were also converted into permanent regression assets. When deployed robots encountered unusual layouts, sensor degradation, traffic interactions, or recovery failures, relevant telemetry and environmental conditions were reconstructed as simulation or integration scenarios. This created a feedback loop between production operation and pre-deployment verification, matching the volume\'s shift-right model in which production monitoring, field data, automated regression triggers, and continuous safety validation extend the conventional CI process. Volume_25_Testing_Validation_an...
The completed AMR test architecture therefore functioned as a verification chain rather than a collection of unrelated test scripts. Unit tests detected local software defects quickly, integration tests protected ROS 2 interfaces, simulation provided scalable behavioral coverage, HIL exposed hardware-dependent failures, physical testing verified real-world behavior, and field telemetry continuously supplied new regression cases. Failures discovered at a later stage were systematically pushed toward earlier and less expensive stages whenever reproduction was technically possible.
The case demonstrates that effective AMR continuous integration is ultimately an evidence-management system for robot behavior. Each software revision accumulates evidence about correctness, navigation capability, timing, robustness, safety, and regression risk before deployment approval. By combining automated testing with simulation, HIL, controlled field trials, measurable benchmarks, and production feedback, the organization can evolve an indoor AMR rapidly while maintaining traceability between software changes and verified physical behavior.
12.02. Outdoor AMR SOTIF Scenario Test Coverage Case
The outdoor autonomous mobile robot presents a fundamentally different validation problem from an indoor AMR because its operational environment cannot be fully constrained. Roads, sidewalks, industrial yards, campuses, and public facilities expose the robot to continuously changing illumination, weather, terrain, traffic, vegetation, pedestrians, and sensor conditions. A SOTIF-oriented scenario test program therefore evaluates not only component failures but also hazardous behavior arising when correctly functioning perception and autonomy systems encounter difficult operating conditions.
The test campaign begins by defining the operational design domain, or ODD, in measurable terms. The ODD describes permitted road and pathway types, speed ranges, slopes, surface conditions, illumination, precipitation, temperature, visibility, pedestrian density, communication availability, and positioning conditions. These boundaries establish where the outdoor AMR is expected to operate and provide the reference against which scenario coverage and validation completeness can be evaluated.
Scenario construction starts with nominal operational situations before progressively introducing challenging conditions. Representative scenarios include straight driving, intersections, curves, narrow passages, ramps, mixed pedestrian areas, parked vehicles, temporary obstacles, construction zones, and transitions between open and constrained spaces. Each scenario is parameterized so that speed, obstacle position, trajectory, visibility, surface condition, and environmental complexity can be systematically varied instead of being represented by a single fixed test.
SOTIF scenario analysis focuses especially on triggering conditions that can expose functional insufficiencies even when no hardware or software fault exists. Examples include low-contrast pedestrians, partially occluded objects, reflective surfaces, strong shadows, direct sunlight, rain-contaminated sensors, unusual obstacle geometry, ambiguous drivable boundaries, and temporary localization degradation. The objective is to discover combinations in which perception, localization, planning, or control remains technically operational but produces an unsafe or insufficient response.
The scenario library is organized using functional, logical, and concrete representations. A functional scenario describes the semantic situation, such as an AMR approaching a pedestrian crossing. A logical scenario converts that situation into parameter ranges for robot speed, pedestrian velocity, crossing angle, detection distance, illumination, and friction. Concrete scenarios assign specific values to these parameters so that individual experiments can be executed deterministically and reproduced during regression testing.
Simulation provides the first scalable mechanism for exploring this scenario space. Thousands of parameter combinations can be executed using variations in obstacle behavior, sensor noise, weather, lighting, localization uncertainty, and vehicle dynamics. Automated scenario execution records collisions, minimum separation distance, time to collision, emergency braking events, path deviation, localization error, mission completion, and recovery behavior. Large-scale simulation therefore identifies potentially hazardous regions before expensive physical testing begins.
Coverage measurement must represent the explored operational space rather than simply count executed test cases. The engineering team tracks combinations of environmental conditions, road geometry, actor behavior, robot state, sensor condition, and autonomy response. Parameter bins and multidimensional coverage matrices reveal combinations that remain insufficiently tested. A campaign with ten thousand similar clear-weather runs may therefore provide less meaningful coverage than a smaller but systematically diversified collection of scenarios.
Risk-based prioritization prevents scenario explosion from making the validation campaign impractical. Scenarios are ranked according to potential severity, exposure, controllability, uncertainty, and observed system weakness. High-priority scenarios receive denser parameter sampling and more physical validation, while lower-risk regions may rely primarily on simulation evidence. Near-collision events, unexpected emergency stops, localization instability, and perception uncertainty can automatically increase the priority of related scenario families.
Sensor perception receives particular attention because outdoor conditions can create performance limitations without explicit sensor failure. Camera exposure can degrade under backlighting, LiDAR returns can change with rain or reflective materials, and GNSS accuracy can deteriorate near buildings or vegetation. Tests therefore vary individual and combined sensing conditions while monitoring object detection confidence, tracking continuity, localization uncertainty, obstacle-map stability, and the resulting navigation decisions rather than evaluating sensors independently.
Localization and navigation scenarios evaluate how the AMR behaves when positioning quality changes gradually rather than disappearing completely. GNSS multipath, temporary RTK correction loss, feature-poor regions, wheel slip, IMU bias, and inconsistent map observations are introduced with controlled severity. Acceptance criteria examine whether uncertainty is detected, whether speed or operating mode is appropriately adjusted, and whether the robot transitions toward a minimal-risk condition when reliable navigation can no longer be maintained.
Dynamic interaction scenarios address pedestrians, bicycles, vehicles, and other mobile equipment whose future motion is uncertain. Tests vary approach direction, relative velocity, occlusion, sudden entry, stopping behavior, and trajectory changes. Successful performance requires more than avoiding collision: the robot should maintain suitable separation, avoid excessive oscillation or unnecessary emergency braking, and behave predictably enough that nearby people can understand and safely interact with its motion.
Environmental testing extends the scenario matrix across day, night, rain, wet pavement, glare, shadows, dust, and surface irregularities within the intended ODD. Rather than treating these as isolated environmental qualification tests, they are combined with navigation situations. A pedestrian crossing at night on wet pavement, for example, evaluates perception degradation, stopping distance, planning response, and vehicle dynamics simultaneously, exposing interactions that component-level testing may not reveal.
Selected high-risk simulation scenarios are transferred to controlled physical tests using equivalent geometry, robot state, actor trajectories, and environmental conditions whenever practical. Differences between simulated and real outcomes are measured to identify the simulation-to-real gap. Scenarios whose physical behavior differs significantly from simulation receive additional investigation, model calibration, or conservative acceptance margins before simulation results are allowed to contribute strongly to validation evidence.
Field testing then evaluates scenario families that cannot be represented adequately in controlled facilities. Instrumented outdoor runs collect synchronized perception outputs, localization estimates, planning decisions, control commands, safety events, environmental observations, and ground-truth references where available. Every significant disengagement, unexpected stop, near miss, or abnormal trajectory becomes a candidate scenario for reconstruction, analysis, and inclusion in the permanent regression library.
Scenario coverage evolves continuously because deployment reveals conditions that were not anticipated during initial design. Production telemetry can identify new combinations of weather, geometry, traffic behavior, sensor uncertainty, or localization degradation. These observations are classified and compared against the existing scenario database. Missing or weakly represented conditions generate new simulation and physical tests, connecting field operation with continuous validation and preventing the test suite from becoming static.
The final SOTIF evidence is therefore not a claim that every possible outdoor event has been tested. Instead, it demonstrates a systematic process for defining the ODD, identifying potentially hazardous scenarios, exploring triggering conditions, measuring scenario-space coverage, prioritizing residual risk, and feeding operational discoveries back into regression testing. This provides a defensible argument that known insufficiencies have been addressed and unknown unsafe scenarios are being progressively reduced.
This outdoor AMR case illustrates why scenario coverage must become a lifecycle capability rather than a one-time release activity. Simulation, controlled testing, field validation, telemetry analysis, and automated regression form a closed evidence loop. As software, sensors, maps, operating environments, and AI models change, the same scenario framework can determine which assumptions have changed, which tests must be repeated, and whether the robot remains acceptably safe throughout continued deployment.
12.03. Mobile Manipulator Safety Function HIL Case
A mobile manipulator combines the hazards of an autonomous mobile base with those of a multi-axis robotic arm, creating safety interactions that cannot be validated adequately by testing either subsystem alone. The HIL campaign therefore treats mobility, manipulation, safety sensing, drive control, and supervisory software as one coordinated safety system. The objective is to verify that safety functions remain deterministic when hazardous motion, communication faults, sensor abnormalities, or conflicting commands occur during combined robot operation.
The test architecture connects production controllers and safety-relevant hardware interfaces to a real-time simulation of the mobile platform, manipulator, actuators, sensors, and surrounding environment. Depending on the verification stage, actual safety PLCs, motor controllers, fieldbus interfaces, emergency-stop circuits, and compute units remain in the loop. Simulated plant models generate encoder, velocity, torque, position, proximity, and safety-zone signals while receiving the same commands that would normally drive physical hardware.
A formal safety-function matrix establishes what must be verified before individual HIL scenarios are created. Each function is associated with its initiating condition, monitored signals, expected state transition, permitted motion, response-time requirement, reset condition, and evidence to be recorded. Emergency stop, safe torque off, safely limited speed, protective stop, communication-loss response, workspace limitation, collision-related stopping, and mode-dependent motion restrictions become independently measurable functions rather than informal observations of robot behavior.
Emergency-stop testing verifies the complete chain from input activation to elimination of hazardous motion. The HIL environment triggers emergency-stop signals while the base is moving, the arm is moving, or both are operating simultaneously. Measurements capture input recognition, safety-controller processing, drive response, torque removal, simulated stopping behavior, and final state. Repeated tests at different velocities and manipulator configurations expose timing dependencies that may not appear during stationary acceptance checks.
Safe Torque Off, or STO, testing focuses on whether torque-producing capability is removed according to the defined safety behavior. STO requests are injected during representative base and manipulator states while drive feedback and command paths are monitored. The test verifies that ordinary motion software cannot override the safety action and that restoration of the safety signal does not automatically produce unexpected motion. Restart behavior is therefore evaluated as carefully as the initial transition into the safe state.
Safely Limited Speed, or SLS, requires a different verification strategy because motion remains permitted within defined limits. The HIL simulator creates conditions in which commanded or simulated velocity approaches and exceeds configured thresholds. Tests determine whether speed supervision detects the violation, whether the appropriate protective response occurs within the required interval, and whether different operating modes apply the correct limits to the mobile base and manipulator joints.
Protective fields and proximity-based safety functions are evaluated by injecting virtual people or obstacles into monitored regions around the robot. The simulator varies approach direction, distance, relative velocity, and robot motion while safety scanners or their interfaces generate corresponding zone states. Tests verify transitions between normal operation, reduced-speed operation, protective stop, and recovery, including difficult cases in which the manipulator extends beyond the footprint of the mobile base.
Combined-motion scenarios are particularly important because the arm can alter the effective hazard envelope while the platform is moving. HIL tests reproduce base translation while the arm extends, rotates, carries a payload, or approaches a workspace boundary. The safety system must correctly interpret the resulting robot state and enforce restrictions appropriate to the combined configuration. This exposes integration defects that isolated mobile-base or manipulator tests cannot detect.
Communication fault injection evaluates whether safety behavior remains valid when information exchange becomes unreliable. EtherCAT, CAN, Ethernet, ROS 2, or controller-level messages can be delayed, dropped, duplicated, frozen, or corrupted according to the interface under test. The campaign measures watchdog behavior, timeout detection, state transitions, command suppression, and recovery. Safety-relevant functions must not depend on continued availability of non-safety communication paths when those paths are lost.
Sensor and actuator abnormalities are introduced systematically through the HIL plant model. Encoder disagreement, implausible joint positions, stale velocity feedback, actuator saturation, brake-state inconsistencies, scanner faults, and localization abnormalities can be generated without damaging physical equipment. The expected response depends on the fault classification, allowing engineers to verify whether diagnostic logic distinguishes recoverable degradation from conditions requiring an immediate or controlled safety stop.
Timing measurement is a central advantage of HIL safety verification. Each event is timestamped from stimulus injection through controller recognition, safety decision, output transition, drive response, and simulated mechanical stopping. Engineers analyze nominal latency as well as repeated-run distributions and worst observed values. This separates software processing delay, network delay, drive reaction time, and mechanical stopping behavior, making failures easier to localize and safety margins easier to justify.
Reset and recovery testing ensures that reaching a safe state does not create a new hazard when operation resumes. The test suite checks whether faults must be cleared, safety inputs restored, reset conditions satisfied, and deliberate restart commands issued before motion is enabled. Scenarios also verify behavior when reset is attempted prematurely or repeatedly. Unexpected automatic restart is treated as a critical failure even when the original stopping function performed correctly.
Automation integrates these safety scenarios into a repeatable regression framework. Test definitions configure the initial robot state, simulated environment, motion command, fault or safety stimulus, expected transition sequence, timing limits, and final condition. The framework executes scenarios, synchronizes logs, calculates response metrics, and generates pass-or-fail evidence. This follows the broader HIL structure of safety-function verification, fault injection, automation, coverage measurement, and managed HIL infrastructure established for robot testing.
Regression testing becomes essential when controller software, safety parameters, drive firmware, robot geometry, payload limits, or communication configurations change. The CI system selects affected safety scenarios and executes them on available HIL stations before release approval. Critical functions receive mandatory regression coverage, while broader scenario combinations can run periodically. A previously validated safety function is therefore not assumed to remain valid merely because its source module was unchanged.
Traceability connects every automated HIL result to the corresponding safety requirement and configuration. Evidence includes software and firmware versions, hardware identifiers, parameter sets, scenario versions, stimulus timestamps, controller states, response measurements, logs, and final verdicts. Failed runs preserve sufficient information for reconstruction and root-cause analysis. This transforms HIL output from laboratory data into structured verification evidence that can support broader functional-safety assessment activities.
The completed campaign demonstrates that safety validation for a mobile manipulator must focus on interactions as strongly as individual functions. A base that stops correctly and an arm that stops correctly do not automatically guarantee that the combined machine behaves safely during simultaneous motion, faults, or recovery. HIL provides a controlled environment in which these interactions can be exercised repeatedly, including hazardous conditions that would be expensive, difficult, or unsafe to reproduce on the complete physical robot.
The resulting HIL system becomes a reusable safety regression asset throughout the robot lifecycle. As motion algorithms, hardware, payloads, safety zones, and operating modes evolve, the same framework can re-execute safety functions under controlled conditions and identify behavioral changes before deployment. By combining real controllers, simulated physical dynamics, precise fault injection, timing measurement, automated verdicts, and requirement traceability, the organization maintains continuous evidence that critical safety behavior remains effective as the mobile manipulator evolves.
12.04. Quadruped RL Policy SiL Sim2Real Validation Case
Quadruped locomotion based on reinforcement learning introduces a validation problem that differs fundamentally from conventional deterministic control. A policy may achieve high reward during training while remaining sensitive to terrain geometry, contact timing, actuator delay, observation noise, or model mismatch. The validation campaign therefore evaluates not only task success but also stability, robustness, repeatability, safety, and the transfer gap between simulated and physical robot behavior.
The process begins with Software-in-the-Loop, or SiL, validation using the trained policy inside a controlled simulation environment. The policy receives the same observation structure expected on the physical robot and produces equivalent joint-level or task-level commands. Robot dynamics, contact models, sensors, actuator characteristics, control frequency, command latency, and termination logic are configured explicitly so that evaluation is separated from the assumptions and randomization used during training.
A baseline test suite first measures performance under nominal conditions. The quadruped executes standing, forward and lateral locomotion, turning, velocity tracking, stopping, gait transitions, and representative terrain traversal. Each trial records command-tracking error, base velocity, orientation, body height, joint motion, torque, foot contact, slip, energy consumption, and episode termination. Repeated runs establish distributions rather than relying on a single successful demonstration.
Terrain coverage is expanded systematically because locomotion policies often fail outside the geometry emphasized during training. The simulation suite includes flat ground, slopes, stairs, discrete obstacles, stepping surfaces, uneven terrain, compliant regions, and low-friction patches where supported by the simulator. Terrain parameters are varied across increasing difficulty so that validation identifies performance boundaries rather than reporting only whether the robot can complete a predefined course.
Robustness testing perturbs observations and physical parameters independently before combining them. IMU noise, joint-position error, velocity estimation uncertainty, command delay, packet jitter, actuator strength variation, payload changes, center-of-mass shifts, friction changes, and external disturbances are injected within defined ranges. The resulting sensitivity profiles reveal which uncertainties produce gradual degradation and which can trigger abrupt instability or falls.
Monte Carlo evaluation provides statistical evidence across randomized initial states and environment parameters. Large batches of episodes vary terrain, friction, mass, latency, sensor noise, commanded velocity, and disturbance timing while preserving the policy version and evaluation protocol. Success rate, fall rate, tracking error, recovery rate, energy use, and constraint violations are aggregated with distributions and confidence measures, making comparisons between policy revisions more defensible.
Safety constraints are monitored independently from accumulated reinforcement-learning reward. A policy can obtain acceptable task reward while producing excessive joint velocity, torque peaks, foot impact, body inclination, self-collision risk, or unstable recovery behavior. Validation therefore introduces explicit limits for joint state, actuator load, orientation, contact behavior, and commanded motion. Constraint violations can fail a test even when the simulated robot reaches its destination.
Adversarial and edge-case scenarios concentrate evaluation around the boundaries discovered during broad simulation testing. Disturbances can be applied during single-support transitions, terrain height can change immediately before contact, friction can decrease asymmetrically, or observation quality can degrade during rapid turning. These tests are intended to expose fragile policy dependencies and identify combinations that ordinary randomized evaluation may encounter too rarely to characterize reliably.
Before physical deployment, the simulation model is reviewed for parameters that strongly influence locomotion transfer. Actuator dynamics, torque limits, joint friction, control delay, contact properties, sensor filtering, body mass distribution, and command execution frequency are compared with measurements or specifications from the real platform. Parameters with significant uncertainty are retained as ranges rather than tuned to one ideal value, preventing validation from becoming dependent on an unrealistically precise digital model.
The Sim-to-Real validation stage begins conservatively on the physical quadruped. Initial experiments use low velocities, simple terrain, restricted command ranges, controlled surroundings, and appropriate safety supervision. The same metrics used in simulation are collected where practical, including base motion, joint states, estimated contacts, actuator behavior, command tracking, and policy inference timing. This common measurement structure enables direct comparison between simulated and physical executions.
Transfer-gap analysis compares distributions rather than expecting identical trajectories. Differences in velocity tracking, body orientation, foot timing, slip, actuator effort, oscillation, energy consumption, and recovery behavior indicate where the simulator fails to reproduce relevant physical dynamics. A policy that succeeds in both domains may still require investigation when physical margins are substantially smaller than those predicted in simulation.
Real-world disturbances are then introduced progressively after nominal transfer has been demonstrated. Tests can include moderate pushes, payload variation, surface friction changes, slopes, uneven terrain, obstacles, and command transitions within the intended operating envelope. Difficulty increases through predefined gates so that a policy must demonstrate adequate stability and recovery at one level before being exposed to more demanding conditions that could increase hardware risk.
Failures discovered on the physical robot are converted into reproducible simulation cases whenever the underlying condition can be modeled. Logs identify robot state, command history, terrain characteristics, contact sequence, sensor behavior, and actuator response near the failure. Engineers reconstruct the event in simulation and determine whether model calibration, broader domain randomization, policy retraining, safety constraints, or controller integration changes are required before repeating the physical test.
Policy revisions are evaluated through regression testing rather than replacing previous evidence with new demonstrations. A candidate policy executes the established SiL scenario library, Monte Carlo campaign, robustness suite, safety checks, and selected physical benchmarks. Results are compared against the approved baseline to detect tradeoffs such as improved speed accompanied by increased energy use, reduced stability margin, greater torque demand, or weaker disturbance recovery.
Reproducibility requires strict control of evaluation artifacts. Policy weights, observation normalization, robot model, simulator version, terrain definitions, random seeds, control configuration, safety thresholds, and analysis scripts are versioned together. Physical tests additionally record robot hardware configuration, payload, calibration state, software revision, environment, and test conditions. This allows a reported improvement or regression to be traced to the exact configuration that produced it.
The validation result is therefore expressed as an operating envelope rather than a binary statement that the RL policy works. Evidence describes the terrain classes, velocity ranges, disturbance levels, payload conditions, uncertainty ranges, and safety margins under which acceptable performance has been demonstrated. Conditions outside this validated envelope remain unsupported until additional evidence is produced, preventing successful simulation behavior from being generalized beyond what testing actually establishes.
This quadruped case demonstrates that Sim-to-Real validation is a continuous evidence loop rather than a final transfer experiment. SiL provides scalable scenario exploration, randomized testing quantifies robustness, physical trials reveal modeling gaps, and real failures generate new regression scenarios. By repeatedly connecting simulation evidence with measured robot behavior, the organization can evolve RL locomotion policies while controlling transfer risk and preserving traceable evidence of stability, performance, and safety.
12.05. Humanoid VLA Policy Evaluation Framework Case
Humanoid Vision-Language-Action evaluation requires a framework that measures more than whether a robot eventually completes a commanded task. A VLA policy must interpret language, understand visual context, select appropriate actions, coordinate whole-body behavior, and maintain safety across long interaction sequences. The evaluation framework therefore separates task success from semantic understanding, action quality, generalization, robustness, latency, and physical execution reliability.
The case begins by defining a standardized task taxonomy for humanoid operation. Tasks are grouped by capabilities such as navigation, reaching, grasping, pick-and-place, object handover, articulated-object interaction, tool use, bimanual manipulation, and combined locomotion-manipulation. Each task includes an instruction, initial scene state, permitted interaction region, success condition, failure condition, timeout, and required safety constraints so that different policy versions can be evaluated consistently.
Task success rate provides the primary outcome metric but is decomposed into intermediate stages to expose where failures occur. A command such as retrieving an object may require instruction interpretation, target identification, navigation, approach, grasp preparation, grasp execution, transport, and placement. Recording completion at each stage distinguishes perception or language-grounding failures from planning, manipulation, locomotion, or low-level execution failures.
Instruction-following evaluation measures whether the policy correctly associates natural-language commands with the intended physical objective. Tests vary vocabulary, sentence structure, object references, spatial relations, ordering constraints, and paraphrased instructions while preserving task meaning. The framework records whether the correct object, location, action, and sequence are selected, preventing successful physical motion toward an incorrectly interpreted objective from being counted as task success.
Visual grounding is evaluated by changing object appearance, position, orientation, background, clutter, illumination, and partial occlusion. The humanoid must connect linguistic references to entities in the current scene rather than memorize fixed layouts or object positions. Evaluation distinguishes target recognition, spatial understanding, object-state interpretation, and action-relevant perception so that visual failures can be traced to the stage at which they influence physical behavior.
Generalization testing introduces novel combinations of known capabilities. Objects may appear in unseen positions, familiar tasks may be performed in new rooms, instructions may use previously unseen phrasing, or known actions may be required in different sequences. More difficult evaluations introduce novel object instances or scene configurations. Performance is reported separately for in-distribution and increasingly out-of-distribution conditions to avoid hiding generalization weakness inside an aggregate success score.
Long-horizon evaluation examines whether the VLA policy can maintain task coherence across extended action sequences. Multi-stage activities require the humanoid to preserve context, recognize completed subtasks, recover from minor execution errors, and continue toward the original objective. Metrics include full-task success, subtask completion, unnecessary action count, recovery frequency, repeated actions, time to completion, and the point within the sequence where irreversible failure occurs.
Physical execution quality is evaluated independently from semantic correctness. A policy may select the correct action but execute it with unstable posture, inefficient motion, excessive force, poor grasp quality, or unnecessary contact. Joint limits, balance indicators, end-effector trajectories, contact forces, object displacement, execution time, energy-related measures, and motion smoothness can therefore supplement task-level metrics when supported by the humanoid platform and instrumentation.
Safety evaluation operates as a parallel gate rather than a component of average task reward. Tests monitor unsafe human proximity, excessive contact force, unstable whole-body motion, prohibited workspace entry, object dropping, self-collision risk, loss of balance, and failure to respond to stop conditions. A task that reaches its semantic goal while violating a critical safety constraint is classified as unsuccessful regardless of the apparent quality of the final result.
Robustness testing introduces controlled disturbances into perception, language, and physical execution. Camera noise, temporary occlusion, ambiguous instructions, displaced objects, grasp perturbations, localization uncertainty, communication delay, and minor external disturbances are applied individually and in combinations. The evaluation measures whether the policy continues correctly, requests clarification where supported, replans, recovers, or transitions to a safe state when reliable execution is no longer possible.
Human interaction introduces additional evaluation dimensions when tasks involve shared spaces or handovers. The framework measures whether the humanoid detects people, respects interaction distance, coordinates timing, responds to changed human behavior, and avoids unexpected motion. For object handover, successful completion may require appropriate approach, stable presentation, release timing, and recognition that the receiving person has securely acquired the object before the robot withdraws.
Inference performance is measured because VLA intelligence is useful only when decisions arrive within the timing requirements of physical interaction. The framework records perception-to-action latency, policy inference latency, action generation rate, throughput, accelerator utilization, and latency variation under representative workloads. Tail latency is particularly important because occasional slow decisions can disrupt balance, manipulation timing, or interaction even when average inference performance appears acceptable.
Evaluation is executed across simulation, controlled laboratory environments, and physical humanoid trials. Simulation enables broad scenario variation and inexpensive repetition, while laboratory testing reveals sensor, actuator, contact, and timing effects that simulation cannot reproduce perfectly. A shared task definition and metric schema allows results from both environments to be compared, making simulation-to-real performance gaps visible instead of treating simulated success as equivalent to physical success.
Repeated trials are required because VLA policies and physical environments can produce stochastic outcomes. Each task is executed across multiple seeds, initial conditions, object configurations, and environmental variations where appropriate. Reports include success distributions, failure categories, latency statistics, safety violations, and confidence measures rather than presenting a single demonstration. This makes comparisons between candidate models more resistant to favorable scenario selection.
Failure analysis uses a structured taxonomy linking observed outcomes to perception, language grounding, reasoning, planning, action generation, locomotion, manipulation, control, safety, or system integration. Complex failures may receive multiple contributing labels rather than one simplified cause. Video, sensor logs, model outputs, robot states, actions, and timestamps are synchronized so that evaluators can reconstruct the decision sequence that produced the physical outcome.
Model comparison is performed using a multidimensional evaluation matrix rather than a single leaderboard score. Candidate VLA policies are compared across task success, generalization, instruction following, long-horizon completion, robustness, safety, latency, and physical execution quality. This reveals tradeoffs such as a larger model achieving stronger semantic reasoning while producing slower inference, or a faster model achieving higher throughput but weaker performance on novel tasks.
Reproducibility requires the complete evaluation context to be versioned. Model weights, prompts or instruction templates, robot software, perception configuration, action representation, task definitions, scene layouts, object sets, random seeds, safety thresholds, and metric implementations are associated with every evaluation campaign. Physical trials additionally record hardware configuration and calibration so that improvements can be distinguished from changes in the experimental environment.
The resulting framework treats humanoid VLA evaluation as continuous system validation rather than a one-time model benchmark. New field failures, novel tasks, difficult instructions, and previously unseen scene conditions are converted into permanent regression cases. By connecting semantic evaluation, physical execution, safety, generalization, latency, and reproducible real-world testing, the organization can determine not merely whether a VLA model appears capable, but where its demonstrated capability is sufficiently reliable for humanoid deployment.
12.06. Cargo UAV DO 178C SW Verification Campaign Case
A cargo unmanned aerial vehicle requires software verification discipline beyond ordinary robotic testing because software behavior can directly affect flight safety, vehicle control, navigation, and mission execution. In this case, the verification campaign is organized around DO-178C principles while remaining integrated with the broader UAV testing architecture. The objective is to establish traceable evidence that airborne software requirements are correctly implemented and verified under representative and abnormal operating conditions.
The campaign begins by defining the software verification scope and its relationship to system and safety requirements. Flight-control functions, navigation logic, mission management, actuator interfaces, sensor processing, communication services, health monitoring, and safety-related mode transitions are identified according to their contribution to aircraft behavior. Verification activities are then planned so that requirements, implementation, test procedures, results, anomalies, and configuration records remain connected throughout the development lifecycle.
Requirements-based testing forms the foundation of the verification strategy. Each software requirement is associated with one or more verification methods and explicit acceptance criteria. Normal-range behavior, boundary conditions, invalid inputs, initialization, mode transitions, timing behavior, and failure responses are considered when deriving test cases. This prevents verification from becoming a collection of demonstrations and instead establishes objective evidence that specified software behavior has been exercised.
High-level requirements are progressively traced to low-level requirements and source-code implementation where applicable. Bidirectional traceability allows engineers to determine which software elements implement each requirement and which verification cases demonstrate compliance. It also exposes unintended functionality because executable code without a justified requirement requires investigation. Traceability therefore becomes both a coverage mechanism and a control against software behavior that has not been formally specified.
Unit verification addresses software components whose logic can be tested independently from the complete flight system. Guidance calculations, state-machine transitions, sensor validation, limit checking, coordinate conversion, command arbitration, fault detection, and utility functions can be exercised with controlled inputs. Boundary values and abnormal conditions receive particular attention because numerical saturation, invalid state transitions, or unhandled data can propagate rapidly into flight-control behavior.
Software integration testing verifies interfaces and data flows between airborne software components. Tests examine message validity, execution ordering, shared data, communication timing, initialization dependencies, mode coordination, and fault propagation. Flight-control software must continue to receive valid and timely information as navigation, mission, health-monitoring, and communication functions interact. Interface failures are therefore evaluated as system behaviors rather than merely as defects inside individual modules.
Software-in-the-Loop testing extends verification into executable flight scenarios without requiring physical avionics. The integrated software is exercised against simulated aircraft dynamics, sensors, navigation states, environmental conditions, and mission commands. Nominal takeoff, climb, cruise, route following, descent, landing, cargo mission transitions, and contingency behaviors can be repeated deterministically while collecting internal software states and outputs that may be difficult to observe during actual flight.
Hardware-in-the-Loop verification introduces representative flight computers, communication interfaces, I/O devices, and control hardware into the campaign. A real-time simulator supplies sensor and aircraft-state signals while receiving commands from the actual avionics configuration. Timing, scheduling, driver behavior, hardware interfaces, actuator commands, and fault responses can therefore be evaluated under conditions closer to flight while retaining the repeatability and controlled fault injection required for rigorous verification.
Structural coverage analysis determines whether requirements-based testing has exercised the implemented software structure to the extent required by the applicable assurance objectives. Statement, decision, and where applicable modified condition/decision coverage are examined according to the selected software assurance level. Coverage gaps trigger analysis to determine whether additional requirements-based tests are necessary, whether code is unreachable, or whether unintended functionality exists.
Coverage closure is not achieved simply by creating tests that execute previously uncovered code. Any additional test must remain justified by software requirements or by analysis consistent with the verification process. If coverage analysis exposes executable behavior for which no requirement exists, the appropriate response may be to correct the requirements or implementation rather than manufacture a test solely to increase the numerical coverage percentage. This preserves the requirements-based nature of the campaign.
Robustness verification evaluates software responses to abnormal and unexpected inputs. Sensor values can exceed valid ranges, messages can arrive late or become unavailable, navigation information can become inconsistent, communication can be interrupted, and internal states can violate expected assumptions. Tests verify that such conditions are detected and handled without uncontrolled software behavior and that appropriate degraded, contingency, or safe modes are entered when required.
Timing and resource behavior are also verified because correct logical output is insufficient if computation is delivered too late. Execution periods, task deadlines, worst-observed processing times, CPU loading, memory consumption, communication latency, and scheduling interactions are measured under representative high-load conditions. Particular attention is given to flight-critical control loops and mode transitions where excessive delay could reduce stability or prevent timely safety responses.
Fault-injection testing complements normal requirements-based verification by exercising defined failure-handling paths. Simulated sensor loss, invalid navigation data, actuator-interface abnormalities, communication timeouts, processor-related events, or subsystem unavailability can be introduced under controlled conditions. The purpose is not arbitrary destructive testing but verification that specified monitoring, isolation, degradation, and recovery logic behaves according to the approved software requirements.
Configuration management ensures that every verification result corresponds to a known software and test baseline. Source code, executable object code, requirements, test procedures, test data, simulation models, scripts, tool versions, configuration parameters, and generated evidence are identified and controlled. When a software change occurs, impact analysis determines which requirements, tests, coverage results, and verification artifacts must be reviewed or repeated before the modified baseline can be accepted.
Problem reporting and corrective action are integrated into the campaign rather than handled outside the verification process. Failed tests, unexpected outputs, coverage anomalies, traceability gaps, tool problems, and configuration inconsistencies are recorded with sufficient evidence for investigation. Corrective changes are subjected to regression analysis so that fixing one defect does not invalidate previously verified behavior elsewhere in the airborne software.
Independence is applied where required by the applicable assurance objectives so that verification is not reduced to developer self-confirmation. Reviews, test execution, result assessment, coverage analysis, and artifact approval can involve personnel or organizational separation appropriate to the software level and verification objective. The degree of independence is planned explicitly and preserved in records so that the resulting evidence demonstrates both technical completion and process integrity.
Flight testing is used to complement, rather than replace, software verification performed at lower levels. Selected scenarios confirm integrated behavior in the actual aircraft environment, particularly where aerodynamic effects, navigation performance, communication conditions, propulsion behavior, or real operational interactions cannot be represented completely in laboratory testing. Flight results are connected to requirements and prior SiL or HIL evidence, preserving continuity across the verification chain.
The completed campaign produces an evidence package rather than a single pass-or-fail result. Requirements traceability, review records, test procedures, test results, structural coverage analysis, problem reports, configuration records, regression evidence, and verification summaries collectively demonstrate the state of software assurance. This case shows how a cargo UAV can combine disciplined DO-178C-oriented software verification with SiL, HIL, fault injection, and flight validation to maintain traceable confidence as airborne software evolves.
12.07. AI Model OOD Detection Evaluation Case
AI models deployed on physical robots encounter operating conditions that cannot be completely represented by training and validation datasets. New objects, unusual environments, sensor degradation, weather changes, lighting variations, unexpected human behavior, or combinations of familiar factors can move observations outside the training distribution. OOD detection evaluation therefore determines whether a model can recognize uncertainty before unreliable predictions propagate into navigation, manipulation, or safety-critical decisions.
The evaluation begins by defining what constitutes in-distribution, near-OOD, and far-OOD data for the target robotic application. In-distribution samples reflect conditions adequately represented during development, while near-OOD cases contain realistic variations close to the operational boundary. Far-OOD samples represent substantially different objects, scenes, sensor characteristics, or environmental conditions. This separation prevents fundamentally different forms of distribution shift from being hidden inside one aggregate metric.
A representative reference dataset is constructed from the approved operational domain and divided carefully to avoid leakage between training and evaluation. Object categories, environments, robot platforms, sensor configurations, locations, acquisition periods, and environmental conditions are considered when creating splits. OOD evaluation data must remain genuinely unseen by the evaluated model and by any calibration procedure used to determine confidence thresholds or uncertainty parameters.
OOD scenarios are generated along several dimensions rather than through a single unknown-object dataset. Semantic shifts introduce unseen classes or object configurations, environmental shifts modify lighting, weather, background, terrain, or clutter, and sensor shifts introduce noise, blur, missing regions, calibration error, or changed sensing characteristics. Compound shifts combine multiple factors to evaluate whether detection performance deteriorates when realistic disturbances occur simultaneously.
The framework evaluates model confidence because incorrect predictions with high confidence represent a particularly important risk for physical AI. Confidence distributions are compared between correct in-distribution predictions, incorrect in-distribution predictions, and OOD samples. A useful detector should preserve confidence for familiar valid inputs while assigning sufficiently different uncertainty or OOD scores to observations for which the model lacks reliable evidence.
Threshold selection is treated as an operational design decision rather than a purely statistical optimization. A low threshold may detect more unfamiliar inputs but produce excessive false alarms, while a high threshold may allow dangerous unknown cases to pass undetected. Thresholds are therefore selected using validation data and application-specific risk requirements, with particular attention to the consequences of false acceptance and false rejection for the downstream robot behavior.
Evaluation metrics characterize both discrimination quality and operational error. Area under the receiver operating characteristic curve, area under the precision-recall curve, false-positive rate at a selected true-positive rate, OOD detection rate, and calibration-related measures can be used depending on the detector. Metrics are reported separately for OOD categories and severity levels because a single average value can conceal serious weaknesses under particular environmental or semantic shifts.
Confidence calibration is evaluated independently from OOD discrimination. Expected calibration error, reliability relationships, confidence histograms, or related measures determine whether predicted confidence corresponds meaningfully to observed correctness. A detector may rank OOD samples effectively while still producing poorly calibrated confidence values. For robotic deployment, both properties matter because downstream decision logic often uses thresholds to trigger reduced-speed, fallback, or human-supervision modes.
The evaluation also measures task-level impact rather than stopping at model-level statistics. For perception systems, an undetected OOD condition may produce incorrect obstacle classifications, missed objects, or unstable scene interpretation. For navigation or manipulation policies, distribution shift may alter action selection directly. Tests therefore trace OOD events through the downstream pipeline and determine whether detection occurs early enough to prevent unsafe or severely degraded physical behavior.
Temporal evaluation is particularly important for robotics because observations arrive as sequences rather than isolated samples. A single uncertain frame may not justify intervention, while sustained uncertainty can indicate genuine distribution shift. The framework measures detection latency, score persistence, temporal stability, and recovery after the robot returns to familiar conditions. This reveals detectors that appear accurate on independent frames but oscillate excessively when deployed in continuous operation.
Robustness testing varies the severity of each distribution shift progressively. Image brightness, sensor noise, occlusion, blur, weather intensity, localization uncertainty, or environmental novelty can be increased from nominal conditions toward clearly unfamiliar states. Performance curves show where confidence separation begins to collapse and where OOD detection becomes reliable. These transition regions help define practical operating boundaries rather than only evaluating extreme unknown cases.
False-positive analysis receives special attention because excessive OOD alarms can make a robot operationally unusable. Frequent unnecessary stops, fallback transitions, or requests for human intervention may reduce mission availability even when safety is preserved. Evaluation therefore records which valid conditions are incorrectly classified as OOD and groups them by scene, object, sensor, and environmental characteristics so that detector thresholds or training data can be improved systematically.
False-negative analysis focuses on unfamiliar conditions that the detector accepts as normal. These cases are prioritized according to the severity of downstream consequences. An unfamiliar visual texture with no behavioral effect may carry limited risk, while an unknown obstacle accepted with high confidence can directly affect collision avoidance. Risk-weighted analysis therefore complements aggregate statistical metrics by connecting detection failure to potential physical outcomes.
Comparative evaluation applies the same dataset splits, perturbations, thresholds, and reporting protocol to candidate OOD methods. Confidence-based baselines, uncertainty estimators, feature-space approaches, or dedicated detection models can then be compared under identical conditions. Model selection considers detection quality together with computational cost, memory consumption, inference latency, accelerator utilization, and integration complexity because an accurate detector must still operate within the robot\'s edge-computing budget.
Real-world testing follows offline evaluation to expose shifts that synthetic corruption or curated datasets may not reproduce. Robots collect data across different locations, times, lighting conditions, surfaces, object populations, and operational states while OOD scores and downstream decisions are logged. Human review or additional ground truth is used for significant events, allowing engineers to determine whether the detector identifies genuinely unfamiliar conditions rather than merely reacting to superficial visual changes.
Production monitoring extends OOD evaluation beyond the initial release. Score distributions, alarm frequency, false alarms, intervention events, and representative high-uncertainty samples are tracked over time. Newly discovered OOD cases are reviewed and, where appropriate, added to permanent regression datasets. This creates a continuous evaluation loop in which field operation expands the evidence base and prevents the detector from being validated only against a static benchmark.
Reproducibility requires strict versioning of the AI model, OOD detector, datasets, preprocessing, calibration procedure, threshold configuration, perturbation definitions, metric implementation, and hardware environment. Every reported result must identify the exact configuration that produced it. When the underlying AI model or training dataset changes, previously established OOD performance is reassessed because confidence behavior and feature distributions can change even if the detector implementation itself remains unchanged.
The final evaluation does not claim that an OOD detector can identify every possible unknown condition. Instead, it documents the distribution shifts tested, detection performance achieved, threshold tradeoffs, known failure regions, computational cost, and downstream safety behavior. By combining statistical evaluation, calibration analysis, temporal testing, risk-weighted failure analysis, real-world trials, and continuous regression, the organization can use OOD detection as a measurable safeguard against AI uncertainty in physical robot deployment.
12.08. Fleet Wide Continuous Testing at Scale Case
Fleet-wide continuous testing extends robot verification from controlled development environments into the operation of hundreds or thousands of deployed systems. A software or AI update that performs correctly on one reference robot may behave differently across hardware revisions, sites, workloads, networks, maps, and environmental conditions. The testing architecture therefore treats the deployed fleet as a distributed source of validation evidence while maintaining strict controls over safety, release progression, and regression detection.
The case begins with a common fleet test model that connects software versions, robot configurations, operational sites, mission types, and measurable quality indicators. Every deployed robot reports sufficient configuration metadata to identify its software build, hardware revision, sensor set, calibration state, map version, AI model, and relevant runtime parameters. Without this context, field observations cannot reliably distinguish a software regression from differences in hardware, environment, or deployment configuration.
Pre-deployment testing remains the first gate before any candidate release reaches operational robots. Unit, integration, simulation, HIL, safety, and system regression suites provide evidence that known requirements and previously discovered failures remain satisfied. Fleet testing does not replace these stages. Instead, it addresses variability that laboratory environments cannot reproduce economically and provides a shift-right extension of the existing continuous testing pipeline.
Release deployment is performed progressively rather than updating the complete fleet simultaneously. A candidate build first reaches internal or designated test robots, followed by a small canary population operating under controlled risk. If health indicators remain within defined limits, deployment expands through progressively larger cohorts. Each stage has explicit promotion, hold, and rollback criteria so that a regression can be contained before it affects a significant portion of the operational fleet.
Cohort design is critical because fleet averages can conceal configuration-specific failures. Robots are grouped according to hardware generation, sensor configuration, site, environmental class, mission profile, software dependency, or other relevant dimensions. Performance is compared both within each cohort and against an approved baseline. A release can therefore be stopped even when fleet-wide averages appear healthy if one hardware revision or operational environment shows significant degradation.
Production monitoring converts normal robot operation into structured test evidence. Mission completion, navigation success, localization quality, recovery frequency, safety events, intervention rate, planning failures, inference latency, CPU and memory usage, network behavior, energy consumption, and subsystem health can be collected continuously. These metrics provide a behavioral signature for each release and allow the organization to detect changes that conventional pass-or-fail tests may not expose.
Regression detection relies on comparison with historical and control distributions rather than fixed thresholds alone. Statistical baselines capture normal variation by robot type, site, mission, and operating condition. A candidate release is evaluated for changes in failure rate, latency, recovery behavior, resource consumption, or safety-related events. Persistent deviations trigger investigation even when individual observations remain inside absolute operating limits, enabling earlier detection of gradual performance degradation.
A/B testing can be used for non-safety-critical behavior when controlled comparison provides useful evidence. Different robot cohorts may operate approved software variants while mission outcomes and performance indicators are measured under comparable conditions. Experimental assignment must respect operational constraints, and safety-critical functions remain protected by independent validation gates. The purpose is to measure real operational effects without treating uncontrolled fleet deployment as an experiment.
Shadow-mode testing provides a lower-risk method for evaluating new planning or AI policies. The candidate component receives production sensor data and system state but does not control the robot. Its proposed outputs are logged and compared with those of the active approved system. Disagreement rates, predicted trajectories, confidence, latency, and hypothetical constraint violations can be analyzed before authority is granted to the candidate policy, providing field-scale evidence without directly changing physical behavior.
Field events automatically generate regression candidates when predefined triggers are detected. Unexpected emergency stops, repeated recovery behavior, localization loss, near-collision indicators, abnormal planner oscillation, perception uncertainty, excessive latency, or operator intervention can preserve relevant telemetry and context. The event is reconstructed in simulation, software integration testing, HIL, or another appropriate environment whenever possible and then added to the permanent regression suite.
Fleet-scale testing requires careful management of telemetry volume because continuous raw-data upload from every robot is rarely practical. Robots perform local filtering, aggregation, event detection, or selective recording so that high-value evidence is retained while routine data is summarized. Triggered uploads preserve detailed sensor and system context around important events, while fleet dashboards use aggregated statistics to track trends without overwhelming network and storage infrastructure.
Safety validation operates independently from general performance optimization. Safety events, protective stops, emergency-stop activations, speed-limit violations, collision indicators, and safety-system faults are monitored with stricter thresholds and escalation rules. A release that improves throughput or mission completion cannot be promoted if safety-related evidence deteriorates beyond accepted limits. Safety regression therefore acts as a release-blocking condition rather than one metric within an averaged quality score.
Rollback capability is designed as part of the test architecture rather than as an emergency afterthought. The deployment system maintains known-good software and configuration baselines and can stop rollout or return affected cohorts to an approved version when defined conditions are met. Rollback decisions use versioned evidence so engineers can determine whether the problem originated from application software, an AI model, configuration, firmware, map data, or interactions among multiple changed components.
Continuous coverage tracking shows which portions of the fleet have actually exercised important operating conditions after a release. Coverage can be classified by robot configuration, mission type, site, environmental condition, runtime duration, scenario category, and safety-relevant state. This prevents a release from being considered broadly validated merely because many operating hours have accumulated on a homogeneous subset of robots that represents only a small part of the intended deployment space.
Scaling the test system also requires control of false alarms and flaky signals. Temporary network loss, maintenance activity, operator behavior, or site-specific disturbances can resemble software regressions. Detection logic therefore correlates multiple metrics, compares affected cohorts, and preserves event context before escalation. Repeatedly noisy indicators are investigated and refined so that engineering teams do not become desensitized to alerts that lack actionable diagnostic value.
Traceability connects every production observation to the deployed configuration and corresponding release decision. Software builds, AI models, configuration packages, deployment cohorts, test results, telemetry summaries, incidents, rollback events, and approval records are retained as linked evidence. When a defect is discovered months later, engineers can determine which fleet populations were exposed, which operational conditions were encountered, and which earlier verification evidence must be reconsidered.
The resulting architecture creates a closed continuous-testing loop between development and production. Laboratory regression suites prevent known defects from reaching the field, progressive deployment limits exposure to unknown defects, fleet monitoring discovers real-world weaknesses, and captured failures become new automated tests. Each deployment therefore expands the organization\'s regression knowledge rather than producing operational data that remains disconnected from engineering verification.
This fleet-wide case demonstrates that testing at scale is not simply running the same test on more robots. It is a coordinated evidence system combining pre-release verification, staged rollout, cohort analysis, shadow testing, telemetry-driven regression, continuous safety validation, coverage tracking, and controlled rollback. By treating fleet operation as a governed extension of the verification process, an organization can evolve robot software and AI rapidly while containing deployment risk and continuously strengthening validation evidence.
12.09. ISO 3691 4 Certification Test Campaign Case
An ISO 3691-4 certification test campaign for an autonomous mobile robot requires more than demonstrating that the vehicle can navigate successfully. The campaign must establish objective evidence that the driverless industrial truck and its associated control functions manage foreseeable hazards throughout intended operation. Testing therefore combines requirements traceability, safety-function verification, stopping performance, personnel detection, operating-mode behavior, environmental conditions, and documented evidence suitable for conformity assessment.
The campaign begins by defining the machine configuration and intended operating conditions to which the certification evidence applies. Vehicle dimensions, mass, payload range, maximum speed, steering configuration, braking system, sensing equipment, safety devices, control architecture, operating modes, and permitted environments are placed under configuration control. The intended operating zone and foreseeable interactions with people, obstacles, infrastructure, and other vehicles provide the context for subsequent safety tests.
A certification requirements matrix translates applicable safety requirements into verifiable test objectives. Each requirement is connected to the relevant robot function, hazard, test method, acceptance criterion, required instrumentation, and resulting evidence. This structure prevents important requirements from being evaluated only indirectly and allows assessors to trace each compliance claim to a specific analysis, inspection, measurement, or physical test result.
Protective stopping behavior is evaluated under representative combinations of speed, payload, direction, and surface condition. Tests measure the response from detection of a hazardous condition through command processing, brake or drive reaction, deceleration, and complete stopping. Repeated trials establish whether stopping behavior remains within the defined protective limits and whether variation caused by payload, tire condition, floor characteristics, or control timing has been adequately considered.
Personnel detection testing verifies the protective sensing system across the regions in which people may be exposed to vehicle motion. Test objects or representative targets are introduced at defined positions and approach directions while the vehicle operates in applicable modes. The campaign examines whether detection causes the required speed reduction or protective stop and whether the resulting response remains effective near straight paths, curves, intersections, and other relevant operating geometries.
Speed control is tested because protective measures frequently depend on the relationship between vehicle velocity, sensing range, and stopping capability. The robot is commanded through normal and restricted operating conditions while actual speed and safety-system state are recorded. Tests confirm that configured speed limits are respected, that reduced-speed modes are entered when required, and that a violation does not persist in a manner that invalidates the protective separation assumed by the safety concept.
Directional and turning behavior receive separate attention because the hazard envelope can change substantially during steering. A scanner field that is sufficient during straight travel may not provide equivalent protection when the vehicle turns, rotates, or changes direction. Test scenarios therefore examine forward motion, reverse motion, curves, tight turns, and transitions between motion states while verifying that protective fields, speed limits, and stopping responses remain appropriate to the actual vehicle trajectory.
Emergency-stop functions are tested from accessible activation devices through the complete control chain to the resulting machine state. Activation is performed during representative motion and operating conditions to verify that hazardous movement is brought to the required condition and cannot restart unexpectedly. Reset behavior is also evaluated to ensure that releasing or resetting an emergency-stop device does not itself initiate motion without the additional deliberate actions required by the safety architecture.
Automatic and manual operating modes are evaluated independently and during transitions between modes. The campaign verifies that mode selection produces the intended control authority, speed limitations, protective functions, and operator interaction behavior. Transitions are tested while the vehicle is stationary and, where relevant and permitted, during controlled operating states. Ambiguous or conflicting commands must not result in uncontrolled movement or unintended bypass of safety functions.
Obstacle interaction scenarios examine objects that differ in geometry, position, and detectability. The test set includes conditions relevant to the intended environment, such as obstacles located within the travel path, near path boundaries, or in regions affected by turning motion. The purpose is not merely to demonstrate obstacle avoidance performance but to verify that the safety-related response prevents unacceptable contact when normal navigation or avoidance behavior cannot resolve the situation safely.
Braking and stopping tests account for foreseeable variations that influence physical stopping distance. Vehicle mass, payload distribution, travel speed, direction, surface friction, slope where applicable, and braking condition are controlled or recorded. Instrumentation measures velocity and stopping distance with sufficient accuracy to compare observed behavior against the assumptions used to configure protective fields and other risk-reduction measures.
Fault-response testing examines whether safety-related functions transition appropriately when required information or components become unavailable. Representative cases can include protective sensor faults, communication loss, invalid control information, drive-related abnormalities, or monitoring failures according to the system design. The expected response is defined before testing so that the campaign verifies specified fault behavior rather than accepting any stop or shutdown as automatically compliant.
Environmental and operational conditions are incorporated where they can influence safety performance. Lighting, reflective surroundings, contamination, floor characteristics, temperature, vibration, electromagnetic conditions, or other relevant factors may affect sensing, braking, control, or communication. The certification campaign identifies which conditions require physical testing, supporting analysis, component evidence, or operational restrictions so that the claimed operating envelope remains consistent with demonstrated safety performance.
Human interaction tests address foreseeable encounters between the mobile robot and workers within shared operating areas. Scenarios include approach from different directions, crossing movement, entry into protected regions, and situations in which a person remains near the vehicle. The robot\'s sensing and control responses are observed together because personnel safety depends on the complete chain from detection through decision and motion reduction rather than on sensor performance alone.
Test instrumentation and data acquisition are controlled as part of the certification process. Speed, position, safety inputs, controller states, braking commands, detection events, and stopping behavior are synchronized where required. Measurement equipment is selected with appropriate accuracy and calibration status, and the test procedure defines how results are calculated. This ensures that acceptance decisions are based on reproducible measurements rather than visual judgment of robot behavior.
Any failed or ambiguous result enters a documented resolution process. Engineers determine whether the cause originates from design, software, configuration, test setup, environmental conditions, or measurement uncertainty. Corrective changes are subjected to impact analysis and appropriate regression testing. Previously passed tests are repeated when the modification can influence their assumptions, preventing local corrections from silently invalidating other parts of the certification evidence.
Configuration management ensures that the robot presented for final assessment corresponds to the configuration represented by the accumulated evidence. Software versions, safety parameters, sensor models, firmware, braking configuration, mechanical characteristics, payload assumptions, maps where relevant, and test procedures are identified. Changes after formal testing are evaluated to determine whether existing evidence remains applicable or whether partial or complete retesting is required.
The completed campaign produces a structured certification evidence package linking requirements, hazards, safety functions, test configurations, procedures, measurements, deviations, corrective actions, and final results. ISO 3691-4-oriented certification testing therefore becomes an evidence-driven validation process rather than a final demonstration event. By maintaining traceability from safety requirements to repeatable physical results, the organization can support conformity assessment while preserving a reusable regression baseline for future AMR revisions.
12.10. Future Testing Automation and AI Driven Test
Future robot testing will evolve from manually authored test collections toward continuously adaptive verification systems that can generate, prioritize, execute, and analyze tests automatically. As Physical AI systems combine perception, foundation models, planning, control, distributed computing, and physical interaction, fixed regression suites alone cannot explore the growing state space. Automation and AI-driven testing therefore become mechanisms for expanding coverage while preserving repeatability, traceability, and human oversight.
The future architecture begins with machine-readable requirements connected directly to test specifications and evidence. Safety, functional, performance, timing, AI, and operational requirements can be represented with structured identifiers, parameters, acceptance criteria, and dependencies. Automated systems use these relationships to determine which tests apply to a software or hardware change, reducing unnecessary execution while ensuring that affected requirements and previously discovered failure modes remain covered.
AI-assisted test generation expands conventional parameterized testing by proposing scenarios from requirements, system models, field failures, and historical test results. A generation engine can vary environment geometry, robot state, actor behavior, sensor conditions, communication quality, payload, weather, and timing while respecting defined operational boundaries. Engineers retain control of scenario constraints and acceptance logic so that generated cases remain meaningful rather than becoming arbitrary combinations with little verification value.
Coverage-guided generation directs computational resources toward poorly explored regions of the operating space. Instead of repeatedly executing scenarios that provide similar evidence, the test system tracks combinations of robot configuration, environment, behavior, uncertainty, and failure conditions. Generation policies then search for coverage gaps, boundary regions, rare interactions, and transitions where existing evidence is weak, allowing simulation capacity to contribute directly to measurable validation completeness.
Risk-guided testing further prioritizes scenarios according to potential physical consequences. Hazard severity, exposure, uncertainty, historical failure frequency, safety margin, and downstream system impact can influence scheduling. High-risk scenarios receive denser sampling and stronger regression requirements, while lower-risk combinations may receive reduced execution frequency. This approach prevents large-scale automation from treating every possible test combination as equally important.
Search-based and adversarial testing can actively discover conditions that push the robot toward failure. Optimization algorithms vary terrain, obstacle motion, sensor degradation, latency, friction, disturbances, or human behavior to minimize safety margins or maximize selected failure indicators. The objective is not to create unrealistic attacks on the system but to identify plausible combinations near operational boundaries that conventional test design or random sampling may overlook.
Digital twins and scalable simulation provide the execution substrate for this expanded scenario space. Thousands of virtual robots can evaluate software, AI policies, safety logic, and system configurations before physical deployment. Simulation fidelity is managed according to the verification objective rather than maximized uniformly. Fast abstract models support broad exploration, while higher-fidelity dynamics, sensor simulation, SiL, and HIL are applied to scenarios where physical timing and interaction significantly affect the result.
AI can also support automatic test-oracle generation, although this capability requires strong governance. Deterministic requirements can produce explicit expected outputs, while complex AI behaviors may require invariant checks, safety envelopes, reference policies, statistical expectations, or human-reviewed evaluation criteria. An AI-generated verdict should not become authoritative merely because it is automated. Critical acceptance decisions require transparent criteria and evidence that can be independently reviewed.
Failure triage is another area where AI-driven automation can substantially reduce engineering effort. Large campaigns may generate thousands of failed or anomalous runs whose symptoms originate from a smaller number of root causes. Automated analysis can cluster similar traces, compare logs, identify correlated configuration changes, summarize failure signatures, and rank likely causes. Engineers can then focus on high-value investigation rather than manually reviewing every individual execution.
Field data creates a continuous source of new test scenarios. Unexpected stops, operator interventions, near-collision indicators, localization degradation, unusual perception uncertainty, manipulation failures, and abnormal timing behavior can trigger selective data capture. Automated pipelines reconstruct suitable events in simulation or HIL and convert them into regression cases. The test library therefore evolves from actual operational experience instead of depending exclusively on scenarios imagined before deployment.
Generative models can assist with synthetic environment and sensor-data creation when real examples are scarce, but generated data must be validated before it contributes to safety claims. Synthetic objects, scenes, weather conditions, visual variations, or language instructions can increase diversity and expose model weaknesses. Provenance, generation parameters, realism checks, and separation from evaluation ground truth are maintained so that synthetic scale does not create misleading confidence.
Predictive test selection reduces the cost of continuous integration by estimating which tests are most relevant to a proposed change. Source-code modifications, model revisions, configuration changes, hardware dependencies, historical failures, and requirement links can inform selection. Critical safety tests remain mandatory, while broader regression suites are scheduled according to impact and risk. Periodic full campaigns verify that optimization has not gradually removed important coverage.
Testing infrastructure itself becomes increasingly autonomous. Simulation farms, HIL benches, physical robot laboratories, and fleet shadow-testing resources can be scheduled according to test priority, equipment availability, runtime, and risk. Failed infrastructure checks can quarantine unreliable stations, while reproducibility controls preserve software, models, configuration, random seeds, calibration state, and tool versions. Automation therefore manages not only test execution but also the trustworthiness of the environment producing evidence.
AI-model evaluation will increasingly require continuous monitoring for distribution shift, uncertainty, generalization loss, and behavioral drift. New model versions can be compared against approved baselines across standardized tasks and automatically generated edge cases. Performance improvements are accepted only when safety, robustness, latency, resource consumption, and previously validated capabilities remain within required limits. A higher average task score alone is insufficient evidence for release.
Human oversight remains essential even as automation expands. Engineers define safety objectives, operational boundaries, acceptance rules, escalation policies, and the authority granted to automated systems. AI can recommend scenarios, classify failures, and prioritize evidence, but significant changes to safety assumptions or certification claims require accountable review. The future testing organization therefore shifts human effort from repetitive execution toward supervision, interpretation, risk judgment, and assurance.
Traceability becomes the backbone connecting autonomous testing decisions to engineering evidence. Every generated scenario records why it was created, which requirement, risk, coverage gap, field event, or software change motivated it, and which configuration executed it. Results, failures, model versions, test-oracle logic, and approval decisions remain linked. This allows automated testing to scale without creating an opaque collection of results whose relevance cannot later be explained.
The long-term objective is a closed-loop verification system in which development changes trigger targeted tests, coverage gaps generate new scenarios, failures create regression cases, field operation supplies new evidence, and release decisions update future test priorities. Such a system continuously learns where verification effort provides the greatest value while maintaining controlled baselines. Testing evolves from a downstream project phase into an always-active engineering function spanning development, deployment, and operation.
AI-driven testing does not eliminate the need for deterministic verification, physical experiments, safety engineering, or certification discipline. Its value lies in extending those practices across a state space too large for manual methods alone. By combining requirements-based testing, risk and coverage guidance, scalable simulation, SiL, HIL, field feedback, automated analysis, and governed AI assistance, future Physical AI programs can increase validation depth while preserving reproducible and defensible evidence.
Author: Richard Minsoo Go
LinkedIn: Richard Minsoo Go










Top comments (0)