DEV Community

Nashtarin Nur
Nashtarin Nur

Posted on

Shadow Validation and Drift Mitigation: Architecting a 4-Engine MLOps Pipeline for Physical AI

Physical AI models in production must be monitored carefully in order to maintain deterministic behaviour. In contrast to software AI models in the cloud with standardised inputs, Physical AI models will be run in harsh operating environments on edge devices. This means the models may be subject to wear and tear, thermal variation, and drift in sensor sensitivity.

If models are not monitored, model performance degradation in production can result in incorrect alerts to maintenance staff and overlooked faults within equipment. However, constant retraining in the cloud is difficult to perform given unpredictable connectivity within manufacturing environments and tight bandwidth limits. To ensure robust AI models within industry, an edge native MLOp design must be implemented. This allows for detection of drift, verification, and deterministic control at the edge.

Four Engine Framework

In order to maintain model performance across hundreds of physical assets, data ingestion, processing, inference, and actuation should be separated. This avoids drift affecting system performance. Within well developed Physical AI applications, four engines are used:

  1. Identification Engine

Drift within data can be erroneously detected due to different physical assets being present within the environment. If tools are changed, lines are re-configured, or mobile carriers changed, the Identification Engine mitigates this issue by combining information from UWB sensors, passive RFID gates, BLE tags, and visual tracking sensors. It provides a persistent location map with precision less than 1m, ensuring that any data is compared only to the particular physical asset in place in the relevant area.

  1. Sensing Engine

Degradation in sensors (including wobbly accelerometers, or dirty thermal sensors) can cause model error by mimicking a fault in equipment. The Sensing Engine collects multi-modal data from edge computing within the physical assets, including tri-axial vibration, sound, thermal and current data. Before passing data to models below, noise from the environment is filtered out, and sensor data integrity is confirmed. In addition, data features are normalised.

  1. AI Decision Engine

It is highly risky to update models used in live industrial processes if new weights produce unpredictable results in edge case scenarios. The AI Decision Engine reduces this risk by providing shadow validation. Normalised data is compared to known physical baselines within the domain to provide deterministic risk scores. In addition, feature drift is also tracked so that if model degradation occurs, engineers are alerted well in advance of reaching threshold values.

  1. Action Engine

In order to manage models effectively, it is important to understand the outcome of model driven decisions. The Action Engine connects to PLCs, control systems and maintenance processes within the business. It uses risk scores provided to make real-time decisions to improve equipment operation (adjusting variable speed drives or creating maintenance tickets for engineers) within validated confidence intervals. Outcomes of decisions made by the model are recorded, to allow for improved baseline calibrations in future.

Controlled Update Ranges

Even well verified AI updates can have unexpected effects in edge-case scenarios. If newly updated algorithms change the operation of heavy machinery without predictable control then potentially dangerous situations can occur within post deployment stabilisation periods.

To prevent this, Physical AI implementations within enterprises implement controlled update ranges with 3 levels of risk:

Low Risk (Fully automated)

Automated actions taken to complete low risk tasks. These could include recording data drift, updating maintenance logs, and ordering standard consumables.

Medium Risk (Partially automated)

Adjustments in operations based on context, but requiring single click confirmation from engineers. This is particularly important if newly updated model weights are being used to change conveyor belt speeds or heating curing processes.

High Risk (Controlled)

Hard coded physical constraints within the equipment mean that in extreme anomalies, algorithms controlling decisions are bypassed. Safe shut down processes for equipment will occur immediately, and alerts will be sent to control engineers within the facility.

Industrial processes within production require effective model lifecycle management. Leaders and founders looking to create new deep tech ventures with edge native MLOPs in their designs, can look into ongoing research at Aperture Venture Studio.

Top comments (0)