We should not design our Industrial AI around just the model.
When a model detects an anomaly or recommends an action, we need to define how it will be reviewed, if it is within acceptable limits and what happens if we are unsure.
Human oversight can be included.
- Detection, Recommendation and Authorization. Detection: The system detects a point of interest.
Recommendation:
The system provides a recommended action based on the information it has.
Authorization:
The correct individual/process with authorisation decides if the action can be taken.
By breaking this down, we can ensure that the recommendation made by the AI does not automatically get authorised.
- Define when an observation should be reviewed.
Within the decision flow we can define what situations will require a human to review.
This could be:
Observations are missing or unreliable
Inconsistent information
Outside of defined limits
Additional authorisation needed for recommendation.
Communication issues have occurred.
This will be application specific.
Provide supporting information.
When providing a recommendation, we should provide additional information so an operator can understand why they are being recommended this action.
This could be the asset in question, what was observed, when it was observed, what was happening within the system and what constraint applies.
To ensure the recommendation can be reviewed.Plan for intervention.
Human oversight should not just be an authorisation button. We should think about how the authorised individual can intervene, how the system will react during communication issues and what conditions need to be met to return to normal operations.Test under challenging scenarios.
A useful way to test would be to look at how the system reacts under challenging scenarios.
This could include time delays, missing observations and uncertainty.
Questions to ask include:
Does the system know when there isn't enough information?
Is the situation escalated correctly?
Can the operator understand why the situation has been escalated?
Has the operator been able to intervene?
Has the system returned to normal based on the agreed operational conditions?
Simple flow chart:
Industrial Observations
Data Validation
AI Analysis
Recommendation
Check Limits and Authorization
+----+----+
| |
Within Review
approved required
conditions |
| v
| Human Review
| |
+----+-----+
This is a simple concept and will depend on the equipment, application, interfaces and operational conditions.
Final Note
Human oversight should be designed into your Industrial AI.
To ensure you have a reliable decision flow you should clearly define limits, recommendations should be easy to understand, situations should be escalated and you should be able to test what happens when things don't go as planned.
Take a look at Aperture Venture Studio's research into Verification and Human Oversight.
Top comments (0)