Quick read · 7 min read
You can't manage AI risk by bolting controls onto existing processes; you need a new control plane built around model behavior, ownership, and continuous evidence.
Key takeaways
- Every AI system is a risk-producing asset that needs tiered controls, not a one-size-fits-all policy.
- Define a numeric risk appetite per use case, like maximum bias ratio or false positive rate.
- Continuous monitoring must check for drift, fairness, security, and data quality, not just accuracy.
- Human escalation paths with clear override and rollback procedures are non-negotiable for high-severity outputs. <!-- omnithium-quick-read:end -->
The operating problem
You can't manage AI risk by bolting controls onto existing processes. You need a control plane built around model behavior, ownership, and continuous evidence. Most AI risk programs fail because they start with the model, not the business process. A team deploys an LLM-based tool for credit decisions, and the model risk group scrambles to apply a policy written for regression models. They either block the launch for six months or wave it through with a vendor attestation. Both outcomes are wrong.
Enterprise risk management frameworks were built for deterministic systems. AI is probabilistic, adaptive, and often opaque. When you bolt AI onto ERM as an afterthought, you get shadow inventories, inconsistent risk acceptance, and control gaps that surface only after an incident. A bank's model risk team doesn't need another policy document. They need a control plane that treats every AI system as a risk-producing asset with tiered controls, continuous monitoring, and human escalation tied to existing accountability.
The architecture that holds up
The control plane has five components: inventory, tiering, monitoring, escalation, and reporting. They form a closed loop; each feeds the next.
Click each stage to inspect the controls that keep an agent workflow reliable after launch.
Inventory as code, not a spreadsheet. Every model, internal or vendor, gets a record with owner, data lineage, deployment environment, downstream business process dependencies, and a rollback commit. If it produces a decision or influences one, it's in the inventory. Reconcile against procurement, cloud logs, and code repositories monthly. A model that exists only in a Jupyter notebook but is called by a production API is still in scope.
Tiering with explicit thresholds. Not all AI is equal. A customer service chatbot that suggests FAQs is tier 1: no direct action, basic monitoring. A fraud detection model that blocks transactions is tier 3: pre-deployment red-team, continuous fairness monitoring, and human review for high-severity outputs. Define tier boundaries by business impact and reversibility. A bank's LLM credit decision support tool might be tier 2 if it drafts summaries for human underwriters, but tier 3 if it auto-approves loans below a threshold. The tier determines validation depth, monitoring frequency, and escalation requirements.
Continuous monitoring that checks more than accuracy. Track model drift (PSI or KS statistic), data quality degradation (schema changes, missing values), adversarial inputs (prompt injection success rate), and fairness metrics (false positive rate by protected class). For a healthcare payer's claims fraud model, a coding guideline update can silently shift input distributions. Continuous monitoring catches that drift before false denials pile up. For a deeper look at validation and simulation, see our AI Agent Testing Playbook.
Human escalation with latency budgets. For high-severity or low-confidence outputs, a human must review before action. Define clear override and rollback procedures. If the fraud model flags a claim as fraudulent with 60% confidence, a human analyst reviews it. If they override, the system logs the override and feeds it back for retraining. Set a maximum time-to-review for tier 3 outputs, like 15 minutes for transaction blocking, and alert if exceeded.
Board reporting that shows control failures, not F1 scores. Non-technical executives need risk tier changes and unresolved high-risk findings. A dashboard showing that three models moved from tier 1 to tier 2 last quarter is actionable. A list of accuracy metrics is not.
Where teams usually fail
The most common failure is treating AI risk as a purely technical problem. When the CISO responds to a prompt injection attack on an internal copilot that leaked supplier contract terms, the first question is "how do we patch the model?" The better question is "which downstream systems consumed that compromised output?" If the answer is "we don't know," you've already failed. You need data lineage and blast radius analysis before you need a patch.
Another failure: relying on vendor attestations or model cards without independent testing. A vendor says their fraud model is 99% accurate on their benchmark. But your claims data has different demographics and coding practices. You need to test against your own data and threat model. A manufacturing firm learned this the hard way when a vendor's "enterprise-ready" copilot leaked contract terms after a simple prompt injection. We've catalogued similar failures in AI Agent Failures: Lessons Learned from Enterprise Deployments.
Missing risk appetite is another. Without a numeric threshold per use case, teams accept models with wildly different failure consequences. A false positive in a marketing recommendation engine costs a few dollars. A false positive in a lending decision can trigger regulatory action. If you haven't defined a maximum bias ratio for lending decisions, you're gambling. Define thresholds like maximum false positive rate by protected class, maximum PSI drift per month, and minimum human review rate for tier 3 outputs.
Finally, monitoring only aggregate accuracy. Accuracy hides fairness, explainability, security, and data lineage risks. A model can be 95% accurate overall but systematically deny credit to a protected group. That's a compliance failure, not a technical one. You need subgroup-level metrics and adversarial test suites, not a single AUC number.
A related trade-off: when you add human review to a high-volume fraud model, you trade latency for safety. If your review queue exceeds 100 cases per analyst per day, you'll get rubber-stamping. That's a control failure. Size the review team to the expected override rate, not the total volume.
How to measure progress
Progress isn't measured by the number of models in production. It's measured by the number of models under active control. Here are the signals that matter.
Inventory coverage. What percentage of known AI systems are in the asset inventory? Reconcile against procurement records, cloud logs, and code repositories. If you can't answer that, you're managing anecdotes. Aim for 100% within 30 days of deployment. A model that's not in the inventory has no owner, no tier, and no monitoring.
Tiering completion. Every model in the inventory should have a risk tier assigned and a validation plan tied to that tier. If a model has been in production for six months without a tier, that's a control failure. Track the percentage of tier 2 and tier 3 models with completed pre-deployment red-team exercises.
Monitoring coverage. How many tier 2 and tier 3 models have continuous monitoring for drift, fairness, and security? If the answer is less than 80%, you're flying blind. But coverage alone isn't enough. Check that alerts actually trigger incident response. A dashboard that shows green while a model silently degrades is worse than no dashboard.
Escalation events. Count how many times a human overrode a model decision and what happened next. Did the override trigger a retraining review? Did the model get rolled back? Track mean time to detect (MTTD) and mean time to respond (MTTR) for model incidents. These are leading indicators of control effectiveness.
Board reporting. If your board hasn't seen a risk tier change or a control failure in the last quarter, either you're not looking hard enough or you're not reporting honestly. Report the number of unresolved high-risk findings, not just the number of models in production.
What to build next
The next step isn't more tools. It's a different operating model. You can use AI itself for risk sensing: anomaly detection on model input distributions, regulatory change monitoring, and third-party risk scoring. But every AI-generated risk alert needs human validation before it triggers action. An AI that flags a vendor as high risk based on news sentiment is useful; an AI that automatically terminates the contract is not. This builds on the predictive and prescriptive approach we outlined in Agentic AI for Enterprise Risk Management.
Extend third-party and supply-chain risk management to cover AI models and data providers. Require independent testing of vendor claims against your own data and threat model. A vendor's SOC 2 report doesn't tell you if their model drifts on your data. Run your own red-team exercises and data drift tests before renewal.
Integrate AI risk signals into existing ERM accountability. The business process owner who accepts a model's risk should be the same person accountable when that model fails. That's how you avoid the "throw it over the fence to IT" problem. Wire model risk findings into the same risk register as operational risk, with clear remediation owners and due dates.
Start with the inventory. You can't manage what you can't see. Build the living asset inventory, tier every model, and wire monitoring and escalation into the control plane. The rest follows.
Top comments (0)